Why the latest deploy is always the first suspect
Timing is persuasive. If errors start eight minutes after a release, the release is the most recent thing anyone did on purpose, and it's the one change everybody on the call knows about.
But teams that ship several times a day almost always have a deploy shortly before any incident, whether or not it's related. Closeness in time tells you where to look first. It doesn't tell you the answer, and two deploys a few minutes apart make it even less reliable: the most recent one gets the blame because it's the most recent.
Check 1: did the errors start after it was running?
Compare the first failing request with when the new version was actually serving traffic, not when the pull request was merged or the pipeline started. A rolling deploy can take minutes to reach every server, and errors that began before the first new server took traffic can't have come from it.
Look at when the error rate rose above its normal level, not only at the first single error. One stray error before a deploy is common; a sustained rise is what you're dating.
Check 2: are the failures coming from the new release?
Most error reporting SDKs tag each error with the release or version that produced it. Group the failing events by release. If they all come from the new version, the deploy stays the main suspect. If the old version is failing just as often on the servers that haven't been updated yet, the deploy didn't cause it: something both versions depend on did.
Check 3: does the failing code run through what changed?
Take the files the deploy changed and the stack traces of the failing requests, and look for overlap. A changed file that appears in every failing stack trace is strong evidence. A deploy that only touched an email template, while the failures are timeouts in checkout, is evidence against it, however close the timing.
With two or more deploys close together, do this for each one separately. The deploy whose changes are in the failing code is the better suspect, even when it's not the most recent.
Check 4: what else changed?
Plenty of incidents have no deploy behind them at all. Before settling on the release, list everything that changed in the hour or two before the incident started:
- A feature flag turned on for more users, or a remote config value edited
- A dependency or runtime upgraded, sometimes as part of an unrelated change
- A database migration, or a change to indexes or connection settings
- Traffic: a surge, a marketing email, a batch job or a crawler hitting one endpoint
- An outside service you call getting slow or returning errors
What a rollback does and doesn't prove
Rolling back is often the right way to stop the damage, and you don't need to be sure of the cause to do it. As evidence, though, it's weaker than it looks. A rollback restarts every process, which also clears exhausted connection pools, full queues and bad caches, so an incident that ends with a rollback wasn't necessarily caused by the deploy.
The opposite is clearer: if the errors carry on after the old version is back, the deploy wasn't the cause, and the investigation should move on straight away.
How ForgeOps checks this for you
When an incident opens, ForgeOps scores every deploy from the 15 minutes before it: how close the timing is, and how much of the spike came from that release. With GitHub or GitLab connected, it reads what each deploy changed and checks those files against the failing stack traces, listing a match as evidence and a deploy whose changes appear in none of them as a point against it. With no deploy reported, it uses the release your SDK sends with each error.
Everything else that changed in the two hours before shows on the incident too, labeled potentially relevant, and the deploy competes with the other rules (an outside service, the database, database connections, background jobs) rather than winning by default. The result is a lead with its confidence and evidence, not a verdict.
Read more:
See it on a real incident
Two deploys, and the wrong one looked guilty
Two deploys minutes apart, part of the telemetry missing, and the most recent deploy isn't the one that did it.
The deploy wasn't the problem
A deploy eight minutes before the failures, and the evidence ends up at a slow query instead.