1. Who is affected, and how badly?
Before looking for the cause, measure the impact: how many customers are failing, whether the same customers are failing repeatedly, which endpoints are involved, and how far the error rate and response times have moved from normal. This decides how urgent everything that follows is, and who needs to be told.
2. Stop the damage first
If there's a safe way to reduce the harm, a rollback, turning off a flag, scaling up, failing over, do it before the investigation is finished. You can stop the damage without knowing the cause. Write down what you did and when, because it changes what the data shows from then on.
3. List what changed
Systems that were working stop working because something changed. Collect everything from the hours before the incident: deploys, feature flags, config, dependency upgrades, migrations, infrastructure, traffic and the outside services you depend on. This list is where most investigations end up finding the cause.
4. Treat each suspect as a hypothesis
For each likely cause, ask what you'd expect to see if it were true, and what would rule it out. A deploy should show errors only from the new release, in code it changed. An outside service should show slow or failing calls to it. A database problem should show in the queries of the failing requests. Look for the evidence that would disprove the favorite first: it's the quickest way to stop chasing the wrong one.
5. Follow one failing request
Aggregates say something is wrong; a single failing request says what. Pick one, open its trace or logs, and see where its time went and where it failed: which query, which call to another service, which line of your code. Then check that what you found holds for the other failing requests too.
6. Keep the timeline as you go
Record each event with its time as it happens: when it started, when someone noticed, what was tried, when it recovered. Reconstructing it afterwards from memory and chat history is slow and usually wrong. The same timeline becomes the postmortem.
7. Root cause, trigger and contributing factors
In the postmortem, keep three things apart: what triggered the incident (a traffic surge), the root cause that let the trigger do damage (a connection pool too small for it), and what made it worse or slower to fix (no alert on pool usage). Fixing only the trigger means the next trigger causes the same incident.
How ForgeOps follows the same order
An incident's page in ForgeOps asks these questions in this order, as numbered steps: who's affected and how badly, what changed just before it, the likely cause with its confidence and evidence, and the affected endpoint's slowest request, its slowest query and the outside service it waited on. It ends with the full timeline, which the generated postmortem reads from too, so the two never disagree.
The likely cause comes from five documented rules over the data on the incident, not from an AI model, and it's labeled a lead rather than a diagnosis.
Read more:
See it on a real incident
The deploy wasn't the problem
A deploy eight minutes before the failures, and the evidence ends up at a slow query instead.
Two deploys, and the wrong one looked guilty
Two deploys minutes apart, part of the telemetry missing, and the most recent deploy isn't the one that did it.