Put everything on one timeline
Start with a single list in one time zone: deploys, every other change (flags, config, migrations, upgrades), when the error rate rose, when response times rose, and when customers were first affected. Most wrong conclusions come from comparing two charts with different time zones or different zoom levels.
Watch the resolution of each source. Errors usually arrive one at a time, to the second. Performance metrics often arrive as averages over one or five minutes, so a slow query can't show up until its period ends. A database problem can look like it started after the errors simply because its metric reported later.
From an error to its request
An error on its own says what failed. The request it failed in says why. If your services are traced, the trace id on an error leads to the whole request: every database query, every call to another service, and how long each took. That's the step that turns "checkout is throwing timeouts" into "checkout spent 2.7 of its 3.2 seconds on one query".
If two of your own services call each other, follow the trace across both. The error your users see is often in the service in front, while the cause is in the one behind it.
From the request to the query
In the failing request, look for three things:
- The slowest query, and what share of the request's time it took
- The same statement repeated many times in one request, the classic N+1, which is many small queries adding up to one slow page
- How the database ran the slow query: a plain EXPLAIN (never EXPLAIN ANALYZE on production, which runs the query again) showing a sequential scan over a large table usually means a missing index
A slow query, or an exhausted connection pool?
These look alike and need opposite fixes. When one query gets slow, it holds its database connection longer, other requests wait for a connection, and errors about failing to get one appear. Fixing that query fixes everything.
When the pool itself runs out, usually because traffic jumped, every query waits for a connection, so every query looks slow, and the slowest one is a symptom. The signs: many different queries slowing at the same moment rather than one, the pool's in-use or waiting metric near its limit, and a rise in request rate just before. Here the fix is capacity or shedding load, and tuning the query that happened to be slowest won't help.
Missing data isn't evidence
Metrics agents restart, sampling drops requests and some services aren't instrumented. When a source goes quiet during an incident, treat that as unknown, not as "nothing happened there". A database chart that stops reporting just as the errors start is a reason to check the database another way, not a reason to rule it out.
How ForgeOps lines them up for you
Every incident in ForgeOps gets one timeline built from what actually happened: deploys and other recorded changes, when the endpoint slowed down, the first error, when the error rate rose, and when customers were affected, each linked to its detail. Errors from the server SDKs carry the request they happened in, so an issue opens with the slowest query in that request, its share of the request's time and a likely N+1, with an opt-in query plan on PostgreSQL.
The likely cause weighs a slow query against an exhausted connection pool explicitly: several different queries slowing at once, pool metrics under pressure, connection errors and a surge in traffic all count toward the pool, and the evidence for each is listed.
Read more:
See it on a real incident
The deploy wasn't the problem
A deploy eight minutes before the failures, and the evidence ends up at a slow query instead.
Two deploys, and the wrong one looked guilty
Two deploys minutes apart, part of the telemetry missing, and the most recent deploy isn't the one that did it.