Averages lie. A query can look fine on average and still have a moment where it spiked memory to the point of nearly killing the node. To catch that, you need to see resource usage over time, second by second, not as a single number at the end. That is what per-second resource timelines give you, and they have saved me from more than one mysterious out-of-memory error.
Why a single number is not enough
When a query finishes, system.query_log gives you its peak memory_usage and total time. That is useful, but it flattens the story. Two queries can have the same peak memory and behave completely differently: one holds a steady amount throughout, the other is cheap for nine seconds and then allocates everything in the last second as it builds a huge hash table. The second one is the dangerous pattern, and the summary number hides it.
A timeline shows you the shape. In the CHOps query profiler you can watch CPU, memory, and disk IO for a query plotted per second across its lifetime. The shape tells you things the totals never will.
Reading the shapes
A few shapes come up again and again.
A steady climb in memory that never comes down usually means an aggregation or a join building state that grows with the input. If it climbs past what the node can hold, that is your out-of-memory error waiting to happen. The fix is often a coarser GROUP BY, an approximate aggregate, or processing in smaller chunks.
A sawtooth in IO, lots of reading followed by pauses, often means the query is reading far more from disk than it should and the rest of the system is throttling it. That points back to a primary key or partition pruning problem, the same root cause I covered in debugging slow queries.
A flat line on CPU while the clock keeps running means the query is waiting on something, usually IO or a lock, rather than computing. That is a sign to look at what it is reading, not at the computation itself.
Where timelines beat flame graphs
A flame graph tells you which stage of the query was expensive. A timeline tells you when, during the run, the cost was paid. These answer different questions. For a query that fails partway through, the timeline is more useful, because you can see the exact second memory crossed the line. For a query that is just generally slow, the flame graph is more useful, because it shows which operation to fix.
I tend to start with the flame graph for slow queries and switch to the timeline for queries that crash, spike, or behave inconsistently. The inconsistent ones are where the timeline really earns its place, because the problem only shows up in a specific second that an average would never reveal.
Catching the noisy neighbor
Timelines are also how I find the query that is hurting everyone else. When the cluster gets slow at a particular time of day, I line up the resource timeline with the moment of the slowdown. Usually there is one query whose memory or IO spikes exactly then. Once you have identified it, you can decide whether to optimize it, reschedule it, or limit it.
This is the kind of investigation that is painful with raw system tables and quick with a visual timeline, because your eye catches the spike instantly. The point of per-second resource data is to make the moment things went wrong visible, instead of buried in an average. The query profiler page shows how the timelines, flame graphs, and query inspector work together.
