Reading a flame graph without guessing
Flame graphs get misread constantly, and almost always in the same two ways.
The x-axis is not time
Frames are sorted alphabetically so identical stacks merge into one wide block. Width means share of samples and nothing else. A block 40% as wide as the root was on-CPU in 40% of samples — which is not the same as consuming 40% of wall-clock time.
perf record -F 99 -a -g -- sleep 30
perf script | stackcollapse-perf.pl | flamegraph.pl > out.svgA thread blocked in read() produces no samples at all. It does not appear anywhere on the graph, no matter how long it waits.
Look for plateaus, not towers
Depth is just call nesting. A thirty-frame stack two pixels wide costs nothing; it only means somebody likes small functions. What you want is a wide flat top — a frame that is broad and has little above it. That is where the CPU actually was.
If it looks idle, you are on the wrong graph
A service that feels slow but profiles flat is spending its time off-CPU. That needs scheduler tracepoints rather than sampling — offcputime from bcc, or perf sched. The two cannot be read on the same axis, and stacking them in one image has misled more people than it has helped.