Skip to main content
The Dark Art of Postmortem DebuggingRead
perfetto
System trace analysis

The stall is the silence on the main thread. You never scrub to find it.

logcat.ai opens the trace as structured, queryable data and reads across frames, scheduling, binder and power to the cause.

reading
6 of 412 tracks
track
4.160 s4.240
Framessf
4180
4181 · 31.2 ms
4182
main4120
in
18.4 ms
doFrame
doF
Binder4120_2
transact
binder1203_7
waits lock
wm.anim1256
runnable
CPU 4sched
idle
kw
sf
aud
1256
idle
4.183 s
playhead4.183 s
the silencemain blocked 18.4 ms on a binder call; the lock holder is runnable, not running, on a capped CPU 4.
slice 8843112 · 3 queriesSynthetic data
01
Profiled on open
02
The frame that dropped
03
The silence18.4 ms in a binder call that waits on a lock. The gap is the finding.
04
The holder, four tracks down
perfetto trace analyzer
take_07.pftrace2.1 M slices412 tracks
reading8 tracks of 412 shown
queried as structured datawindow 80 ms
whole take
0 s8.0 s4.160 to 4.240 s, shown below
track
4.160 s4.1804.2004.2204.240vsync
Framessurfaceflinger
MS
4180
4181 · 31.2 ms · dropped
4182
4183
maintid 4120 · app.vendor.gallery
MS
input
doF
18.4 ms · nothing runs here
doFrame, late
doFrame
doFrame
RenderThreadtid 4137
MS
Draw
DrawFrame
Draw
Draw
Binder:4120_2app → system_server
MS
transact · reply 18.4 ms
tx
binder:1203_7system_server
MS
waits WindowManagerGlobalLock
wm.animatortid 1256 · holds the lock
MS
R
runnable 14.1 ms, no CPU
run
CPU 4sched
MS
idle
kworker
sf
audio@2.0
1256
idle
kworker
CPU 4 freqcounter
MS
1.8 GHz0.9 GHz, capped
4.183 snot read yet
loop: frame 4181
60 Hz · 16.67 ms per frametake 07 · 8.0 s4.183 s
at playhead4.183 sframe 4181, 11.9 ms in
the silencemain is blocked 18.4 ms in a binder call to system_server. Nothing scheduled on this lane.
the holdertid 1256 holds WindowManagerGlobalLock and sits runnable 14.1 ms: CPU 4 is capped and busy.
cited toslice 8843112
sched 4 @ 4.1717
3 queries, 41 rows
The stall is the silence on the main lane: 18.4 ms where a region should be. The panel had already named the lock holder before the playhead reached it.Illustrative. Synthetic data.
System trace analysis

The stall is the silence on the main thread. You never scrub to find it.

A trace holds millions of slices across every thread, CPU and counter. logcat.ai opens it as structured, queryable data, reads across frames, scheduling, binder and power to the cause, and plays the answer back on the tracks that matter.

01
Profiled on openFrames, startups, CPU, binder, power, contention, ANRs. Before anything is asked.
02
The frame that dropped4181 ran 31.2 ms against a 16.7 ms budget.
03
The silence18.4 ms of it, in a binder call that waits on a lock. The lane fills in around the gap. The gap is the finding.
04
The holder, four tracks downRunnable, not running, on a capped core. Cited to the slice.
02why it is hard
take_07.pftrace

The data is all there. Finding the one stall is the problem.

A trace is the ground truth for performance on the device, and also the hardest artifact to read by hand. Four reasons, each drawn on the tracks.

maintid 4120
CPU 0sched
Binder
01Millions of slices

A few seconds of tracing yields millions of slices across hundreds of threads and every CPU. The UI renders it; finding the regression in it is on you.

queryyou
SELECT s.name, ts.state FROM slice s JOIN thread_state ts ON …
answer
nothing plays until the query is right
02SQL or nothing

Real answers live in SQL over the trace tables: joins across slice, sched and thread_state. Powerful, but a language and schema most engineers touch once a quarter.

frame
main
binder
lock
CPU 4
03Cross-track correlation

A janky frame is a render slice, blocked on a binder call, waiting on a lock held by a thread the governor descheduled. The cause is four tracks away from the symptom.

alice
M
reads the trace fluently
team
M
team
M
team
M
04No shared vocabulary

Only a couple of people on the team read a trace fluently. Which tables and tracks matter rarely leaves their head, and leaves with them.

the real cost

Reading traces by hand, on the clock.

Red is time. Each bar is a recurring cost of triaging performance without a tool that reads the trace for you.

01
Hours zooming and panning a timeline to find one frame
02
Hand-written SQL over the trace, rebuilt from scratch each investigation
03
Performance knowledge bottlenecked in one or two specialists
04
Findings that live in a screenshot, not a shareable, cited report
05
The same startup and jank triage repeated release after release
Illustrative. Synthetic data.
03what is a perfetto trace
take_07.pftrace

Every track on the device, down to the nanosecond.

Perfetto is the system-wide tracing format for Android and Linux. A single capture records scheduling, frame rendering, binder IPC, locks, power rails and app startups. The ten tracks below are what one 8.0 s take holds, at whole-take density; the amber line is where the hero zoomed in.

track
0 s2.04.06.08.0 s
what it holds
Frames
Render timeline, jank, dropped frames
Scheduling
Per-CPU thread scheduling and run queues
Thread state
Running, runnable, blocked, sleeping
Slices
Nested trace events on every track
Counters
CPU frequency, memory, custom metrics
Binder IPC
Synchronous and async transaction latency
Power rails
Per-rail energy on supported devices
Startups
Cold and warm app launch breakdown
Contention
Monitor contention and blocked threads
ANRs
none in this take
App-not-responding events and causes
Standard .perfetto-trace and .pftrace files from on-device System Tracing, the Perfetto command-line tools, or record_android_trace.Illustrative. Synthetic data.
04how it works
take_07.pftrace

From a binary trace to cited findings, in one pass.

Queried live over the trace, no lossy flattening. The engine reads the trace directly, so anything in the Perfetto trace format, Android system traces and Linux ftrace-backed captures included, is in scope. The playhead sits where the answer lands.

inputbytes
binary trace · 0x0a 0x0b 0x08 0x84 … · 2.1 M slices · 412 tracks
one passno export
01  Open as structured data
02  Ask in plain English
03  Cited findings
answer lands here
01Open as structured data

The trace is never flattened to text. It opens as structured, queryable tables and is profiled on open across frames, startups, CPU, binder, memory, power, contention and ANRs. That baseline is the same for every trace, before anything is asked of it.

02Ask in plain English

"Why did the UI jank at 4.2 s?" No SQL, no schema. The investigation queries the live trace round after round: query, read the rows, refine the hypothesis, correlate across slices, sched and thread_state.

03Cited findings

Each finding names the slice and track it came from. Every query it ran is auditable, and the run closes by naming the concrete next angles a deeper investigation would pursue.

Large multi-hundred-megabyte traces are streamed straight into the engine rather than loaded into memory wholesale. Queries are bounded, so analysis stays responsive on big captures.

Illustrative. Synthetic data.
05what logcat.ai does differently
take_07.pftrace

Reasoning over the live trace, not a flattened export.

Four things the panel in the hero was quietly doing. Each is drawn on the same tracks.

01Deterministic profile, on open

Every trace is profiled on open across frames, startups, CPU, binder, memory, power, contention, ANRs and more. Each area returns a headline and cited findings. That baseline is the same for every trace, before anything is asked of it.

frames
startups
CPU
binder
memory
power
contention
ANRs
threads
02Deep research, in plain English

Ask "why did the UI jank at 4.2 s?" with no SQL or schema knowledge. Grounded in what your trace actually contains, the investigation digs round after round against the live trace and correlates across slices, sched and thread_state to localize the cause. Every query it ran is auditable.

questionyou
why did the UI jank at 4.2 s?
roundslive trace
round 1 · frames
round 2 · binder
round 3 · sched, lock
answercited
the stall is a lock held by a descheduled thread · slice 8843112
03Charts that build themselves

The trace projects into a dashboard: frame timing, CPU residency, thread states, power. Generated from the trace, never hand-built, and every value traces back to a query.

gallery
40
feed
25
system_server
15
surfaceflinger
6
jank framesby process
every bar traces back to a query
04Delta: compare traces side by side

Delta correlates two traces, before and after a change, or a fast device against a slow one, and isolates which subsystem regressed. The fastest way to prove a fix actually worked: the silence on the main lane is gone.

take_06before
doF
18.4 ms
doFrame, late
take_07after fix
doF
3.1
doFrame
the silence is gone
delta
regressed: binder → lock → sched
Illustrative. Synthetic data.
06watch it work on a real trace
real product footage

The full investigation, end to end.

Not a highlight. The entire deep-research run, ranking the worst frames and following the one it can attribute to a 149 ms binder wait on YouTube's main thread beside a system_server lock storm. The lateness is queueing, not slow drawing.

deep research run · 51 s · 16:10
one trace, one question"The UI drops a frame hard and a couple of apps stutter. What is causing the worst jank in this trace?"
12 steps2 data tiers~30 s traceReal Deep Research · Perfetto system trace · identity as captured
criticalWhatsApp frame 148 ms late — rendered in 7 ms, so a start delay, not draw cost
criticalYouTube frame 133 ms late: a 149 ms binder wait on its main thread
warnsystem_server: five lock stalls over 50 ms on the activity-manager lock
infoRoot: queueing at a startup layer handoff, not slow drawing
The worst misses are not slow drawing: frames that rendered in 4.6 ms still presented 130 ms late, and the queue sat on the producer side, five frames deep. The compositor stayed on time throughout. Real output of the run in the frame.
the run, as steps on one laneReal Deep Research · Perfetto system trace · identity as captured
01Rank the latest frames trace-wide
per process and per layer
02Read the worst WhatsApp frames
148 ms late, 7.2 ms of work
03Attribute YouTube's worst frame
149 ms binder wait at frame start
04Read system_server's lock stalls
five stalls over 50 ms, one lock
05Check the compositor and the queue
48/48 on time; 5 frames queued
06Find where the evidence stops
scheduling starts 11 s late
07what it finds in a real trace
Real Deep Research · Perfetto system trace · identity as captured

Profiled on open. A finding per track, named to the process behind it.

A dozen processors read the trace on open (frames, scheduling, binder, contention, power), each returning a cited finding. These are the per-process findings from one live trace; the bar is how loud each track was.

tracktoneseverityfinding
Frames
critical
WhatsApp is the dominant jank source: repeated 'App Deadline Missed' on the render path.
Binder
warn
YouTube's main thread blocked 149 ms on a synchronous IActivityManager call.
Contention
warn
Sustained background lock contention in system_server (ActivityManager).
CPU
warn
Little cores (CPU0 to CPU3) spent ~63% of the trace pinned at peak frequency.
Startups
info
YouTube warm launch ~495 ms, narrowly inside the 500 ms budget.
Power
info
Energy broadly distributed across CPU clusters, display and modem.
Memory
good
clean
No memory-pressure events, no low-memory kills in the trace.
charts from a real trace

A sample of what comes back.

Built automatically from the trace, every figure tied to the slices behind it.

Jank frame count by process (actual frame timeline)
com.android.systemui
72
com.linkedin.android
47
com.whatsapp
40
com.google.android.yo…
22
system_server
9
com.Slack
6
com.google.android.ap…
2
Jank frames by processfrom a real trace
CPU time share by process
system_server16%
surfaceflinger15%
com.Slack12%
com.google.androi…11%
kcompactd09%
others38%
CPU time by processfrom a real trace
08how a trace reaches you
take_07.pftrace

Three kinds of slow, and what the trace says about each.

A trace is captured because something felt slow. Each kind leaves a different shape on the tracks.

Framessf
4180
4181 · dropped
4182
main
blocked
01Jank you can see but cannot place

The app stutters and the trace is a million slices wide. The analysis pulls the dropped frames out of the noise and ties each one to the thing blocking the main thread. No manual zoom-and-hunt.

startupcold
cold start 1.9 s · bind → activity → first frame
main
lock
binder
inflate
02A startup or stall that is too slow

Cold start drags, a screen hangs, and "it feels slow" is not a bug report. The investigation localizes the stall in time and names the lock, binder call or scheduler gap behind it, cited to the slice.

frames
CPU
threads
power
03A trace you need to see, not read

You captured it on a phone or an embedded target and do not want to live in the timeline. The dashboard reads the trace's structure and what was actually regressing, then builds the charts that fit it: the hot subsystems, the stalls, the frames that dropped.

Illustrative. Synthetic data.
09frequently asked questions
six tracks

Everything you need to know about Perfetto trace analysis.

Formats, mechanism, what it detects, whether you need SQL, size and security. Read down the lane.

01
What trace formats are supported?

Standard Perfetto traces (.perfetto-trace and .pftrace) captured by on-device System Tracing, the Perfetto command-line tools, or record_android_trace. Because we read the trace directly rather than flattening it to text, anything in the Perfetto trace format (Android system traces and Linux ftrace-backed captures included) is supported.

02
How does the analysis actually work?

We never flatten the trace to text. The file is opened directly as structured trace tables and queried with SQL. Every trace is profiled for frames, startups, scheduling, binder, power, memory, contention, ANRs and more; then an investigation runs SQL across the live trace to answer your specific question, self-correcting across multiple rounds.

03
What can it detect?

Jank and dropped frames with the render slice and the blocking cause, slow cold and warm startups with a phase breakdown, main-thread stalls from lock contention or binder latency, CPU contention and frequency capping, power-rail drain on supported devices, low-memory kills and memory pressure, and ANRs with the blocked thread. Findings are cited back to the slices and tracks they came from.

04
Do I need to know SQL or the Perfetto UI?

No. Ask questions in plain English and the agent writes the SQL for you, runs it against your trace, and explains what it found. If you do know SQL, the queries it ran are right there in the cited findings to build on.

05
How large a trace can I analyze?

Large multi-hundred-megabyte traces are handled by streaming the file straight into the engine rather than loading it into memory wholesale. Queries are bounded and run against the live trace, so analysis stays responsive on big captures.

06
Is my trace data secure?

Your trace data is encrypted in transit (TLS 1.3) and at rest (AES-256), processed in your own tenant, and never used to train AI models. Traces are queried live in the engine rather than converted into stored, searchable chunks, so their contents are not persisted as an index. Files are deleted after 90 days, or immediately from your dashboard. Enterprise plans offer on-premise deployment for regulated or air-gapped environments.

10stop living in the timeline
end of take

Stop living in the timeline.

Upload a trace and let logcat.ai surface the stall behind the jank. The panel reads to the end of the take; you read the finding.

Framessf
4180
4181 · 31.2 ms · dropped
4182
4183
4184
8.0 s · end of take
maintid 4120
doF
18.4 ms · the finding
doFrame, late
doFrame
doFrame
doFrame
read to the end
stop
take 07 · 8.0 s · 1 stall found · cited
part of the logcat.ai platform

Perfetto is one surface.

The same engine reads your logs, bugreports and kernel traces.

Deep ResearchMulti-step investigation over the live traceopen →
Delta comparisonCompare traces across builds and devicesopen →
Logcat analyzerCorrelate traces with app-level logsopen →
Bugreport analyzerWhole-system analysis across every Android layeropen →
Illustrative. Synthetic data.