Skip to main content
The Dark Art of Postmortem DebuggingRead
Robotics

The recording shows the symptom.

10:41:004812.00010:42:004872.00010:43:004932.000RECORDINGabsolute clockKERNEL LOGsince bootarm_controller CRITICAL · 10:41:51[ 4863.204] rt task blocked 340 mst+231 s · symptom above, cause below

The cause is in another file, on another clock.

Two files, two clocks. One offset makes every line parallel, and puts the symptom and its cause on the same instant.

A robot is a Linux computer that happens to run a middleware. When a node stops publishing, the recording says exactly when it went quiet and almost never why. logcat.ai reads the recording and the kernel log together, on one timeline, and answers across both with citations.

Illustrative. Synthetic data.
SolutionsRobotics
10:41:004812.00010:41:104822.00010:41:204832.00010:41:304842.00010:41:404852.00010:41:504862.00010:42:004872.00010:42:104882.00010:42:204892.00010:42:304902.00010:42:404912.00010:42:504922.00010:43:004932.000RECORDINGabsolute timestamps, written in the fileKERNEL LOGseconds since boot, no dateone offset for the whole file: kernel [ 4812.000] is recording 10:41:00battery_monitor WARN · 10:41:23arm_controller DEGRADED · 10:41:34/arm/cmd last message, then silence · 10:41:48arm_controller CRITICAL · 10:41:51[ 4846.410] CPU3 thermal throttle, 2.1 GHz to 800 MHz[ 4863.204] rt task blocked 340 ms on run queuet+231 s · symptom above, cause belowt+214 s · same instantnothing logged

The recording shows the symptom.

Two files, two clocks. One offset makes every line parallel, and puts the symptom and its cause on the same instant.

The cause is in another file, on another clock.

A robot is a Linux computer that happens to run a middleware. When a node stops publishing, the recording says exactly when it went quiet and almost never why: the reason is in the kernel log, in memory pressure, in a link that dropped, or on a vehicle bus. logcat.ai reads them together, on one timeline, and answers across all of them with citations.

Illustrative. Synthetic data.
02 / 11 · More than a middleware

A robot is more than its middleware

The recording is one of the files you collected, not all of them. These are the others, and they're where causes usually live. Each keeps its own clock; every one is placed against the recording's.

RECORDINGabsolute clock, the anchorKERNEL LOGseconds since bootPACKET CAPTUREabsolute, another hostSYSTEM TRACEtrace clockVEHICLE BUSbus counterarm_controller CRITICALrt task blocked 340 mspeer stopped answeringarm_controller off CPU 340 msactuator fault 0x1F, decoded from the DBCone instant, five files, five clocks

The recording

Says exactly when a node went quiet. Its absolute timestamps anchor every other clock on this page.

The kernel log

Thermal throttling, an OOM kill, a driver reset, a real-time task blocked past its deadline. None of this reaches the recording, and any of it can stop a node.

The network

A packet capture explains a link that dropped, a discovery that never completed, a peer that stopped answering. The recording only holds the silence that followed.

Memory and scheduling

A system trace shows which process lost the CPU and for how long. A missed control deadline usually has a scheduler or an allocator behind it.

The vehicle bus

Plenty of robots carry CAN. A trace decoded through its signal database puts actuator faults on the same timeline as the node that was commanding them.

one upload

Upload the recording and whatever else you collected from the same machine and the same window. A comparison places them on one timeline and answers across both, rather than leaving you to hold two screens in your head.

Illustrative. Synthetic data.
03 / 11 · Why it is hard

Why this is hard today

Two toolchains, two clocks, and a file that hides its evidence under sensor data.

recording toollog tool

Two toolchains

Recording tools read recordings. Log tools read logs. Nothing reads both, so the correlation happens in your head, off two screens.

10:41:00[ 4812.000]no year?

Two clocks

A recording is in absolute time. A kernel log is often seconds since boot with no year on it anywhere. Lining them up manually is a guess you then build a theory on top of.

camera 6.2 GBlidar 2.9 GBthe two channels that explain the run

Size hides the evidence

Most of a recording is sensor payload nobody needs to read. Tools that insist on reading it turn a five-minute question into an afternoon.

0 after t+228 s0, never publisheda node that dieda channel nobody used

Silence reads as nothing

A node that died and a topic nobody ever used both count zero messages. If your tool can't tell them apart, neither can you.

04 / 11 · Before and after

How a failed run gets debugged

Same recording, same kernel log, two very different afternoons.

Today0 h1 h2 h3 h4 h12345
1

Replay the recording at wall-clock speed and watch for the moment it went wrong.

2

Open the kernel log in a second window and try to reconcile two different clocks.

3

Write a throwaway script to pull the health reports out into something readable.

4

Ask whoever wrote the node, because only they know what its states mean.

5

Do all of it again for the next run, and again for the next engineer.

Hours per run, and a correlation that lives in one person’s head.

With logcat.ai0 min1 min2 min3 min4 min123
1

Upload the recording and whatever else you collected from the same robot.

2

Ask in plain English what happened, or what changed since the last run.

3

Get an answer across every file, on one timeline, cited and with its coverage stated.

Minutes, and an answer the next engineer can check.

05 / 11 · What it does

What the platform does with a robot's files

The same investigation engine every other surface uses, pointed at a recording and its neighbours.

the recordingquestionround 1round 2round 3answer, coverage stated

Deep Research

A multi-round investigation over one recording that queries it, follows what it finds across rounds, and reports what it couldn't read as plainly as what it could. To bring in the robot's other logs, use Delta.

Learn more
runkernelthe same two rails, one offset

Delta

Two runs, or one run and the kernel log underneath it, placed on one timeline. The recording's absolute timestamps anchor files that never wrote down a year.

Learn more
inventoryin fulllog streamin fullhealthin partnot readevery answer says how much of the file it rests on

Coverage, stated every time

Every answer says how much of the recording it rests on: the inventory, the log stream and the health reports each report whether they were read in full, in part, or not at all. A channel nobody decoded and a channel the robot never published on are different facts, and they're reported as different facts.

camera frames, 6.2 GB, inventoried, never decodedpoint clouds, 2.9 GB, inventoried, never decoded/rosout and /diagnostics, readcost tracks these

Sensor payload skipped on purpose

Camera, lidar and point clouds are inventoried and never decoded, so the reading cost tracks the channels that explain a run rather than the ones that fill the file. Where a payload-heavy recording can only be read in part, the answer says so.

Illustrative. Synthetic data.
06 / 11 · When this comes up

When this comes up

Four situations that arrive with a recording and not much else.

one recording, nobody watchingthen the kernel log from the same machine

A run failed in the field

One recording came back and nobody was watching. Ask which components reported a problem and in what order, then pull in the kernel log from the same machine when the recording stops explaining itself.

yesterdaytodaywhat changed

It worked yesterday

Compare the two runs directly: what changed in component health, which channels went quiet, and whether the difference shows up in the robot's system logs rather than in its middleware at all.

robot 1robot 2robot 3same fault?

The same fault on more than one robot

Upload the recordings and the logs from each machine and ask whether it's the same failure, so you know whether you're chasing one robot or a build.

nodes that came upnever published: the driver is the reason

Bring-up on new hardware

New board, same stack. The recording shows which nodes came up and which never published anything, and the kernel log says which driver is the reason.

07 / 11 · What comes back

What comes back from a recording

All three ROS recording formats, read as they came off the robot.

All three formats

The classic single-file bag, the MCAP container, and the database-backed ROS 2 recording. Recognised by content rather than file name, so there's nothing to convert.

Channel inventory

Every channel, its message type, its message count and the window it covers, read from the recording's own index. Channels that carried nothing are kept, because they're often the point.

Component health

Each component's worst state across the run, what it said at the time, and every report that wasn't OK in the order the robot made them.

The log stream

The robot's log messages become a searchable log beside the recording, in the same viewer as every other log you upload.

channel inventory · window on the recording’s clock · messages · coveraget+180 st+210 st+240 st+270 st+300 s/diagnostics120read in full/rosout4,812read in full/arm/cmd2,280stops at t+228 s/arm/state2,360stops at t+236 s/camera/image_raw3,600not decoded, 6.2 GB/lidar/points1,200not decoded, 2.9 GB/arm/estop0never publishedIllustrative. Synthetic data.
08 / 11 · Recording on the robot

Recording on the robot

Nothing here needs a special capture mode. Upload whatever your robot already writes.

ROS 1rosbag record -a
ROS 2, default storageros2 bag record -a
ROS 2, MCAPros2 bag record -a -s mcap
Worth recording

A recording that includes the robot's log channel and its health channel gives you the most, because those are the two read in depth. Recording them costs almost nothing next to sensor data. Collect the kernel log from the same window while you're at it.

sensor payloadlog channel and health channel: the two read in depthalmost free to recordkernel log from the same window
09 / 11 · Analyzers

Analyzers for a robot's stack

A recording is one file among several. Each of these reads another part of the same machine.

Robot Recording Analyzer

What the robot published, reported and stopped doing.

Logcat Analyzer

Root cause from a raw logcat stream, cited to the line.

Dmesg Analyzer

Kernel panics, oopses, and driver faults from dmesg.

Bundle Analysis

Cross-file correlation across a whole diagnostic archive.

Perfetto Analyzer

Jank, startup, and scheduling from a Perfetto trace.

10 / 11 · Questions

Questions about robot debugging

What to upload, how the files line up, and where a person comes in.

01

What do I upload for a robot?

Start with the recording, in whatever format your robot already writes: classic bag, MCAP, or the database-backed ROS 2 format. Then add whatever else you collected from the same machine over the same window, especially the kernel log. The recording tells you when a node went quiet, and the other files usually tell you why.

02

How do you line a recording up with a kernel log?

Recordings carry absolute timestamps, so the recording anchors the comparison. A kernel log that only counts seconds since boot is placed against that anchor rather than guessed at. Where a file's clock genuinely can't be resolved, the comparison says so instead of quietly aligning it wrong.

03

Do I need to convert or export anything?

No. All three ROS recording formats are read as they come off the robot, and they're recognised by content rather than by file name. There's no conversion step and nothing to declare in the upload.

04

Can it answer questions about what the robot saw?

No. Sensor payload (camera frames, lidar returns, point clouds) is inventoried but never decoded, which is what keeps a very large recording quick to analyze. Questions about what the robot did, what it reported, and when it stopped are all answerable. Questions about the contents of an image are not.

05

Is this only for ROS robots?

The recording surface is ROS specific, in all three of its recording formats. Everything else on the robot is not. Kernel logs, packet captures, system traces, bus traces and crash dumps are read the same way whatever middleware runs above them, so a robot without ROS still gets everything except the recording surface.

06

Where does a person come into this?

Throughout. The investigation runs on its own and reports what it found, cited back to the file and the channel, including what it couldn’t read. Deciding what that means, and what to change on the robot, stays with the engineer. Nothing is applied to your fleet.

07

What do I need enabled?

Robot recording analysis needs robotics log analysis on your account, which is the exact wording you'll see if a recording is held back. Without it a recording still uploads and is recognised for what it is, and the analysis step is skipped. The rest of the robot's logs need nothing special. Email sales@logcat.ai if you need it enabled.

RECORDINGKERNEL LOG
11 / 11 · Robotics

Bring the recording and the rest of the robot

Upload a run and the logs from the machine that made it, then ask one question across both.

Illustrative. Synthetic data.