Skip to main content
The Dark Art of Postmortem DebuggingRead
dmesg analyzer
verdict · T+142.118 731

The oops is atT+142.118. The reason is atT+139.880.

A kernel log stamps every line with seconds since power-on, so the trace is never the first thing that went wrong. logcat.ai reads the whole sequence and names the driver, the failed allocation, and the line to open first.

T+ since power-on23 go ·1 no go
0.000000Booting Linux on physical CPU 0x0GO
0.000000Memory: 3900000K/4194304K availableGO
0.320005smp: Brought up 1 node, 4 CPUsGO
1.204220mmc0: new HS400 MMC card at address 0001GO
2.010000VFS: Mounted root (ext4 filesystem)GO
2.744410vendor-isp 1c00000.isp: probedGO
3.871441eth0: Link is Up - 1Gbps/FullGO
4.115000usb 1-1: new high-speed USB deviceGO
5.3 to 139.8134.5 s nominal, 212 lines folded
139.880210cma_alloc: alloc failed, 2048 pages, -12WARN
141.203112vendor-isp: frame buffer alloc failedWARN
142.118731Unable to handle kernel NULL pointerNO GO
142.118738Internal error: Oops: 96000005 [#1]
142.118901pc : vendor_isp_queue_frame+0x4c
142.119218---[ end trace ]---
T+139.880memory refused 2048 pages
T+141.203the ISP frame buffer never got them
T+142.118the driver dereferenced NULL
Illustrative. Synthetic data.
dmesg analyzer
boot sequence boardIllustrative. Synthetic data.
T+ seconds since power-onevb0 · 6.1.0-evb
24 stages23 go1 no go
stampsubsysreportstatus
[ 0.000000]cpuBooting Linux on physical CPU 0x0 [0x000000ff]GO
[ 0.000000]cpuLinux version 6.1.0-evb (build@evb) #1 SMP PREEMPTGO
[ 0.000000]mmMemory: 3900000K/4194304K available (cma 65536K reserved)GO
[ 0.012340]cpuCPU: All CPU(s) started at EL2GO
[ 0.104112]cpuclocksource: arch_sys_counter: mask 0xffffffffffffffGO
[ 0.320005]cpusmp: Brought up 1 node, 4 CPUsGO
[ 0.612200]blockSCSI subsystem initializedGO
[ 0.640100]usbusbcore: registered new interface driver usbfsGO
[ 0.901330]powervendor-pmic 0-0034: regulators registered (14)GO
[ 1.204220]blockmmc0: new HS400 MMC card at address 0001GO
[ 1.522000]blockEXT4-fs (mmcblk0p2): mounted filesystem with ordered data modeGO
[ 2.010000]blockVFS: Mounted root (ext4 filesystem) readonly on device 179:2GO
[ 2.744410]drvvendor-isp 1c00000.isp: probed, firmware v0.0.0 loadedGO
[ 3.871441]neteth0: Link is Up - 1Gbps/Full - flow control rx/txGO
[ 4.115000]usbusb 1-1: new high-speed USB device number 2 using xhci-hcdGO
[ 5.002900]powerthermal thermal_zone0: trip points registered, passive at 85CGO
[ 5.330017]secaudit: initializing netlink subsys (disabled)GO
5.33 to 139.88134.5 s nominal, 212 lines folded
[ 139.880210]mmcma: cma_alloc: alloc failed, req-size: 2048 pages, ret: -12WARN
[ 141.203112]drvvendor-isp 1c00000.isp: frame buffer allocation failed (-12)WARN
[ 142.118731]drvUnable to handle kernel NULL pointer dereference at virtual address 0000000000000038NO GO
[ 142.118738]Internal error: Oops: 96000005 [#1] PREEMPT SMP
[ 142.118744]Modules linked in: vendor_isp vendor_pmic xhci_hcd ipv6
[ 142.118901]pc : vendor_isp_queue_frame+0x4c/0x1a0 [vendor_isp]
[ 142.118903]lr : vendor_isp_start_stream+0xa4/0x1d8 [vendor_isp]
[ 142.119218]---[ end trace 0000000000000000 ]---
[ 142.119220]allno further reports. ring buffer ends here.HOLD
dmesg · kernel ring buffer · 1,418 linesread in full · cited by stamp
verdict · T+142.118 731

The oops is at
T+142.118.
The reason is at
T+139.880.

A kernel log stamps every line with seconds since power-on, so the trace is never the first thing that went wrong. logcat.ai reads the whole sequence: it names the driver behind the fault, finds the allocation that failed three seconds earlier, and tells you which line to open first.

T+139.880contiguous memory refused 2048 pages (-ENOMEM)
T+141.203the ISP driver's frame buffer never got its pages
T+142.118vendor_isp_queue_frame dereferenced NULL

A kernel log is a launch sequence: every line is stamped with seconds since power-on, every subsystem reports in order, and the oops is the one that does not.

02why kernel log analysis is hard
Illustrative. Synthetic data.

Kernel messages require deep system knowledge to read.

Four things make a kernel log harder than any user-space log. Each is marked in the raw evidence on the left, and named in plain words on the right.

stamp02refraw line, as printed
[ 142.118731]01BUG: unable to handle page fault for address: ffff9a3e0000c038
[ 142.118734]#PF: supervisor read access in kernel mode
[ 142.118736]#PF: error_code(0x0000) - not-present page
[ 142.118737]PGD 0 P4D 0
[ 142.118744]03Oops: 0000 [#1] PREEMPT SMP NOPTI
[ 142.118801]RIP: 0010:vendor_isp_queue_frame+0x4c/0x1a0 [vendor_isp]
[ 142.118812]Code: 48 8b 47 38 48 85 c0 74 1e 48 8b 40 08 <48> 8b 40 38 ...
[ 142.118820]RSP: 0018:ffffb0c0c0e7bd48 EFLAGS: 00010246
[ 142.118901]Call Trace:
[ 142.118904]04vendor_isp_start_stream+0xa4/0x1d8 [vendor_isp]
[ 142.118907]v4l2_ioctl+0x5c/0x80 [videodev]
[ 142.118910]__x64_sys_ioctl+0x8f/0xc0
[ 142.118913]do_syscall_64+0x3b/0x90
[ 142.118940]Modules linked in: vendor_isp videodev vendor_pmic xhci_hcd ipv6
[ 142.119218]---[ end trace 0000000000000000 ]---
every stamp in this column is seconds since power-on, not a time of day
01
Cryptic messages

Abbreviations, hex addresses and internal terminology. "BUG: unable to handle page fault": what does it mean?

02
No wall-clock time

Default dmesg shows seconds since boot, not actual time. Lining it up with any other log means converting every stamp by hand.

03
Kernel panics and oops

When the kernel crashes, the stack trace is raw symbols and offsets. Naming the failing driver or subsystem takes expertise.

04
Subsystem complexity

Driver bug, hardware failure, or configuration? Kernel subsystems interact, and the trace shows you only the last one to speak.

the real cost
What manual analysis bills you for.
cost 01Hours reading kernel documentation to understand messageshours
cost 02Trial and error with driver versions and kernel parametersdays
cost 03Search results that never match your exact issuehours
cost 04Hardware replaced for what was a software issuebudget
cost 05Production downtime while a boot failure is diagnoseddowntime
03what a kernel log is, and what its levels mean
status vocabulary of the board

Dmesg is the kernel ring buffer: every message since boot.

Hardware detection, driver initialization and system errors that user-space logs never see. Four message levels tell you how bad a line is; eight subsystems tell you where it came from.

stampthe line, as the kernel prints itboard statuswhat it means for the system
[ 142.120001]Kernel panic - not syncing: Fatal exception in interruptNO GO · haltKernel panic.System halt. Unrecoverable, needs a reboot.
[ 142.118738]Internal error: Oops: 96000005 [#1] PREEMPT SMPNO GOOops.Recoverable kernel error. The system may continue.
[ 88.410220]BUG: sleeping function called from invalid context at mm/slab.c:0HOLDBUG:Kernel assertion failed. An unexpected state was detected.
[ 139.880210]WARNING: CPU: 2 PID: 0 at drivers/media/isp.c:0 vendor_isp_irq+0x0/0x0WARNWarning.Non-critical. The system continues normally.
kernel subsystems
Eight consoles report in. Every line is filed to one.
blockBlock I/O.Disk drivers, filesystems, storage errors.
drvDrivers.Device drivers, module loading, firmware.
netNetworking.Network drivers, protocol stack, connectivity.
usbUSB.USB devices, hubs, enumeration.
mmMemory.OOM killer, memory allocation, page faults.
powerPower.ACPI, suspend/resume, power states.
cpuCPU.Scheduling, frequency scaling, thermal.
secSecurity.SELinux, audit, security modules.
04what logcat.ai does differently
Illustrative. Synthetic data.

Kernel analysis that connects a hardware event to its system-wide effect.

Four things the analyzer does with the sequence. On the left, what it reads; on the right, what you get.

what is readwhat you get
> What caused this kernel panic?
[ 142.118731]drvvendor_isp_queue_frame: NULL dereference
[ 141.203112]fwisp firmware v0.0.0 requested a 2048-page frame buffer
[ 139.880210]mmcma_alloc refused it: contiguous pool exhausted
[ 139.412000]usercamera service opened a second capture session
Trace failures across system layers

Ask what caused the panic and the investigation follows the chain through drivers, firmware and framework, and lines kernel events up with user-space behavior. ARM, x86 and RISC-V.

[ 142.118738]Internal error: Oops: 96000005 [#1] PREEMPT SMPNO GO
[ 142.118901]pc : vendor_isp_queue_frame+0x4c/0x1a0 [vendor_isp]module
[ 88.410220]BUG: sleeping function called from invalid contextHOLD
[ 139.880210]WARNING: CPU: 2 PID: 0 at drivers/media/isp.c:0WARN
Automatic panic and oops detection

Panics, oops, BUG lines and warnings are found without a query. Stack traces are parsed and mapped to the failing module, and each finding is classified so you know what to fix first.

subsysspanlinesstatus
blockT+0.612 to T+139.02214GO
mmT+0.000 to T+139.88041WARN
drvT+2.744 to T+142.118307NO GO
powerT+0.901 to T+120.3077GO
Subsystem-aware grouping

Messages are filed by subsystem: block I/O, networking, memory, drivers, power. A timeline shows the sequence, so related issues across subsystem boundaries sit next to each other.

> Show USB errors during resume.answered in 4.1 s
[ 120.301004]powerPM: suspend exit
[ 120.344110]usbusb 1-1: device descriptor read/64, error -71
[ 120.612008]usbusb 1-1: reset high-speed USB device number 2
Ask questions in plain English

"Show USB errors during resume." No kernel expertise required, and the answer comes back in under 5 seconds with the stamped lines it was read from.

05how it works
count from the log to the cause

Four steps from kernel log to root cause.

The same sequence every time, counted down like the board on the left of the hero: each step has to report before the next one starts.

countstepwhat happensreports
T-4Capture kernel logsUpload dmesg output, a boot log, a serial console capture, or the kernel ring buffer pulled out of a bugreport.1,418 lines in
T-3Detect anomaliesKernel panics, oops, BUG lines, warnings and errors are found and stamped before anything is asked.1 no go·2 warn
T-2Map to subsystemsEach finding is filed to the driver, module or kernel component that wrote it, so related lines sit together.drv · mm
T-1Root cause analysisThe chain is read in order and the verdict says whether the failure is hardware or software, and which line to open first.3 to 4 min
T+0The answerThe oops is atT+142.118. The reason is atT+139.880: the ISP driver's frame buffer was refused by the contiguous memory pool.cited by stamp
06watch it work on a real one
Real Deep Research · kernel suspend/resume log · device identity substituted

A real board, a real log, three real root causes.

Nothing on the left is drawn. It is the product's own investigation card. The sequence on the right is the run's steps, in the order it took them.

Deep Research · one kernel log, one question

“This board rapidly suspends and resumes, and I2C, the camera, and Wi-Fi fail on wake. What's the root cause of each?”

Platform Snapdragon 820A automotive development platformKernel Linux 4.4.138-perf+Subsystem Kernel PM · Camera CCI + I2C · WLAN-PCIe
The investigation6 of 35 steps
Fork wake-source, I2C and PCIe threads
suspend entry every ~10 s
Read the kernel's own wake report
four wake IRQs every resume, one unnamed
Time the actual sleep
~700–760 ms asleep per cycle
Find who drives the cadence
sys.wakeup.request from str.rc scripts
Trace the camera failure
msm_sensor_power_up -22 on every resume
Attribute the I2C NACKs
one burst at one slave, before the first suspend
The answer

Three symptoms, three causes: a camera resume fault, one boot NACK burst, Wi-Fi unloaded on suspend

The three symptoms have three separate causes: the camera@1 instance on the CCI bus fails its resume callback on each of the four captured wake cycles, the i2c-msm-v2 errors retrieved are a single boot-time NACK burst on 7575000.i2c with none found in any captured resume window, and Wi-Fi is unavailable across each wake because the driver is deliberately unloaded before suspend and cold-reloaded afterwards — while the ~10 s cadence is re-armed from userspace via sys.wakeup.request, with no suspend-abort message found in the evidence retrieved. Camera: `msm_sensor_power_up:181` is entered with two of its four printed arguments zeroed — byte-identical (`ffffffc06ec25850 ffffffc06ec23d80 0000000000000000 0000000000000000`) at 71.844963, 82.125034, 92.364804 and 102.594367 s — returning rc -22 and failing `a0c000.qcom,cci:qcom,camera@1` with PM error -14 on each of the four captured wake cycles, while the CCI master itself initialises normally. I2C: the 25 i2c-msm-v2 error/warning lines counted for this log fall in one boot-time burst on bus 7575000.i2c at slave address 0x11 between [36.802748] and [37.053395], and no i2c-msm-v2 error was found at any resume in the evidence retrieved — the I2C-path failure seen at wake is the camera device on the CCI node. Wi-Fi: the PCIe endpoint is reset and the WLAN module removed before suspend, so the PCI PM core's attempt to restore the endpoint meets `cnss_pci 0000:01:00.0: Refused to change power state, currently in D3`, logged against the endpoint under its cnss_pci driver name, and connectivity returns only after the post-wake `msm_pcie_enable` + insmod sequence completes.

The camera sensor instance camera@1 on the CCI bus fails its resume callback on every captured wake, with an identical argument signature each time.
Each captured suspend ends with the kernel's own wake-IRQ report: the same four wake lines are reported on every resume, one of them resolving to "null", and no suspend-abort message was found in the evidence retrieved.
No i2c-msm-v2 error was found in any captured resume window; the I2C-path failure visible at wake is the camera device on the CCI node.
A device at slave address 0x11 on bus 7575000.i2c never acknowledged during a ~250 ms boot-time burst, which accounts for the 25 i2c-msm-v2 error lines counted for this log.
Next: map the unnamed wake IRQ to its device, then fix the sensor power-up arguments on the resume path
service_failure · Kernel PM / Camera CCI + I2C + WLAN-PCIe
Real Deep Research output · device identity substituted
frame35 steps · 4 findings
the question, as askedplatform: Snapdragon 820A ADP (automotive) · linux 4.4

> This board rapidly suspends and resumes, and I2C, the camera, and Wi-Fi fail on wake. What's the root cause of each?

stepwhat the run did, and what it foundconf
01Fork wake-source, I2C and PCIe threadssuspend entry every ~10 slow
02Read the kernel's own wake reportfour wake IRQs every resume, one unnamedhigh
03Time the actual sleep~700–760 ms asleep per cyclehigh
04Find who drives the cadencewake request from the suspend-to-RAM scripthigh
05Trace the camera failurepower-up with zeroed arguments, EINVALhigh
06Attribute the I2C NACKsone burst, one slave, before the first suspendhigh
07Separate the Wi-Fi storyscripted module teardown, not a link faulthigh
Three symptoms, three causes: a script-driven cadence, a camera fault on every wake, a scripted Wi-Fi teardown.
07the full investigation, end to end
Real Deep Research · kernel suspend/resume log · device identity substituted

Not a highlight. The entire run.

The whole deep-research run, scrolling from the triage report through all three root causes. Read it at the speed you would read the log.

T+0.000the run, as recordedT+end · three root causes
frameReal Deep Research · kernel suspend/resume log · device identity substituted
08how a kernel log reaches you
Illustrative. Synthetic data.

A kernel log shows up after something already went wrong.

Three ways that happens. On the left, what lands on your bench; on the right, what the analysis makes of it.

what arriveswhat you get back
[ 142.118738]Internal error: Oops: 96000005 [#1] PREEMPT SMP
[ 142.118901]pc : 0xffffffc0108a3b4c
[ 142.118903]lr : 0xffffffc0108a3e10
[ 142.119012]Call trace: 0xffffffc0108a3b4c 0xffffffc0108a3e10 0xffffffc0104f22a8
[ 142.119218]---[ end trace 0000000000000000 ]---
A panic with a trace you can't read

The kernel dumped an oops and a call stack of raw addresses, and the board is sitting on your bench. A real run traced one such fault to an ISP/CMA allocation failure: decoded, cited, and tied to the subsystem behind it.

[ 1.020114]vendor-isp 1c00000.isp: supply vdd-isp not found, deferringHOLD
[ 1.204220]vendor-pmic 0-0034: regulators registered (14)GO
[ 1.311800]vendor-isp 1c00000.isp: probe retry 2 of 3HOLD
[ 1.402010]cma: cma_alloc: alloc failed, req-size: 4096 pages, ret: -12WARN
[ 1.402221]vendor-isp 1c00000.isp: probe failed with error -12NO GO
A board that won't boot clean

Bring-up on custom silicon: drivers probe in the wrong order, allocations fail, and the log scrolls past before you can read it. The analysis localizes the failure in the boot sequence and names the driver or resource behind it.

1,418 lines
as a shape
CPUMMBLOCKDRVT+0.000T+142.118
A log you need to see, not read

You want the shape of the boot, not a thousand kernel lines. The dashboard reads the log's structure and what was actually failing, then draws the charts that fit it: the boot timeline, the failing subsystems, the allocations that didn't come back.

09charts from a real kernel log
real analysis output, redrawn for this page

Built straight from dmesg.

The visual layer that comes with every analysis. Every value here traces to a stamped line in one real kernel log; the analysis chose these two charts because they fit what that log contained.

kernel errors by subsystem194 error lines
cpu-boost181
binder6
ADSPRPC2
logd2
wcnss_wlan2
q6asm_callback1

One subsystem wrote 93% of the error lines. That is the row to open first, and the chart says so before anyone reads a line.

free memory above reserved (kB) over time35 samples
21,065 kB9,092 kBfloorT+37T+2235T+3449

Free memory kept dropping to the same floor and being pulled back. The two amber marks are the two lowest samples in the log.

10frequently asked questions
six questions · six answers

Everything you need to know about kernel log analysis.

Questions on the left, answers on the right, the same split as everywhere else on this page.

questionanswer
01What dmesg formats are supported?

Raw dmesg command output, journalctl -k kernel logs, /var/log/dmesg and /var/log/kern.log files, serial console captures, and kernel logs extracted from Android bugreports. Different timestamp formats and kernel versions are handled automatically.

02How long does analysis take?

Quick Search answers kernel questions in under 5 seconds. AI Insights analysis that identifies panics, oops and driver issues completes in 3 to 4 minutes. Deep Research investigations that trace issues through subsystems and reference kernel documentation take 5 to 10 minutes depending on complexity.

03What issues can it detect?

Kernel panics and oops with full stack traces, BUG and WARNING messages with call chains, OOM kills with memory allocation details, driver probe failures and initialization errors, hardware errors (ECC, PCIe, USB), boot failures and device tree issues, and subsystem-specific problems in networking, storage, and memory management.

04Which Linux distributions are supported?

All of them: Ubuntu, Debian, Fedora, RHEL/CentOS, Arch, Alpine, openSUSE, and any custom Linux build. Android kernel logs, ChromeOS and embedded Linux systems too. The kernel log format is standardized across distributions, so if Linux runs it, we can analyze it.

05Is my kernel log data secure?

Your kernel log data is encrypted in transit (TLS 1.3) and at rest (AES-256), processed in your own tenant, and never used to train AI models. Files are deleted after 90 days, or immediately from your dashboard. Enterprise plans offer on-premise deployment for regulated or air-gapped environments.

06Does it work with embedded Linux?

Yes. All major architectures are supported: ARM (32/64-bit), x86/x64, MIPS, RISC-V and PowerPC. It works with embedded build systems like Yocto, Buildroot, OpenWrt and custom Linux configurations. Whether you're debugging a Raspberry Pi, an industrial IoT device or a custom SoC, the parser handles it.

11start debugging the kernel layer
Illustrative. Synthetic data.
stampyour kernel logstatus
[ 0.000000]Booting Linux on physical CPU 0x0GO
[ 0.000000]Linux version [your kernel] ([your build])GO
0.0 to [T]everything your board reported, read in full
[ your stamp ]the line that did not report, and the reason before itNO GO
[ +3 to 4 min ]verdict, cited by stamp, connected to the rest of the systemREAD

Bracketed values are yours to fill. Everything else on this page is how the board reads once you do.

Start debugging the kernel layer.

Upload your kernel logs and let logcat.ai connect them to the rest of the system.