Skip to main content
The Dark Art of Postmortem DebuggingRead
Sheet 1 · Bring-up
Embedded Linux · BSP · Silicon

Bring-up, read as the schematic.

A board comes up in a fixed order: rails, reset, DDR, kernel, probes. No Crashlytics down here, only dmesg, the U-Boot console and a device tree blob. logcat.ai walks that order and stops at the block that never lit, with the kernel line that says why.

T0T1T2T3T4T5 STALLEDT6U1U2U5U8PMICSoCI2C MUXGPIO EXPVDD_COREVDD_IO_1V8 neverLDO3_EN · never driven · from U8, behind U5[ 2.014] gpio-expander 1-0020: parent busi2c-mux 0-0070 not ready

A board comes up in a fixed order, so the block that never lit is the finding and the kernel line that says why is the citation.

T0 · 12 V inJ1 sees 12 V. Nothing to read yet.
T1 · PMIC railsU1 raises the core and DDR rails.
T2 · Reset releasedU2 leaves reset. U-Boot on the console.
T3, T4 · DDR trained, kernelImage verified, kernel handed off.
T5 · Driver probes · 06 / 07The i2c mux waits on a rail whose enable pin sits on an expander behind the mux.
T6 · RootfsNever reached. A blocked task at 120 s.
Illustrative. Synthetic data.
Sheet 1 · Bring-up
Embedded Linux · BSP · Silicon

Bring-up, read as the schematic.

A board comes up in a fixed order: rails, reset, DDR, kernel, probes. No Crashlytics down here, only dmesg, the U-Boot console and a device tree blob. logcat.ai walks that order and stops at the block that never lit, with the kernel line that says why.

T0 · 12 V inJ1 sees 12 V. Nothing to read yet.
T1 · PMIC railsU1 raises the core and DDR rails.
T2 · Reset releasedU2 leaves reset. U-Boot on the console.
T3, T4 · DDR trained, kernelImage verified, kernel handed off.
T5 · Driver probes · 06 / 07The i2c mux waits on a rail whose enable pin sits on an expander behind the mux.
T6 · RootfsNever reached. A blocked task at 120 s.
123456123456ABCDABCDT0 12V INT1 PMIC RAILST2 RESET_NT3 DDR TRAINEDT4 KERNELT5 PROBES · STALLEDT6 ROOTFSVIN_12VVDD_DDR_1V2VDD_CORE_0V9RESET_NDDR_DQ[31:0]MMC0RGMIII2C0_SDA/SCLVDD_IO_1V8 · never enabledI2C_CH0I2C_CH1LDO3_EN · never driven · GPIO on U8, which sits behind U5J1DC INT0 · OKU1T1 · OKPMICBUCK2BUCK1PGLDO3VINEN3U2T2 · OKSoC4x ARM64 · 6.6.13-evb-r3VDD_CORERESET_NDDRMMC0RGMIII2C0U3T3 · OKDDR4trained in U-BootU4T4 · OKeMMCkernel + dtb loadedU6T5 · OKETH PHYprobed 1.902 sU5T5 · DEFERREDI2C MUX0-0070 · probe never completedVCCCH0CH1U7NEVERTEMP SENSORU8NEVERGPIO EXPANDERGPIO0SDAU5 · WHY IT NEVER LIT · dmesg[ 1.214] i2c-mux 0-0070: probe deferred: supplier vdd_io_1v8 not ready[ 1.215] regulator vdd_io_1v8: probe deferred: enable gpio 1-0020 line 0 not ready[ 2.014] gpio-expander 1-0020: parent bus i2c-mux 0-0070 not ready[ 122.901] INFO: task kworker/u8:1:57 blocked for more than 120 seconds.BOARDREF-EVBREVCSHEET1 OF 4CYCLEU5 · LDO3 · U8CITED3 dmesg linesNODEmux@70 vdd-supplyDRAWN BY logcat.ai DiagnoseIllustrative. Synthetic data.

A board comes up in a fixed order, so the block that never lit is the finding and the kernel line that says why is the citation.

Reached the kernel and then died? Bring the core it wrote. Read without a debug image: recovered kernel log, kernel release, boot command line, the fault, and where each processor was.

Sheet 2 · Errata
Sheet 2 · Errata

Why embedded debugging hurts.

Every BSP is bespoke. Every device tree is different. Every bringup is a new fight. Each line below is pinned to the place on the sheet where it bites.

ERRFailure modeOn the sheet
E01
Bringup is its own disciplineDriver probe ordering, device tree overlays, board revisions: the failure modes that block a bring-up engineer do not appear in app-layer tools at all.
U5 · probe order
E02
BSP fragmentationVendor-specific kernels, downstream patches, recipe-level integrations: every BSP has its own drift from upstream. Generalist AI tools assume mainline; embedded reality does not.
U2 · 6.6.13-evb-r3
E03
Build and runtime tangled togetherA boot failure can root-cause back to a Yocto recipe, a meta-layer dependency, or a kernel config. Investigators need the runtime signature and the build provenance side by side.
U4 · kernel + dtb
E04
Multi-arch divergenceARM, x86, MIPS, RISC-V: the same image, the same recipe, different behavior per machine. Without an engine that reasons across architectures, every fix is a one-off.
U2 · 4x ARM64
Four errata, one sheet. The engine reads the sheet, not the vendor's deltas.
Sheet 3 · Two traces
Sheet 3 · Two traces

10 hours of grep, or 10 minutes of investigation.

What bringup debugging looks like today against what it looks like with logcat.ai. Same board, same boot, two ways of walking the sheet.

Before · manual reconstruction
01Grep dmesg, syslog, journalctl across multiple boots
02Eyeball device tree source and overlay output
03Diff working BSP image against broken one byte by byte
04Email the SoC vendor and wait three days
05Hold tribal knowledge in two engineers' heads
Σ10 hours per investigation. Knowledge stays in those two engineers.
After · one investigation
01Upload dmesg, journalctl, U-Boot output, and the device tree blob
02Ask: why does this board fail to come up cleanly?
03Get cited root cause across BSP layers in minutes
ΣMinutes per investigation. Knowledge captured in the platform, not in two heads.
Sheet 4 · Delta
Sheet 4 · Delta

Delta tells you what changed across the BSP.

Two boot traces across kernel revisions, BSP rebases, or vendor patches, aligned, diffed, and root-caused. Rev B boots clean; rev C stalls at U5. Delta names the net that moved.

REV B · CLEAN BOOTU1U2U5U8PMICSoCI2C MUXGPIOVDD_CORELDO3 always-onvdd-supply = &ldo3 · U5 probed at 0.94 s
REV C · STALLS AT U5U1U2U5U8PMICSoCI2C MUXGPIOVDD_CORELDO3 gpio-enabledvdd-supply = &ldo3_gpio · U5 never probed
NetRev BRev CRead from
LDO3always-ongpio-enabled from U8device tree, both revs
LDO3_ENno such netnew, driven by U8 GPIO0device tree, rev C
U5 probe0.94 s, OKdeferred x5, never OKdmesg, both revs
rootfsmounted 3.4 snever reachedjournalctl, rev B only
2 traces aligned · 1 divergent block · 1 phandle. Illustrative. Synthetic data.
Sheet index
Sheet index

What the engine brings to embedded.

The same primitives that power Diagnose elsewhere, applied to BSP-specific signals. Each one is a way of reading the sheet.

S1
Deep Research on a single bootMulti-step investigation over one bringup attempt: kernel oops, journalctl boot logs, U-Boot output. Hypothesis, evidence, cross-check, root cause.
one sheet, every layer
S2
Delta across BSP variantsCompare boot traces across kernel versions, board revisions, or meta-layer integrations. Isolate which downstream patch, or which dropped one, changed behavior.
two sheets, aligned
S3
Format-agnostic profilingThe engine generates domain context dynamically for any plain-text log: custom drivers, vendor diagnostic streams, in-house tooling. No per-format parser required.
any plain-text net
S4
Citation disciplineEvery claim cites the log line, the dmesg session, the device tree node it came from. Failed hypotheses surface as failed, so reviewers know what was ruled out.
every claim pinned
One platform · three doorways into embedded
DiagnoseInvestigation engine for any signal: bringup, runtime, regression across BSPs.Open Diagnose
AI architectureHow the engine grounds, cites, and refuses to train on customer data.Read the AI architecture
On-prem readyCustomer logs never leave the customer environment in on-prem deployments. Architecture diagram published.See the security architecture
Sheet 1, annotated
Sheet 1, annotated

Built for hardware bringup.

The failure modes that block a board bring-up engineer, named explicitly and pinned to the sheet.

1
Driver probe-ordering failuresEPROBE_DEFER cascades, regulator-before-device races, i2c-mux phandle resolution: the engine reasons about declared against implicit ordering across the device graph.
2
Device tree validationDTC compile errors, overlay conflicts, phandle resolution failures, and the runtime signatures of malformed nodes.
3
Board-revision quirksWhen the same BSP boots cleanly on rev B and hangs on rev C, the engine compares dmesg from both and isolates the divergent driver attach sequence.
4
Cross-architecture divergenceThe same recipe behaving differently on ARMv7, ARM64 or RISC-V. Format-agnostic profiling means the same investigation surface applies.
5
U-Boot and bootloader debuggingReset failures, DDR initialization issues, signed-image verification mismatches, surfaced from U-Boot console output before the kernel even runs.
6
Multi-board test analysisBoot traces from a fleet of test boards: same recipe, different units. Delta isolates which boards diverge and what subsystem they diverge in.
123456123456ABCDABCDT0 12V INT1 PMIC RAILST2 RESET_NT3 DDR TRAINEDT4 KERNELT5 PROBES · STALLEDT6 ROOTFSVIN_12VVDD_DDR_1V2VDD_CORE_0V9RESET_NDDR_DQ[31:0]MMC0RGMIII2C0_SDA/SCLVDD_IO_1V8 · never enabledI2C_CH0I2C_CH1LDO3_EN · never driven · GPIO on U8, which sits behind U5J1DC INT0 · OKU1T1 · OKPMICBUCK2BUCK1PGLDO3VINEN3U2T2 · OKSoC4x ARM64 · 6.6.13-evb-r3VDD_CORERESET_NDDRMMC0RGMIII2C0U3T3 · OKDDR4trained in U-BootU4T4 · OKeMMCkernel + dtb loadedU6T5 · OKETH PHYprobed 1.902 sU5T5 · DEFERREDI2C MUX0-0070 · probe never completedVCCCH0CH1U7NEVERTEMP SENSORU8NEVERGPIO EXPANDERGPIO0SDAU5 · WHY IT NEVER LIT · dmesg · callouts 1 to 6 are the failure modes at left[ 1.214] i2c-mux 0-0070: probe deferred: supplier vdd_io_1v8 not ready[ 1.215] regulator vdd_io_1v8: probe deferred: enable gpio 1-0020 line 0 not ready[ 2.014] gpio-expander 1-0020: parent bus i2c-mux 0-0070 not ready[ 122.901] INFO: task kworker/u8:1:57 blocked for more than 120 seconds.BOARDREF-EVBREVCSHEET1 OF 4CYCLEU5 · LDO3 · U8CITED3 dmesg linesNODEmux@70 vdd-supplyDRAWN BY logcat.ai DiagnoseIllustrative. Synthetic data.123456

The same sheet as the hero. Each numbered marker is where that failure mode shows up on a real board.

Who reads the sheet
Who reads the sheet

Where embedded teams use it.

Across bringup, integration, and validation work.

T1
Yocto integration engineersRecipe-level regressions after a meta-layer bump, kernel version change, or BSP rebase. Diagnose investigates the runtime signature; Delta compares before and after images.
sheets 3, 4
T2
BSP bringup teamsDriver probe failures, device tree resolution races, board-revision-specific quirks. Investigations cross U-Boot, kernel init, and driver attach in one pass.
sheet 1
T3
IoT firmware engineersOpenWrt and Buildroot firmware behaving differently across field units. Compare crash signatures across units to isolate hardware against firmware causes.
sheet 4 · Delta
T4
Silicon validation teamsFirst boots on new silicon revisions. The engine treats unknown signals as plain text and generates domain context dynamically. No SDK integration required.
sheet 1 · first boot
BSPs the engine sees signals fromWorks on dmesg, journalctl, and U-Boot output from these silicon-vendor BSPs. Coverage is signal-level, not specialist-level. The engine reasons about what the logs say, not about vendor-proprietary deltas.
NXP Linux BSPs (i.MX family)Renesas RZ/G and RZ/VTI SitaraQualcomm IoTRockchipAllwinnerPlus any BSP that emits standard kernel and systemd signals
J2 · Inputs
J2 · Inputs

Signals supported.

Drop any of these on the connector and Diagnose investigates them end to end. Five pins today; new binary formats are added on request.

Where Diagnose stops, todayBuild-time intelligence lands with Remediate, now in research preview. Today, Diagnose covers everything from the kernel up, on any image you have already booted.
J2INPUTS12345
P1
dmesgKernel ring buffer across ARM, x86, MIPS, RISC-V. Oops, panics, driver attach traces, EPROBE_DEFER cascades.
P2
Kernel core dumpsThe core a crashed machine writes, x86-64 or ARM64, plain or compressed, read without a debug image: recovered kernel log, kernel release and boot command line, the fault the kernel reported, and where each processor was. Add the matching debug image for symbol-level depth.
P3
journalctlSystemd journal binary streams or text exports. Boot phases, unit failures, service-dependency resolution issues.
P4
U-Boot · bootloader logsBoot console captures from U-Boot, GRUB, or vendor bootloaders. Reset, DDR init, image verification, kernel handoff.
P5
Yocto / OpenEmbedded artifactsdo_compile logs, bitbake task output, image manifests. Surfaces the runtime signatures of build issues on already-booted images.
What Diagnose investigates today
ONRuntime signals from any image built with Yocto, Buildroot, OpenWrt, or custom toolchains
ONBoot-time failures: U-Boot, kernel init, systemd
ONBuild artifacts that contain runtime signatures (oops captures, dmesg snapshots)
What's on the roadmap, not today
NCThe bitbake build process itself (build-time failures during compilation)
NCRecipe-level dependency resolution before image generation
NCLayer compatibility analysis as a static check
NC: no connection yet. These land in Remediate (now in research preview). Today, Diagnose owns everything from the kernel up.
Sheets elsewhere

Analyzers for the BSP stack

Kernel logs, traces, and crash dumps from custom silicon, each with a dedicated deep-dive.

A1Bugreport AnalyzerCrashes, ANRs, and 30+ sections from an Android bugreport.
A2Logcat AnalyzerRoot cause from a raw logcat stream, cited to the line.
A3Dmesg AnalyzerKernel panics, oopses, and driver faults from dmesg.
A4Perfetto AnalyzerJank, startup, and scheduling from a Perfetto trace.
A5Bundle AnalysisCross-file correlation across a whole diagnostic archive.
A6Ramdump AnalyzerCrash root cause from a kernel ramdump, no logs needed.
A7Rosbag AnalyzerWhat the robot published, reported and stopped doing.
Notes on the sheet
Notes on the sheet

Embedded-specific questions.

What embedded engineers ask before they upload their first log.

Q01
Does the engine handle vendor-proprietary log formats?Yes for plain-text streams. The system generates domain context dynamically for any text log: vendor diagnostic streams, in-house tooling, custom drivers. For structured binaries (Android bugreports, CAN bus traces) there's a native parser. New binary formats are added on request.
Q02
Which silicon-vendor BSPs do you support?Signal coverage works on dmesg, journalctl, and U-Boot output from NXP i.MX, Renesas RZ/G, TI Sitara, Qualcomm IoT, Rockchip, and Allwinner BSPs, plus any BSP that emits standard kernel and systemd signals. Coverage is signal-level; the engine reasons about what the logs say, not about vendor-proprietary deltas.
Q03
Can you debug Yocto build failures?Not today. Diagnose covers runtime: boot failures on a built image, kernel oops from a misconfigured driver, dmesg from a broken device tree. The bitbake build process itself and static recipe-level analysis land with Remediate, now in research preview.
Q04
Do you parse journalctl as binary or text?Both. Binary journal streams are parsed natively. Text exports (journalctl --no-pager output, copy-paste from a terminal) are read by the format-agnostic profiling layer. Boot phases, unit failures, and service-dependency issues all surface the same way.
Q05
How does the engine handle multi-architecture divergence?The investigation engine reasons about what the logs say, not about the target architecture. ARMv7, ARM64, MIPS, and RISC-V dmesg all go through the same path. Delta surfaces where the same recipe diverges per architecture.
Q06
Where do my logs go for an on-prem deployment?Customer logs never leave the customer environment. LLM inference runs on a customer-controlled frontier-model endpoint inside the customer VPC. See the security architecture page for the boundary diagram.
Sheet 4 of 4 · Title block
Bringup integration

Bring us a real boot trace.

Upload a dmesg, a journalctl session, a U-Boot capture, or a Yocto-produced image's runtime artifacts. We'll show you Diagnose on it: your actual board, your actual BSP.

BOARD[your board]REV[the one that fails]
BSP[your BSP, any vendor kernel]INPUTSdmesg · journalctl · U-Boot · dtb · core
FINDINGthe block that never litCITEDthe kernel line that says why
DRAWN BY logcat.ai Diagnose · YOUR LOGS STAY IN YOUR ENVIRONMENT ON-PREMSHEET 4 OF 4
Bracketed fields are yours to fill. Everything else on this page is illustrative, synthetic data.