Skip to main content
4.3 Million Vehicles, One Race Condition: What the Ford ITRM Recall Teaches Us About Cross-Layer DebuggingRead
Test-Failure-to-Root-Cause Automation

From Test Failure to Root Cause — In Minutes

CTS/VTS debugging means manually bridging test results, device logs, and upstream patches. logcat.ai replaces that entire workflow — correlating failures with device state and surfacing Gerrit fixes automatically.

Three Disconnected Worlds. One Massive Workflow Problem.

Every CTS/VTS failure forces engineers to manually bridge three completely separate information sources. Today, that bridging is done with grep, timestamps, and hours of Gerrit archaeology across 2M+ test cases.

World 1

Test Results

The CTS XML says FAIL. It gives you a test name, maybe a stack trace. No device context. No system state. No explanation of what actually happened on the device during the test.

World 2

Device Logs

The 100MB+ bugreport contains the answer — buried somewhere in logcat, dmesg, and dumpsys output. You have to manually correlate timestamps between the XML failure and the device log to find the relevant events.

World 3

Upstream Fixes

AOSP Gerrit may already have a patch that fixes your exact failure. But finding it means manually searching commit history, reading changelogs, and cross-referencing test names with code changes. Per failure.

logcat.ai bridges all three worlds automatically. Upload the XML and the logcat — get correlated root causes with Gerrit patches in minutes, not hours.

See It in Action: Camera CTS Failure

A real investigation flow that takes 2 days manually — completed in 5 minutes

CTS XML

testCameraCapture marked FAIL in CTS results XML. No useful context beyond the test name and a generic assertion error.

Logcat

logcat.ai correlates the failure timestamp with device logs. Finds camera HAL timeout during capture — the HAL returned -EBUSY instead of completing the capture request.

Dmesg

Kernel logs reveal the camera driver was returning -EBUSY due to a race condition during HAL initialization. The driver was not fully ready when the capture request arrived.

Gerrit

Deep Research finds an AOSP Gerrit patch that fixes the exact race condition in camera HAL initialization sequence. Patch is merged — the fix exists, it just needs to be cherry-picked.

Root cause identified and fix located in 5 minutes. Without logcat.ai, this is 2 days of manual log correlation and Gerrit browsing.

Before vs After

The workflow transformation for every failing CTS/VTS module

Before: Manual Investigation

  • Run CTS suite, wait for completion
  • Grep XML for FAIL results across modules
  • Open logcat, manually search for timestamps matching each failure
  • Read through thousands of log lines looking for relevant errors
  • Manually search Gerrit for each failure pattern
  • 2+ hours per module, repeated across dozens of modules

After: logcat.ai

  • Upload CTS XML + device logcat
  • logcat.ai correlates every failure with device state automatically
  • Root causes categorized: device config, test bug, framework regression
  • Gerrit patches surfaced with status, links, and relevance
  • 5 minutes per module — regardless of failure count

Why Manual Investigation Breaks Down

The three problems that make CTS/VTS debugging a workflow nightmare

Opaque Failure Reports

XML says FAIL but not why. Engineers waste hours correlating XML results with device logcat manually — matching timestamps, tracing PIDs, and reading through megabytes of unrelated output.

Test Code vs Device Issues

Is the failure a device config problem, a framework regression, or a bug in the AOSP test code itself? Without correlating the XML with device state, you can't tell — and Gerrit often has the fix already.

Scale of Test Suites

CTS alone has 2M+ test cases across 100+ modules. When a suite run produces dozens of failures, manual investigation of each one is days of work. It doesn't scale.

How It Works

From upload to root cause in three steps

1
Step 1

Upload Test Results + Logs

Upload CTS/VTS result XML alongside device logcat and bugreport from the test session. logcat.ai parses both formats automatically.

2
Step 2

Automated Correlation

Delta correlates test outcomes with runtime evidence — matching failure timestamps with logcat errors, dmesg events, and system state at the moment each test failed.

3
Step 3

Root Cause + Gerrit Discovery

Deep Research traces each failure to its root cause and searches AOSP Gerrit for existing fixes. You get categorized failures, evidence chains, and patch links.

Powered by the logcat.ai Platform

CTS/VTS analysis is built on the same core engines that power all of logcat.ai

Deep Research

Investigates failing test behavior autonomously — traces from XML assertion errors through logcat, dmesg, and framework state to find root causes. Searches AOSP Gerrit for existing patches.

Learn more about Deep Research

Delta

Compares passing vs failing builds side-by-side. Upload logcat from a build that passed CTS and one that failed — Delta isolates exactly what changed to cause the regression.

Learn more about Delta

What We Detect

AI-powered categorization of test failure root causes

Test Code Bugs

Missing capability guards, incorrect resource paths, broken assumptions in AOSP test code — with Gerrit patches that fix them.

Device Configuration Gaps

Missing codecs, unconfigured permissions, absent hardware capabilities that cause failures unrelated to code quality.

Framework Regressions

Android framework behavior changes between versions that break test expectations. Delta pinpoints the version where the regression appeared.

Environment Issues

Missing test media files, harness misconfiguration, incomplete test setup — issues that waste debugging time but have simple fixes.

Automatic AOSP Patch Discovery

Deep Research autonomously investigates each test failure and determines whether a fix already exists in AOSP. Relevant patches appear in the analysis report with commit details, status, and direct Gerrit links.

No manual Gerrit browsing required. Relevant patches across the AOSP codebase are surfaced alongside the root cause analysis — merged, in review, or abandoned. The Gerrit archaeology that used to take hours per failure happens automatically.

Supported Test Suites

Full coverage across the Android certification ecosystem

CTS

Compatibility Test Suite

VTS

Vendor Test Suite

GTS

Google Test Suite

STS

Security Test Suite

CTS-on-GSI

CTS on Generic System Image

Cross-File Test Correlation with Delta

Delta correlates CTS/VTS result XML with device logcat to connect test outcomes with runtime evidence. Compare passing vs failing builds to isolate regressions.

Delta Correlation Engine

Multi-file cross-layer analysis

Uploading...
CTS Result XML
<Module name="CtsMediaDecoder"
pass="3" fail="1" total="4"
Test result="fail" HEVC trick play
AssumptionViolatedException
Device Logcat
10:33:14 MediaCodec: h264.decoder OK
10:33:15 MediaUtils: could not open resource
10:33:15 TestRunner: failed: DecodeOnlyTest
10:33:15 AssumptionViolatedException

Use Cases

How teams use CTS/VTS analysis to accelerate certification

OEM Certification

Pass CTS/VTS for device certification. Quickly identify whether failures are device issues or known AOSP test bugs — with Gerrit patches ready to cherry-pick.

ODM Validation

Validate custom ROMs and BSP integrations against CTS requirements across multiple device variants. Delta compares results across variants to isolate BSP-specific failures.

QA Regression Testing

Catch regressions introduced by firmware updates before they reach certification. Delta compares pre- and post-update CTS runs to flag new failures instantly.

Frequently Asked Questions

Common questions about CTS/VTS test analysis.

XML result files from CTS, VTS, GTS, STS, and CTS-on-GSI. Upload the test_result.xml alongside the device logcat from the test session.

Deep Research cross-references test failures against AOSP's public commit history to determine whether a fix already exists. If a relevant patch is found — merged or in review — it surfaces in the analysis report with status, authorship, and a direct link. You get the answer without the manual Gerrit archaeology.

Yes. You'll get test result analysis, but without logcat correlation the root cause analysis will be limited to what's in the XML (failure messages and stack traces). For full root cause analysis, upload both the XML and the device logcat.

Currently optimized for standard Android test suites (CTS/VTS/GTS/STS). Custom test frameworks can be uploaded as unstructured text.

Up to 100MB per file. Large CTS result XMLs with thousands of test cases are supported.

Part of the logcat.ai Platform

CTS/VTS analysis uses the same Deep Research and Delta engines that power bugreport analysis, dmesg debugging, and cross-build comparison across the entire logcat.ai platform.

Stop Bridging Three Worlds Manually

Upload your CTS/VTS results and device logs. Get correlated root causes with Gerrit patches — in minutes, not hours.