Intro:
Intermittent embedded failures are difficult because the most useful evidence often disappears when the device is reset, power-cycled, or reprogrammed.
This project grew from a real failure in a navigation and telemetry system built around an ESP32-S3 CrowPanel, an STM32MP1 Linux companion, GNSS hardware, and an RM3100 magnetometer. The magnetometer occasionally stopped publishing valid measurements even though the device remained visible over I2C and no ordinary bus error was reported.
My background in automotive diagnostics shaped the central idea: scan it while it is hot. Instead of immediately resetting the system, capture the complete live state while the failure is still active.
What it does
HotTrace is a Codex-assisted embedded diagnostic workflow for capturing and analyzing intermittent faults before a reset destroys the evidence.
The system records:
- the last healthy state and first stale state
- sensor polling and publication counters
- data-ready activity
- I2C and parser error counters
- sensor configuration registers
- expected versus observed configuration
- timestamps and data age
- firmware commit and binary provenance
- bounded snapshots suitable for later replay
A portable analyzer then reads sanitized capture data and produces a human-readable report showing what continued working, what stopped advancing, what configuration changed, and what conclusions are supported by the evidence.
The demonstration can be run without the original hardware by replaying recorded sample captures.
How I built it
The live system uses an ESP32-S3 CrowPanel running ESP-IDF firmware and an STM32MP1 Linux companion that receives telemetry and preserves diagnostic evidence.
HotTrace uses Python 3.11+ with no runtime dependencies, standard-library JSON
and HTTP support, exact Decimal timeline coordinates, static HTML/CSS/JavaScript,
and unittest. The repository includes deterministic synthetic .hottrace
packages so judges can run the complete viewer without hardware or private evidence.
Codex helped with several stages of the work:
- Adding bounded diagnostic counters and state reporting.
- Designing quick hot-state snapshot collection.
- Comparing healthy and failed captures.
- Tracking expected sensor configuration separately from observed configuration.
- Identifying that an early diagnostic register read could itself disturb continuous measurement mode.
- Replacing that observer with a non-destructive diagnostic path.
- Analyzing later evidence after the failure still reproduced.
- Building a portable replay and reporting workflow for judges and other developers.
Git commits, firmware hashes, timestamps, and capture directories are retained so the diagnostic conclusions can be traced back to the exact software and evidence used.
How it works
- Standard-library Python ingestion converts each source line into one normalized event or explicit parse-error event.
- A
.hottracepackage stores authoritative metadata, events, findings, hashes, and source provenance. - The state engine selects the latest explicit observation at or before a cursor—never an interpolated value.
- The comparison engine summarizes structure, transitions, cadence, flat-later behavior, range, and continuity with published generic scoring factors.
- A loopback-only read-only server exposes bounded JSON queries to a repository-contained browser frontend.
Challenges
The largest challenge was avoiding observer effects.
During early diagnostics, reading one RM3100 control register could terminate continuous measurement mode. That meant the diagnostic code intended to observe the failure could potentially create or alter it. Codex helped identify this problem and replace the destructive read with cached expected-state tracking and safer register observations.
Another challenge was distinguishing several possible failure classes:
- an I2C communication failure
- parser or telemetry corruption
- a defective sensor
- loss of sensor configuration
- a power, reset, or connection disturbance
- a software publication failure
The captured evidence showed that the sensor could remain responsive while normal measurements stopped, I2C errors stayed at zero, and a configuration register changed from the expected value. The same reset/configuration-loss signature later appeared with a replacement sensor, making a defective-sensor-only explanation less likely.
The project therefore reports evidence and uncertainty rather than claiming a root cause that has not yet been proven.
What I learned
I learned that a diagnostic tool must be treated as part of the experiment. Reading the wrong register, collecting too much data, or resetting too early can destroy the very evidence needed to understand the failure.
I also learned how valuable provenance is in embedded debugging. A snapshot becomes far more useful when it includes timestamps, counter deltas, expected configuration, Git commits, firmware hashes, and a clear record of which actions were and were not performed.
Most importantly, Codex was useful not only for writing code, but for challenging assumptions, comparing evidence, preserving safety boundaries, and turning raw telemetry into a repeatable engineering workflow.
What's next
The next steps are to:
- add more capture formats and sensor adapters
- improve automated healthy-versus-failed comparisons
- generate Markdown and HTML reports
- add timelines and counter-delta visualizations
- support additional embedded buses and devices
- connect scope or logic-analyzer evidence with software telemetry
- continue investigating the RM3100 configuration-loss event without resetting the system prematurely
The long-term goal is a reusable diagnostic toolkit for embedded failures that disappear when the device is restarted.
Inspiration
HotTrace grew from a real intermittent RM3100 magnetometer investigation on an ESP32-S3 CrowPanel system. The most useful evidence existed only while the system was still running, and a reset could erase the state needed to explain what changed. Static logs also made it too easy to confuse a successful read with new data or an observer gap with a device reset.
The goal became: preserve the evidence first, then make two captures easy to replay and compare without overstating what they prove.
How we built it
HotTrace uses Python 3.11+ with no runtime dependencies, standard-library JSON
and HTTP support, exact Decimal timeline coordinates, static HTML/CSS/JavaScript,
and unittest. The repository includes deterministic synthetic .hottrace
packages so judges can run the complete viewer without hardware or private evidence.
Accomplishments that we're proud of
- Built a deterministic evidence pipeline with one event per source line.
- Implemented timestamp-honest sample-and-hold state reconstruction.
- Built a generic, field-name-neutral cross-capture comparison engine.
- Created Simple Compare with synchronized relative playback and independent event resolution.
- Added an immutable safe capture registry and bounded read-only APIs.
- Reached 99 standard-library tests for the submission package.
- Bundled a deterministic synthetic demo library requiring no hardware.
What we learned
Forensic tooling is valuable when it says exactly what it knows and what it does not know. A successful transport operation is not proof of a new measurement. A repeated value is still a new observation. An observer gap is not necessarily a device reset. A current-value difference is factual, but it is not itself a diagnosis. Codex was especially effective as an implementation collaborator when those rules were made explicit and enforced in tests. Specific input to the Ai is critical. Hardware operations and knowledge are necessary to build a tool even though the Ai is very good.
What's next for HotTrace-SnapShot
Future work includes separately authorized STM32MP1 and STM32MP2 capture agents, additional component profiles, independently referenced power-rail evidence, large IMU/ADC capture optimization, a static sanitized adapter, and report export. These capture agents and hardware integrations are not complete today. Adding a board selection area and more detailed file handling with more component specific attributes.
Log in or sign up for Devpost to join the conversation.