CrashTest

CrashTest turns firmware failure into a test: FREE-WILi injects faults, measures response and recovery, verifies safe behavior, and generates evidence-backed PASS/FAIL reports.

Comment

Inspiration

Embedded systems are usually demonstrated when everything is working perfectly. But real hardware does not live under perfect conditions. Sensors disconnect, communication fails, timing changes, wires come loose, and devices behave in ways we did not expect.

While working on embedded projects, I kept running into the same problem: testing failure usually meant manually disconnecting a wire, watching a serial terminal, and trying to decide whether the firmware had actually handled the problem correctly.

That led me to a simple question:

Cars are crash-tested before we trust them. Why shouldn't firmware be crash-tested too?

That idea became CrashTest.

Instead of treating failure as something that might ruin a demo, CrashTest makes failure the demo.

What it does

CrashTest is a hardware-in-the-loop firmware resilience testing platform built around FREE-WILi OG.

The user physically arms the system using FREE-WILi and triggers a controlled fault. CrashTest then verifies that the device was healthy before the test, injects the fault, monitors the ESP32 response, measures detection latency, checks that the firmware enters a safe state, removes the fault, verifies recovery, and generates a deterministic PASS or FAIL result.

The entire process is visible in real time.

The FREE-WILi display and RGB LEDs show states such as:

READY → ARMED → CHECKING → INJECTING → FAULT → RECOVERING → PASS/FAIL

At the same time, a web dashboard mirrors the physical hardware and displays the live test timeline, root cause, detection latency, safe-state result, recovery status, and previous runs.

After every test, CrashTest automatically generates a detailed engineering evidence report containing the configuration, timestamps, fault information, measurements, captured telemetry, timeline, acceptance criteria, final verdict, and an SHA-256 evidence digest.

In one successful run, CrashTest detected the injected sensor fault in approximately 222.7 ms, verified the safe-state response, recovered successfully, and confirmed stable post-recovery IMU telemetry.

How I built it

The device under test is an ESP32 NodeMCU-32S / ESP-WROOM-32D connected to an ICM-20948 IMU over I2C.

The ESP32 continuously reads the IMU and sends structured telemetry over serial. Instead of printing only human-readable debug messages, the firmware reports events such as FAULT,SENSOR, ROOT_CAUSE,INJECTED_SENSOR_FAULT, SAFE,OK, and RECOVERY,SENSOR,OK.

For the current MVP, FREE-WILi GPIO27 is connected to ESP32 GPIO18 as a dedicated hardware fault-injection input. FREE-WILi can therefore command the DUT to enter its sensor-failure handling path in a repeatable way.

An important technical distinction is that the current implementation does not electrically short the physical I2C SDA or SCL lines. It uses a dedicated hardware fault input to trigger the firmware's sensor-failure path. I chose this architecture because it made the hackathon demo repeatable while still exercising the actual detection, safe-state, and recovery logic of the DUT.

I also built a custom application for the FREE-WILi OG display. The board itself becomes the test console. The yellow button arms the test, the green button starts fault injection, and the red button stops the test or returns to the home state. The seven RGB LEDs and display change with the current validation state.

On the laptop, a Python orchestration layer coordinates FREE-WILi, the ESP32 serial connection, fault timing, validation logic, dashboard state, and report generation.

The validation logic is intentionally deterministic. A PASS requires a healthy baseline before injection, successful fault assertion, correct fault detection, the expected root cause, confirmation of a safe state, successful fault removal, recovery within the allowed timeout, and stable IMU telemetry after recovery.

The browser dashboard is built with Flask, HTML, CSS, and JavaScript. It visually recreates the FREE-WILi interface so the physical device and dashboard feel like one system.

The final architecture combines FREE-WILi OG, RP2040, ESP32, ICM-20948, C/C++, Python, Flask, serial communication, GPIO, and I2C into one hardware-in-the-loop validation workflow.

Why the PASS/FAIL decision is deterministic

One design decision I cared about was keeping the engineering verdict separate from any future AI analysis.

CrashTest does not ask an AI model whether the firmware passed.

The test rules determine the verdict.

If the required behavior is not observed, the test fails. If recovery is incomplete, the test fails. If the reported root cause does not match the injected fault, the test fails.

This makes the output repeatable and auditable.

An AI agent can eventually use the evidence package to explain failures, compare runs, identify patterns, and recommend the next fault to test, but it should never be allowed to change the deterministic PASS/FAIL result.

Challenges I faced

One of the biggest challenges was working with the original FREE-WILi OG hardware. Some older Python interfaces were not compatible with the firmware on the device, and early GPIO-control attempts would hang. I ended up working with the modern FREE-WILi OG BSP, setting up the Pico SDK toolchain, compiling the required firmware components, and verifying GPIO control directly.

Fault injection was another major challenge.

My first approach tried to manipulate the physical I2C line using a transistor. It was interesting electrically, but it was not reliable enough for a hackathon demo. Breadboard connections, transistor behavior, and bus conditions made the outcome inconsistent.

That forced me to rethink the problem. I realized that a fault-testing system is only useful if it can reproduce the same failure reliably. I switched to the dedicated hardware fault-input architecture, which made the entire test deterministic.

Another challenge was avoiding false PASS results.

Initially, simply seeing a recovery message could make the system look recovered. But a sensor could reconnect briefly and fail again immediately afterward. I strengthened the test so CrashTest requires sustained post-recovery telemetry for several seconds and a minimum number of valid sensor samples before declaring success.

Synchronizing everything was also harder than expected. The FREE-WILi buttons, physical display, RGB LEDs, ESP32 firmware, serial telemetry, Python orchestrator, Flask dashboard, and report generator all needed to agree on the same test state.

Getting to the point where I could press a physical button on FREE-WILi and watch both the hardware and browser transition together through ARMED, FAULT, RECOVERING, and PASS was one of the most satisfying parts of the project.

A personal moment that changed the project

One of the experiences that shaped CrashTest happened when my IMU connection became loose.

Sometimes the ESP32 received perfect telemetry. Sometimes the sensor disappeared. Sometimes it recovered.

At first, it was simply frustrating debugging.

Then I realized that this was exactly the kind of behavior embedded systems need to survive in the real world.

The interesting question was no longer just:

Did the sensor fail?

It became:

Did the firmware recognize the failure?
How quickly did it react?
Did the system remain safe?
Could it recover?

That was the point where CrashTest stopped being just a hardware demo and became a firmware resilience project.

What I learned

This project changed how I think about embedded development.

Most projects start with the question, "Can I make this system work?"

CrashTest made me ask a different question:

"Can I prove that this system fails safely?"

I learned a lot about hardware-in-the-loop testing, embedded fault handling, custom hardware interfaces, deterministic validation, serial protocol design, firmware state machines, latency measurement, fail-safe cleanup, recovery verification, and generating evidence from real hardware tests.

I also learned that reliability testing is not simply about causing a failure. The difficult part is making the fault repeatable, defining exactly what successful recovery means, and collecting enough evidence to prove what happened.

What's next

The current MVP validates a controlled sensor-failure scenario, but the architecture can expand far beyond one fault.

Future versions of CrashTest could support physical I2C SDA/SCL faults, UART failures, stuck GPIO, corrupted sensor data, delayed responses, watchdog events, brownouts, power interruptions, actuator failures, timing faults, and automated multi-fault regression campaigns.

I also want to add an engineering agent that can read the deterministic evidence produced by CrashTest, compare previous runs, identify recurring failure patterns, and recommend which fault should be tested next.

The long-term vision is simple:

Connect CrashTest to an embedded system and ask, "What happens when this device starts failing?"

Instead of finding out after deployment, CrashTest helps answer that question during development.

Build it. Break it. Prove it recovers.

Accomplishments that we're proud of

What we learned

What's next for CrashTest

Built With

Share this project:

Updates

Submission history