Inspiration

CrashCart was inspired by a common problem in distributed GPU training: the loudest log message is often only a symptom. A rank may report a collective timeout while the earlier clue is buried in a kernel log or visible only in telemetry. I wanted to make it easier to connect those clues and show exactly where a diagnosis came from.

What I built

CrashCart is a Python and FastAPI app for investigating multi-node GPU training incidents. It ingests rank logs, kernel logs, scheduler data, configuration, and a telemetry dashboard. Parallel workers scan log chunks, findings are grouped, and the final diagnosis is checked against its cited source lines. The app can also produce a postmortem and a dry-run runbook for human review.

The project is configured to use NVIDIA Nemotron models through Nebius Token Factory: a swarm model for log scanning and a reduce model for the final diagnosis. A dashboard-reading model role is also supported, though its provider model ID still needs live verification. The included incidents are synthetic, and the public demo runs in mock mode without making model API calls.

What I learned

I learned how much an incident diagnosis depends on preserving evidence details: the source file, line number, and quoted text need to survive every step from ingestion through the final report. I also learned to keep measured results separate from design goals. Until a live evaluation is run, accuracy, latency, and cost comparisons should be reported as not yet measured.

Challenges

The main challenge was making the synthetic incidents realistic enough to distinguish root causes from downstream symptoms. For example, many healthy ranks can end with similar timeout messages even though only one earlier log contains the useful clue. Dashboard-only faults add another challenge because the logs alone do not identify the cause.

I addressed these challenges with deterministic synthetic bundles, source-grounded citation checks, and an offline mock mode for development. The remaining work is to verify the live model configuration and run an evaluation before making performance claims.