trace-repro

Inspiration

Stack traces lie. Our checkout incident showed a 500 in capture_payment, but the same orders:8812 row had been cancelled 38 ms earlier by a background expire_cart task. Processes restart, traces are partial, and the failing test never reproduces locally. Teams burn days in war rooms treating symptoms. We wanted the opposite: a system that refuses to guess, proves the causal collision with a test, and fixes the defect graph-natively.

What it does

• Replays the production workload and ingests spans plus state mutations into a FalkorDB (:StateEntity) property graph.
• Finds cross-trace collisions: every independent trace that mutated the same entity inside the causal window, with millisecond offsets.
• Compiles a deterministic, plain-pytest reproduction and shows it failing on the production error (Pipeline #1, RED).
• Plans and applies AST repairs to every unlocked upstream writer — row locks, session isolation, idempotency guards — then proves the test passes (Pipeline #2, GREEN) and reads the before/after lock state back from the graph as governance proof.
• Ships as GitLab CI (one self-heal job per incident), JUnit + Code Quality reports, an automated MR path, a Duo Agent Platform skill and ambient flow, and a six-tool MCP server for GitLab Duo.
• Three real scenarios run end to end: checkout lost update, inventory oversell, payments double charge.

How we built it

Python engine (src/trace_repro) with a FalkorDB backend plus an in-process twin whose traversal tests assert parity. Telemetry ingestion records locks and mutations; a collision analyzer computes the causal window; a code generator emits standalone pytest; an AST repair planner chooses among five strategies and reverts cleanly. GitLab CI runs the loop per scenario with a FalkorDB service, publishes JUnit and Code Quality artifacts, and exports a patch, the compiled test and an incident report. GitLab Pages hosts the landing page and the evidence page. GitLab Duo gets a project-level Agent Skill, a flow-registry v1 ambient flow, an execution sandbox config, and the MCP stdio server.

Challenges

• Proving a race deterministically: the replay harness synchronizes the two traces on a barrier instead of sleeping, so RED is reproducible.
• Finding all offending writers, not just the one in the trace: the repair plan is derived from the graph, and the tests assert every writer is locked.
• Keeping the in-memory graph twin in exact query parity with FalkorDB so CI and sandboxes behave the same.

Accomplishments

• Three distinct concurrency incidents healed end to end in one command each.
• Duo integration that is real files, not claims: skill, flow, sandbox config,MCP config.
• A public Pages showcase with the evidence and the demo walkthrough.

What we learned

The victim is never the culprit. Once the collision is a graph query, "fix the bug" becomes "lock every writer on the entity", and the fight moves from debugging to proving.

What's next

• Ingest live OpenTelemetry streams instead of replay-only.
• Widen the repair library to more ORMs and languages.
• Turn the governance proof into a merge gate that blocks on unlocked writers.

Built With

• docker
• falkordb
• fastapi
• gitlab-ci
• gitlab-duo
• gitlab-pages
• mcp
• pytest
• python
• sqlalchemy