Traceback

Inspiration

When a request fails across several services, the error shows up in one place and the cause is in another. Pasting logs into a chatbot gives a fluent guess with nothing to verify. We wanted an investigator whose every claim points to a real record, and whose conclusion is tested by asking: if this were the cause, would switching it off make the failure go away?

What it does

Traceback takes the logs, spans, metrics and deploy events from a failed multi-service request and:

1. aligns the services' clocks (estimated only from causal span pairs, so clock skew can't scramble the timeline),
2. correlates spans into traces and builds the dependency graph,
3. produces ranked candidate causes with the evidence records behind each (deterministic rules, no model),
4. asks NVIDIA Nemotron on Nebius Token Factory to rank, explain and check those candidates, outputting a causal chain with a confidence label and confirmed vs uncertain per claim, where every claim must cite real record IDs (unknown IDs are rejected and the model is asked again),
5. replays the incident with the suspected cause switched off and reports whether the symptom disappears, which can lower confidence but never changes the answer.

It is not a log summarizer: the analysis layer works without any model, and a baseline that feeds raw logs to the same model is measured against it.

How we built it

C++20 core (simulator, clock estimator, correlator, graph, rules, libcurl Token Factory client, verifier), CMake, a vendored single-header JSON parser, Python (standard library) for the evaluation harness, and a static, dependency-free web UI. Evaluation uses a deterministic simulator of a four-service backend with background noise, lost lines, clock skew, misleading symptoms and decoy faults, a held-out test split locked before any model ran on it, and later pre-registered tests on public real-telemetry benchmarks (RCAEval) and the OpenTelemetry Demo. Scoring rules were committed before each run.

Results (every number is regenerated from saved runs)

Simulators we wrote (valid for those simulators only). On the first locked test set (26 incidents) the accuracy gaps are within one incident; what holds up is calibration and cost: Full Traceback was never confidently wrong (raw-logs baseline 7.7%) at about 1/13 of the cost. On two later, harder locked sets (48 incidents across two simulated backends) Full Traceback finds the cause in 95.8% against 68.8% for the raw-logs baseline, with 0% confidently wrong against 29.2%, at about a tenth of the cost.

Real telemetry (public RCAEval benchmark and the OpenTelemetry Demo; injected faults on demo applications, so not production).

• Our hand-written analyzer does not generalize: 0 of 30 on RCAEval Online Boutique, where the raw-logs baseline gets 16.
• Letting the model propose causes and checking them against measured evidence ("hybrid") helped on code faults in unseen systems: baseline 14 of 60, hybrid 22 of 60, with confident wrong answers cut from 19 to 1. With the larger Nemotron Ultra model: 19 vs 38 of 60.
• Resource faults are hard for the model (baseline 2 of 90, hybrid 5 of 90), but a simple per-service metric rule with no model gets 62 of 90 on Online Boutique and 35 of 90 on Train Ticket, with weak network-fault detection and false alarms on 20 of 30 Train Ticket clean windows.
• We tried combining the two. It did not help (18 of 60 combined vs 22 of 60 hybrid alone), and we report that too.
• The replay verifier needs the simulator, so on real data it checks nothing.

Every wrong answer is listed in the repo.

Generalization check on a second simulated topology

An event-driven fan-out topology with a message broker (14 incidents), analyzer and prompts frozen. Accuracy dropped for every system; Full Traceback ties the raw-logs baseline at 78.6%. With the replay verifier 0% of answers were confidently wrong versus 21.4% for the baseline, at about 1/13 of the cost, and every cited ID was valid.

Challenges

• Models invent IDs and latch onto the first error they see: we added hard citation validation with retry, and a symptom-onset field that ignores isolated early noise.
• Short incidents were invisible to a whole-window failure rate in the verifier; we switched to a peak 30-second rate.
• Keeping evaluation honest: we found and disclosed a simulator bug, relocked before any test scoring, and report a tool crash and its fix.

What we learned

Deterministic structure plus a model that is forced to cite evidence and a replay that checks it gives answers you can verify and that are rarely confidently wrong, at a fraction of the cost of reading raw telemetry. Accuracy on a simulator is not accuracy on real systems: hand-written rules did not transfer, model proposals checked against measured evidence did better, and a larger model helped most.

What's next

Better network delay/loss detection on real data, a validated way to choose between model proposals and metric rules, adapters for real log schemas, and replay against canary/feature-flag rollbacks.

Built With

• c++20
• cmake
• css
• github
• html
• javascript
• kokoro-tts
• libcurl
• nebius-token-factory
• nemotron-3-super
• nemotron-3-ultra
• nlohmann/json
• nvidia-nemotron
• opentelemetry
• python
• rcaeval