BREAKPOINT

Inspiration

What it does

How we built it

Challenges we ran into

Accomplishments that we're proud of

What we learned

What's next for BREAKPOINT

Inspiration

Distributed systems rarely fail in the clean, isolated ways described in runbooks. The difficult incidents come from interacting dependencies, stale assumptions, and diagnostic checklists that keep asking low-value questions while the system changes underneath them. We wanted an agent that does not merely summarize telemetry or inject faults at random, but actively decides which safe experiment is worth running next.

What it does

BREAKPOINT is a self-calibrating active reliability agent for distributed systems. It starts from incident symptoms and a probability distribution over plausible root services. NVIDIA Nemotron, served through Nebius Token Factory, acts as a grounded Probe Architect: it turns topology and symptoms into hypotheses and a shortlist drawn only from a safety-approved probe registry.

A deterministic Bayesian planner then chooses the next probe by expected reduction in classification error. The probe executes in a constrained environment, the observation updates the posterior, and the loop continues until confidence is sufficient or the probe budget is exhausted. A separate Model Skeptic monitors verified post-incident errors for drift and triggers recalibration when yesterday's reliability model no longer matches current behavior.

Nemotron never receives authority to execute arbitrary shell commands. It reasons over approved tools; BREAKPOINT validates the shortlist and retains deterministic control over what can actually run.

How we built it

The core is a Python Bayesian decision engine. For every candidate probe, BREAKPOINT computes the expected posterior accuracy and selects the experiment with the highest expected decision value under the current budget. We added a drift detector that watches verified diagnostic error and activates recalibration only when the observed error becomes statistically inconsistent with the calibrated model.

The AI layer is intentionally separated from the safety-critical selector. The Nebius Token Factory client uses the OpenAI-compatible chat-completions API with nvidia/Nemotron-3_5-Lightning. Nemotron receives structured incident context and may propose only probe IDs already present in the registry. Invalid or out-of-registry proposals are rejected. If Token Factory is unavailable, the demo falls back to deterministic behavior rather than granting the language model more authority.

For reproducibility we built local loopback-only HTTP fault labs, automated tests, synthetic drift studies, and external validation against RCAEval-derived microservice datasets.

Evidence

In a controlled synthetic drift study across 20 seeds and 3,000 incidents per seed, the adaptive planner recovered post-drift accuracy from 46.82% for a frozen model to 73.09%. The detector identified 20/20 shifts with zero false alarms in that experiment. Against periodic recalibration every 100 incidents, the advantage was concentrated immediately after the shift rather than in long-run steady state.

On an RCAEval Online Boutique development benchmark with a three-probe budget, active root-service localization reached 66.7%, compared with 56.7% for a learned static order and 50.7% mean across fixed orders. Because we inspected that corpus during development, we do not present it as pristine confirmation.

We then froze the configuration before evaluating RCAEval Train Ticket. On 30 held-out cases, BREAKPOINT reached 73.3% (22/30) root-service localization versus 60.0% (18/30) for the learned static order. The exact mean across all 120 fixed orders was 53.7%, and even the retrospectively best fixed order reached 70.0%. The exact two-sided McNemar p-value is 0.125, so the sample is too small for a definitive significance claim. We treat the result as promising, not conclusive.

Challenges we ran into

The biggest challenge was avoiding a benchmark that accidentally favored our own model. Early synthetic scenarios made information gain and simpler heuristics look artificially similar. We therefore added paired evaluation, model-misspecification tests, explicit drift, a periodic-recalibration baseline, and finally a second external corpus whose configuration was frozen before evaluation.

Another challenge was deciding how much authority to give the language model. Giving an LLM direct fault-injection or shell authority would make the demo easier but the system less trustworthy. We chose a narrower and inspectable role: hypothesis formation and safe-probe shortlisting, with deterministic execution control.

What we learned

The useful advantage is not that one statistical rule wins forever. Fixed and periodically recalibrated systems eventually converge. BREAKPOINT matters most in the window immediately after the system changes, when stale diagnostic assumptions are most dangerous. We also learned that external validation can change the product direction: service-level root-cause localization generalized better than trying to predict fine-grained fault type.

What's next

Next we are running the live Nemotron path on Nebius Token Factory, publishing the reproducible benchmark suite and public demo, and expanding confirmatory evaluation to more independent microservice systems. The long-term goal is to turn BREAKPOINT into an active reliability layer that continuously asks the cheapest, safest, highest-value diagnostic question before an engineer has to run the entire checklist.

Built With

• python