Autonomous Reliability Lab

Overview

Autonomous Reliability Lab is a reproducible reliability-testing system for tool-using AI agents. It searches for failures that happy-path testing can miss by running an agent through controlled variations of its environment, recording its tool trajectory, explaining observed failures, testing candidate repairs, and preserving successful repairs as regression evidence.

The project uses NVIDIA Nemotron 3 Super through Nebius Token Factory as the live model-driven agent. A local evidence pipeline and browser UI make the experiment inspectable instead of presenting only a single model response.

The problem

An agent can make a reasonable decision based on what it observed and still take an unsafe action if reality changes before it acts. Agents can also repeat actions that have already succeeded. These failures are difficult to catch with a single successful demo or a static test case.

How it works

The workflow is:

1. Discover — run controlled scenario variations and record agent/tool outcomes.
2. Explain — inspect the trace and identify a causal hypothesis for each failure.
3. Repair — propose an intervention that supplies fresh evidence before a consequential action.
4. Preserve — replay the repaired path, verify the result, and retain regression evidence with provenance.
5. Validate — evaluate the live model trajectory and clearly distinguish an exercised repair from an inconclusive run.

Recorded experiment

The saved discovery sweep evaluated 10 scenarios: 8 succeeded, 1 produced an unsafe action, and 1 degraded.

• Failure A — stale observation: the agent observed billing as ACTIVE, while the environment had changed to CANCELLED by step 4; the attempted cancellation produced an unsafe-action outcome.
• Failure B — redundant action: the recorded failure is classified under the redundant-action hypothesis.

The candidate repair uses a required_observation intervention to obtain fresh get_billing_status evidence before cancel_subscription. In offline replay, the repaired run succeeded, verification passed, and regression validation passed.

Honest live-model result

The saved live Nemotron counterfactual stopped after get_customer and never reached the repair intervention. Its status is INCONCLUSIVE, not a failed repair and not a validated live repair. Three supplemental runs of the same setup were also inconclusive because the intervention was not reached.

This distinction is central to the project: the system should report what the evidence demonstrates, and should not label an unexercised intervention as validated.

Technology

• NVIDIA Nemotron 3 Super — live model-driven agent
• Nebius Token Factory — hosted inference API
• Python — orchestration, scenario execution, diagnostics, counterfactuals and regression pipeline
• Deterministic environment/replay components and structured tool traces
• Pytest — automated verification
• HTML, CSS and JavaScript — evidence console

Running the project

See the repository README for prerequisites, environment configuration, test execution, report generation and local demo instructions. API credentials must be supplied through environment variables and must never be committed.

The recorded evidence and generated report can be inspected locally without rerunning paid model calls. The live counterfactual and replication outcomes described here are the saved results of the recorded runs.

Built With

• agent-reliability
• ai-agents
• causal-analysis
• counterfactual-testing
• css
• html
• javascript
• nebius-token-factory
• nvidia-nemotron
• pytest
• python
• regression-testing
• reliability-engineering