What it is
Agent Reliability Lab is a bounded, evidence-first reliability layer for AI agents that use tools. It makes the execution path visible: which model ran, which allowlisted tool was requested, what evidence came back, what failed, what was retried, how many tokens were consumed, and whether the workflow stayed inside its declared boundaries.
The project addresses a practical problem with autonomous workflows: a confident final answer is not enough when a developer later needs to understand what the agent actually did.
How it works
A task enters a bounded agent loop powered by NVIDIA Nemotron through Nebius Token Factory. The runtime enforces:
strict machine-readable action/final contracts; tool allowlists; hard step ceilings; bounded retry policy; fail-closed malformed-output handling; undeclared-tool rejection; structured evidence events; provider/model/response/finish-reason/latency/token evidence; token accounting; runtime-only credential handling; reproducible run digests; local evidence search for grounded execution.
The browser interface presents the final answer together with model-call count, tool-call count, retries, token usage, reliability checks, and the underlying event trace.
Nebius + NVIDIA
The hackathon runtime uses Nebius Token Factory for inference and NVIDIA Nemotron 3 Nano:
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B
Nemotron acts as the bounded planning/reasoning layer: it decides whether to request an allowed tool or emit a final answer. Nebius supplies the inference runtime. Model metadata, response identifiers, finish reasons, latency, and token usage are recorded as reliability evidence without placing provider credentials in the browser evidence trail.
A certified private end-to-end canary completed with two Nebius/Nemotron model calls, exactly one allowlisted local evidence_search request/response, zero retries, positive token accounting, and a contract-conforming final answer. The public release preserves that bounded core and adds a judge-facing reliability UI plus abuse controls.
Product design
The intended judge path is deliberately short:
Enter a bounded task and local evidence. Run the agent. See the final answer. Inspect model calls, tool calls, retries, and tokens. Review the human-readable timeline and raw structured evidence. See violations fail closed rather than being hidden.
The public demo intentionally limits the tool surface to supplied local evidence. Tavily is not part of the demonstrated runtime.
Why it matters
Agent Reliability Lab turns reliability from an invisible implementation detail into an observable workflow artifact. That can support debugging, regression testing, evaluation, human review, and safer deployment of agentic systems.
The immediate audience is developers and teams building AI agents that use external tools. The broader pattern applies anywhere someone must answer: What did the agent actually do? What evidence did it use? What failed? What did it retry? Did it stay inside its boundaries?
Provider independence
Agent Reliability Lab is the product. Nebius Token Factory and NVIDIA Nemotron are the current hackathon runtime/model assignments. Model and tool adapter boundaries are kept replaceable so the reliability architecture is not defined by one provider.
Open source
The public repository is Apache-2.0 licensed and includes source, setup instructions, tests, deployment configuration, and release checksums.
Log in or sign up for Devpost to join the conversation.