The problem
Imagine a release manager is seconds away from publishing a customer-facing update. They ask an AI agent whether the available evidence authorizes release. The agent gives a confident answer — but the team cannot see which evidence it used, which tools it called, what failed, what it retried, or whether it stayed inside policy.
For consequential agentic workflows, a good-looking final answer is not enough. Teams need to know what the agent actually did before they trust the outcome.
Judge it in 30 seconds
- Give the agent the bounded release-authorization task and supplied evidence.
- Run the workflow on NVIDIA Nemotron 3 Nano through Nebius Token Factory.
- Watch the agent use only the declared evidence_search tool and fail closed when the supplied policy does not authorize public release.
- Open the reliability record and inspect the model calls, tool call, evidence use, retries, finish reasons, latency, token usage, and final outcome.
The key idea is visible immediately: the answer is not the only product output; the execution evidence is.
The solution
Agent Reliability Lab turns the agent black box into an inspectable workflow. A bounded task runs through NVIDIA Nemotron 3 Nano on Nebius Token Factory, with only declared tools available. The user receives both the answer and a reliability record showing the execution behind it.
The judge path is intentionally simple:
- Give the agent a bounded task and supplied evidence.
- Run the workflow.
- See the final answer and PASS / fail-closed outcome.
- Inspect model calls, tool calls, retries, and token usage.
- Review the human-readable event timeline and raw structured evidence.
In the release-authorization demonstration, the supplied policy evidence says private testing is approved but public publication requires separate approval. The agent must search the supplied evidence and refuse to invent authorization that is not there.
Public live proof
The public judge build is deployed on Vercel and calls Nebius Token Factory server-side with NVIDIA Nemotron 3 Nano. A fresh public end-to-end verification returned HTTP 200 and PASS with two real Nebius/Nemotron model calls, exactly one allowlisted evidence-search request/response, zero retries, 667 total tokens, and a seven-event structured reliability trace. The final answer correctly concluded: “Public release is not authorized based on the supplied evidence.”
The trace included provider identity, model identity, response IDs, finish reasons, per-call latency, prompt/completion/total token accounting, the tool request and response, the final answer, and the terminal PASS state. No credential is exposed to the browser.
Why it is different
Most agent observability tells you something went wrong after the fact. Agent Reliability Lab makes reliability evidence part of the product itself — before a consequential decision is trusted.
The bounded runtime enforces:
- strict machine-readable action/final contracts
- allowlisted tool execution
- hard step ceilings
- bounded retry policy
- undeclared-tool rejection
- fail-closed malformed-output handling
- structured model/tool/evidence events
- provider/model/response/finish-reason/latency/token evidence
- server-side credential handling
- reproducible run digests
If the provider, model, tool, or contract fails, the workflow surfaces the failure instead of manufacturing success.
Nebius + NVIDIA
The competition runtime uses Nebius Token Factory with NVIDIA Nemotron 3 Nano (nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B).
Nemotron is the bounded reasoning/planning layer: it decides whether to request an allowed evidence tool or emit a final answer. Nebius provides the inference runtime. Provider metadata, response identifiers, finish reasons, latency, and token usage become part of the reliability evidence while credentials remain server-side.
Nebius + Nemotron are structurally important to the project rather than decorative integration: the model drives the action/final loop, and provider-level runtime evidence becomes part of the reliability record.
Who it is for
The immediate audience is developers and teams deploying AI agents that use tools or make consequential recommendations: release and compliance workflows, operations, support automation, financial or procurement review, internal approvals, and other places where someone eventually has to answer:
What did the agent actually do, what evidence did it use, and did it stay inside its boundaries?
Potential impact
The same reliability pattern can be reused wherever a tool-using agent needs an inspectable record before a human trusts the result, including:
- software release authorization
- compliance and policy review
- procurement or vendor approval
- customer-support escalation
- financial or operational review
- internal approval workflows
The provider/tool adapter boundaries are intentionally replaceable so the reliability architecture can outlive any single model or vendor.
Known limitations and failure cases
The limitations are explicit by design:
- The public demo searches only supplied local evidence; it is not an open-web research agent.
- The public path is intentionally single-model and single-tool so the reliability mechanics remain easy to inspect.
- Provider failure, malformed model output, undeclared tool requests, contract violations, and boundary overruns fail closed rather than silently falling back.
- Conservative execution and rate ceilings make this a bounded demonstration, not a high-throughput production service.
- A trace does not by itself prove an AI decision is correct, safe, or compliant in every domain.
- Human operators still own consequential decisions; the system makes the agent's process inspectable rather than removing accountability.
The design thesis is simple: a trustworthy agent system should make its boundaries and failures as inspectable as its successes.
Engineering proof
The public repository is Apache-2.0 licensed and includes the released source, setup instructions, deployment configuration, release checksums, and 28/28 deterministic tests passing across the bounded agent core and judge-facing public UI.
Agent Reliability Lab is the product. Nebius Token Factory and NVIDIA Nemotron are the current competition runtime and model; provider/tool adapters remain replaceable so the reliability architecture is not locked to one vendor.
Built With
- apache-2.0
- css
- html
- javascript
- nebius-token-factory
- nvidia-nemotron-3-nano
- openai-compatible-api
- python
- structured-event-tracing
- vercel
