HG Agent Reliability Lab

Inspiration

AI agents are increasingly responsible for completing complex workflows, but a successful API call does not necessarily mean the agent completed its task correctly.

We built HG Agent Reliability Lab to make agent failures visible, measurable, and easier to diagnose.

What it does

HG Agent Reliability Lab helps developers evaluate how AI agents respond to tool failures, unavailable information, and unreliable cached data.

The interactive Replay Studio provides five reproducible scenarios, allowing users to compare safe and unsafe recovery decisions and inspect structured evaluation results.

The project distinguishes technical execution success from actual task reliability.

How we built it

We developed the evaluation framework using Python and built the interactive demonstration with HTML, CSS, and JavaScript.

We integrated NVIDIA Nemotron 3 Nano 30B A3B through Nebius Token Factory and verified the connection using GitHub Actions.

Our authenticated tests demonstrated successful model connectivity and identified a case where an API request completed successfully but returned no visible answer after exhausting its configured output-token budget.

Challenges we faced

One major challenge was distinguishing a successful API response from a correct agent decision.

We also investigated how limited completion-token budgets can affect reasoning-model output and why unverified cached information must not be presented as trustworthy evidence.

Accomplishments

• Built an independent, open-source agent reliability evaluation toolkit.
• Created five reproducible failure scenarios.
• Developed an interactive demonstration of recovery decisions.
• Implemented structured scoring and regression tests.
• Successfully connected NVIDIA Nemotron through Nebius Token Factory.
• Documented real model usage and limitations transparently.

What we learned

Reliable AI systems need more than successful API calls. They require validated outputs, transparent decision policies, reproducible evaluations, and clear reporting when a model cannot complete its task.

What's next

We plan to expand live-model evaluations, improve structured-output reliability, and test agent recovery across more complex multi-step workflows.

The current public Replay Studio uses simulated decisions. Nemotron connectivity has been verified, while successful end-to-end recovery remains a future evaluation objective.nspiration

What it does

How we built it

Challenges we ran into

Accomplishments that we're proud of

What we learned

What's next for HG Agent Reliability Lab

Built With

• css3
• github-actions
• html5
• javascript
• nebius-token-factory
• nvidia-nemotron
• python
• vercel