VeriLoop

Report Page
Evaluation Page
Login and sign up page
Regression Firewall Page
Drift Monitoring Page
Agents Page
Experiments Page
Dashboard Page
Failures Page
Evidence Page
Settings Page
Home page

Inspiration

AI agents are becoming increasingly capable, but building an AI agent is only half the challenge. The bigger question is: How do we know that an AI agent is actually reliable?

An agent may provide incorrect information, hallucinate answers, misuse tools, fail to retrieve relevant knowledge, or behave unpredictably when given unexpected inputs. A successful demo does not necessarily mean an agent is ready for real-world use.

We wanted to address this gap by creating a platform that doesn't just test AI agents but also helps developers understand why they fail, improve their behaviour, and verify whether those improvements actually work.

That idea led to VeriLoop — The AI Engineer That Tests Your AI Engineer.

What it does

VeriLoop is an AI Agent Reliability Engineering platform designed to evaluate, diagnose, and improve existing AI agents through a continuous testing and verification workflow.

Instead of relying on a few manual test cases, VeriLoop provides a structured approach to discovering and resolving agent failures.

Key capabilities include:

Agent Evaluation: Generate and execute test scenarios to assess an AI agent's reliability.
Failure Detection: Identify problems such as hallucinations, incorrect tool usage, retrieval failures, policy violations, missing context, and ungrounded responses.
Root Cause Analysis: Investigate why an agent failed and identify potential causes using available evidence.
AI-Powered Interventions: Recommend targeted improvements across prompts, retrieval, tools, guardrails, memory, and workflows.
Candidate Comparison: Compare the original agent with an improved candidate to measure performance changes.
Holdout Validation: Evaluate improvements on previously unseen test cases to check whether they generalize beyond the optimization dataset.
Regression Firewall: Detect and block changes that introduce new failures or reduce reliability.
Verification Reports: Generate structured reports containing evaluation results, detected failures, comparisons, and verification outcomes.

The complete workflow follows:

Connect Agent → Generate Tests → Execute → Detect Failures → Diagnose → Improve → Validate → Verify.

Our main differentiator is the closed-loop approach: VeriLoop doesn't stop at identifying a failure; it helps developers investigate, address, and verify it.

How we built it

We built VeriLoop as an AI-powered platform combining agent evaluation, reasoning, external evidence, and a structured reliability-testing pipeline.

Our implementation uses three important AI services:

Nebius Token Factory: Used as the inference infrastructure to power AI operations and model requests.
NVIDIA Nemotron: We use nvidia/llama-3.1-nemotron-70b-instruct for reasoning-intensive tasks, including test generation, failure analysis, and evaluation of candidate improvements.
Tavily: Integrated to retrieve external information and supporting evidence when a failure investigation requires additional context or current information.

The platform organizes evaluation into three stages:

Optimization (60%): Used to identify failures and develop targeted improvements.
Validation (20%): Used to assess candidate improvements during development.
Holdout (20%): Reserved for final evaluation to determine whether improvements generalize to unseen scenarios.

This separation helps reduce overfitting and prevents us from treating improvements on familiar test cases as proof of overall reliability.

We also developed dedicated modules for agent management, evaluations, failure analysis, experiments, evidence, regression testing, and verification reports.

Challenges we ran into

One of our biggest challenges was designing a system that could evaluate AI agents beyond simply checking whether their responses were correct.

Some important challenges included:

Non-deterministic behaviour: AI agents can produce different responses to similar inputs, making consistent evaluation difficult.
Root cause identification: Detecting an incorrect response is easier than understanding whether it originated from the prompt, retrieval system, tool selection, missing context, or another component.
Meaningful evaluation: We needed to distinguish actual agent failures from infrastructure errors and unsuccessful requests.
Reliable improvements: A change that improves one test case can unintentionally damage other capabilities.
Holdout integrity: Keeping unseen evaluation cases separate from optimization was important to make the final results meaningful.
Integration complexity: Connecting inference, reasoning, and external evidence into a unified workflow required careful coordination.

These challenges pushed us to focus on reproducibility, evidence-based diagnosis, controlled interventions, and independent validation.

Accomplishments that we're proud of

We are proud of turning an idea into a working AI agent reliability platform with an end-to-end evaluation and improvement workflow.

Some of our key accomplishments include:

Building a unified platform dedicated to testing and improving AI agents.
Integrating Nebius Token Factory with NVIDIA Nemotron for AI-powered evaluation and reasoning.
Implementing test generation, failure detection, and root cause analysis workflows.
Developing a structured optimization, validation, and holdout evaluation approach.
Creating candidate comparison and regression-checking capabilities.
Integrating Tavily for external evidence retrieval.
Building dedicated dashboards and verification reports to make evaluation results easier to understand.

Most importantly, we developed VeriLoop around a practical principle: An AI agent should not be considered reliable just because it works once. It should demonstrate reliable behaviour across different scenarios, including scenarios it has never encountered before.

What we learned

Throughout this project, we learned that developing reliable AI systems requires much more than connecting an application to a powerful language model.

We gained practical experience in AI inference, model integration, agent evaluation, test generation, failure analysis, external grounding, and building structured validation pipelines.

We also learned the importance of separating optimization data from holdout data, measuring improvements objectively, and ensuring that a proposed fix does not introduce new failures.

Working with Nebius Token Factory and NVIDIA Nemotron helped us understand how inference infrastructure and reasoning-focused models can work together in a practical AI engineering application.

One of our biggest takeaways was that testing an AI agent is not a one-time activity; reliability must be continuously measured, improved, and verified.

What's next for VeriLoop

We see VeriLoop as a foundation for a more comprehensive AI Agent Reliability Engineering ecosystem.

Our future plans include:

GitHub Integration: Allow developers to connect repositories and evaluate agent implementations within their development workflow.
More Agent Frameworks: Expand compatibility with popular agent frameworks, tools, and deployment environments.
Automated Remediation: Introduce more advanced, approval-based fixes with version control and rollback support.
Continuous Monitoring: Monitor agent behaviour over time and detect performance degradation or behavioural drift.
CI/CD Integration: Enable automated reliability checks before an agent update reaches production.
Advanced Security Testing: Expand testing for prompt injection, unsafe tool execution, data leakage, and other agent-specific security risks.
Enterprise Readiness: Add team collaboration, access controls, audit trails, and scalable evaluation workflows.