AI Agent Hallucination Tester

AI validation tool that identifies hallucinations, evaluates output accuracy, and generates risk insights to improve AI system reliability.

Comment

AI Agent Hallucination Tester

Inspiration

AI agents are increasingly used in enterprise workflows, but hallucinations can lead to costly mistakes such as incorrect invoice amounts, dates, or bank account details. We wanted to build a solution that helps organizations trust AI by automatically detecting these errors before they impact business operations.

What it does

AI Agent Hallucination Tester validates AI-generated invoice data against ground-truth records. It detects hallucinations, calculates risk scores, flags high-risk cases, and routes them for human review through an intuitive dashboard.

How we built it

We built AI Agent Hallucination Tester using Python, FastAPI, Google Gemini, HTML/CSS/JavaScript, and a modular testing framework. The system combines AI extraction, hallucination detection, risk scoring, analytics, reporting, and human review workflows inspired by UiPath Test Cloud and Action Center.

Challenges we ran into

The biggest challenge was accurately identifying hallucinations and assigning meaningful risk scores to different fields. We also had to balance automation with human oversight and create a workflow that mirrors enterprise testing environments.

Accomplishments that we're proud of

We successfully built a working end-to-end platform that automatically tests AI agents, detects hallucinations, prioritizes risks, and provides actionable insights through dashboards and reports.

What we learned

We learned how to evaluate AI reliability, design risk-based testing systems, implement human-in-the-loop workflows, and build scalable AI governance solutions for enterprise environments.

What's next for AI Agent Hallucination Tester

We plan to integrate fully with UiPath Test Cloud, Maestro, and Action Center, expand support for additional document types, enable real-time monitoring, and introduce advanced AI governance and compliance features.

Built With

Share this project:

Updates

posted an update —

Built the first version of AI Agent Hallucination Tester!

The platform automatically tests AI-generated invoice data against ground-truth records, detects hallucinations, calculates risk scores, and routes high-risk cases for human review.

Features completed: • AI-powered invoice validation using Google Gemini • Hallucination detection engine • Risk scoring system (0–100) • Interactive analytics dashboard • Human review workflow • Detailed reporting and insights

Currently working on deeper UiPath integration and expanding support for additional business document workflows.

Log in or sign up for Devpost to join the conversation.