AuditFlow

Inspiration

Audit review is often slowed not by the final judgment, but by the repetitive work before it: finding invoices, purchase orders, receipts, bank records, and ledger evidence, checking whether they agree, and documenting exceptions.

We built AuditFlow to automate that evidence-review layer while keeping the human reviewer responsible for exceptions.

What we built

AuditFlow is a five-agent AI workflow for the Office of the CFO:

Intake → Evidence → Anomaly → Decision → Workpaper

Each agent has a bounded responsibility. The system identifies the available evidence, cross-references related records, detects a focused set of audit exceptions, applies a routing policy, and produces a review-ready workpaper.

Clean cases can be auto-cleared. Cases with findings are routed to human review with the evidence and rationale needed for the next decision.

How we built it

We used AO (Agent Orchestrator) to coordinate the agent pipeline, Anthropic Claude for reasoning, and Neatlogs for LLM observability.

We built a controlled 12-case benchmark covering clean transactions, missing evidence, mismatches, duplicates, conflicting evidence, and multi-issue cases. Repeated benchmark runs helped us catch issues in routing, anomaly boundaries, missing-receipt reasoning, model selection, and observability compatibility.

Our final model strategy uses Claude Sonnet 5 for most stages and Claude Opus 5 for the more reasoning-intensive Anomaly Agent.

Challenges and lessons

The hardest part was not getting a multi-agent pipeline to run. It was making the agents reliable enough that one ambiguous piece of evidence would not silently become a wrong decision.

We found that clear agent boundaries, explicit routing rules, benchmark-driven testing, and strong observability mattered as much as the model itself.

One intentionally ambiguous benchmark case, CASE_06, still shows some variance in full-sequence LLM runs. We kept that limitation visible rather than hiding it.

What we learned

The strongest workflow was not "AI replaces the auditor."

It was:

AI handles repetitive evidence review → exceptions remain explainable → humans make the final decision.

That became the core design principle behind AuditFlow.

Built With

• anthropic-claude
• ao-agent-orchestrator
• claude-opus-5
• claude-sonnet-5
• fastapi
• neatlogs
• next.js
• python
• react
• typescript
• uvicorn
• vercel