Inspiration
What it does
How we built it
Challenges we ran into
Accomplishments that we're proud of
What we learned
What's next for verify.ai
Verify.ai
Inspiration
AI agents are increasingly capable of completing real customer-support tasks, including initiating refunds and other financial actions. However, a critical question remains: how does a bank know that an agent did exactly what it was authorized to do?
An agent may report that a refund succeeded even when the provider rejected it. It might refund the wrong amount, execute the same refund twice, or approve something that conflicts with the merchant's policy. Traditional monitoring often records what the agent said or attempted, but that is not the same as independently proving the final financial outcome.
We built Verify.ai to provide that missing assurance layer. The core idea is simple: never rely on the agent's claim alone. Compare it with trusted authorization, approved policy, and independently retrieved provider evidence before issuing a verdict.
What it does
Verify.ai verifies refunds performed by AI support agents. For every report, it reconciles four evidence sources:
- Trusted authorization — what the bank allowed the agent to do.
- Agent claim — what the agent says it did, treated as untrusted input.
- Provider evidence — what the payment provider independently recorded.
- Approved policy — the reviewed merchant-policy version applicable to the transaction.
We can express the decision as:
$$ V = f(A, C, R, P) $$
where $A$ is the trusted authorization, $C$ is the agent claim, $R$ is the provider record, and $P$ is the approved policy. A refund is verified only when all required controls agree:
$$ \mathrm{VERIFIED} \iff A_{\mathrm{valid}} \land (C \equiv R) \land P_{\mathrm{allow}} \land (N_{\mathrm{executions}} = 1) $$
The system returns an explainable verdict such as VERIFIED, MISMATCH, PENDING, NEEDS_REVIEW, or UNKNOWN. It also shows every deterministic control, the expected and observed values, and the evidence used.
The demo includes eight reproducible scenarios covering a correct refund, false success, wrong amount, policy contradiction, pending execution, duplicate refunds, unavailable evidence, and idempotent replay.
How we built it
The frontend is a responsive React and TypeScript dashboard built with Vite. It provides a verification simulator, evidence comparison, policy review workflow, deterministic control results, evidence export, read-only rechecking, and an append-only audit log.
The backend contains two separate FastAPI services:
- The Verifier API stores trusted authorizations, agent reports, policy versions, and verification snapshots.
- The mock payment provider maintains a separate synthetic transaction database and exposes independent read and write APIs.
The separation is intentional. A simulated agent uses a provider write credential to execute a refund, while Verify.ai uses only a read credential to retrieve evidence. The verifier never treats the agent's response as proof of success and never executes a refund during verification.
For policy discovery, Verify.ai integrates with the You.com Search and Contents APIs. You.com acts as the policy researcher: it searches allowlisted official merchant domains and retrieves public refund-policy content. The result is stored with its source URL, retrieval time, and content hash. Candidate rules remain untrusted until a human reviewer inspects, edits, and approves them.
No customer, account, payment, or refund identifiers are sent to You.com. Because the current API key requires additional credits, the local demonstration uses a clearly labeled policy fixture rather than presenting fixture content as a live result.
SQLite and SQLModel provide isolated storage for the verifier and mock provider. Verification snapshots are append-only, report submission is protected by idempotency keys, and rechecking only reads the provider again—it cannot create a second refund.
Challenges we faced
Separating claims from evidence
The most important design challenge was maintaining clear trust boundaries. An agent's statement may be useful, but it cannot be treated as authoritative evidence. We separated authorization, claim, policy, and provider data throughout the data model and interface so that a reviewer can see exactly where each fact originated.
Making uncertain outcomes explicit
Financial systems do not always return immediate or complete results. We avoided collapsing every non-success case into a generic failure. Pending provider execution returns PENDING, missing evidence returns UNKNOWN, and ambiguous or unapproved policy returns NEEDS_REVIEW.
Preventing verification from causing another transaction
Reverification needed to be demonstrably safe. We implemented it as a strictly read-only operation that appends a new snapshot. It never calls the provider's write endpoint.
Working with public policy content
Public webpages are untrusted, may change, and can contain ambiguous language. We added an official-domain allowlist, source hashing, persisted excerpts, editable candidate rules, and mandatory human approval. We also had to handle You.com's HTTP 402 billing state explicitly and provide a labeled fixture path for a reliable local demo.
Keeping the demo deterministic
A hackathon demonstration needs reproducible behavior. We created eight seeded scenarios and isolated test databases so automated testing cannot pollute the demo data. The result is a repeatable scenario matrix with predictable verdicts.
What we learned
We learned that verification for agentic systems is less about adding another AI model and more about establishing strong evidence boundaries. The agent that performs an action should not be the authority that proves the action succeeded.
We also learned that policy retrieval and policy enforcement are different responsibilities. Search can locate relevant public information, but the retrieved content should not become an active financial control without provenance, versioning, and human approval.
Finally, explainability is most useful when it is concrete. A single score is not enough for financial review. Showing the authorization, claim, provider record, policy decision, and individual controls makes the result understandable and auditable.
What's next
The current project is a local MVP using synthetic transactions. The next step is to replace the mock provider with bank-specific read-only ledger adapters, add tenant authentication and authorization, expose a trusted task-creation API, and deliver decisions through webhooks or event streams.
Future versions could also support additional transaction types, policy-change monitoring, reviewer workflows, configurable control packs, and integrations with existing fraud, compliance, and case-management systems.
Verify.ai's goal is to let autonomous agents perform useful financial work while giving institutions independent proof that every action was authorized, policy-compliant, and actually completed.
