Veritas

Veritas uses a multi-agent AI pipeline to catch data leakage, validate statistical claims, and generate evidence-scored audit reports — so you can trust your data before you trust your model.

Comment

Inspiration

We kept seeing the same failure pattern: a promising dataset, an exciting model result, and then — months later — someone discovers the "signal" was actually data leakage, or the statistical claim behind a decision didn't hold up under scrutiny. Bad data doesn't just produce bad models; it produces bad decisions that ripple outward. We wanted a tool that catches these problems before a model or a claim ever gets trusted, not after.

What it does

Veritas is an AI-powered data integrity auditor. You feed it a dataset (and optionally the claims or analysis built on top of it), and a pipeline of specialized AI agents goes to work:

  • A planner agent maps out what needs to be checked
  • An interpreter agent reads the data and the claims being made about it
  • A critic agent stress-tests those claims against statistical evidence (confidence intervals, effect sizes, feature importance)
  • A consensus agent reconciles findings across the pipeline into a final verdict

Along the way, Veritas actively screens for data leakage, runs validation and preprocessing checks, and scores every finding with supporting evidence — so instead of a black-box "trust me," you get a transparent, evidence-backed audit report you can actually act on.

How we built it

Veritas is built as a Python backend with a Streamlit front end. The backend is organized around a multi-agent architecture: each agent (planner, interpreter, critic, consensus) has a narrow responsibility and hands off structured findings to the next stage. Underneath, a statistics and analysis layer computes confidence intervals, effect sizes, and feature importance, while a dedicated data module handles leakage detection and dataset auditing. An evidence scorer aggregates everything into a confidence-weighted report, which is rendered through report-generation and UI components (cards, charts, tables) in the Streamlit app.

Challenges we ran into

[e.g., "Getting the agents to disagree productively instead of just rubber-stamping each other's output was harder than expected — we had to design the critic agent to actively look for counter-evidence rather than confirm the interpreter's read." Or note the .env/secret scanning issue, LLM prompt tuning, handling messy real-world CSVs, etc.]

Accomplishments that we're proud of

  • A working multi-agent pipeline that produces evidence-scored, explainable audit reports rather than an opaque "risk score"
  • Real data leakage detection integrated directly into the pipeline, not bolted on afterward
  • An end-to-end demo (dataset in, audit report out) that runs on real-world-style data like our student performance sample

What we learned

[e.g., how to coordinate multiple LLM agents with distinct roles, the practical challenges of statistical validation on real messy data, lessons about handling secrets/config safely in a fast-moving hackathon codebase]

What's next for Veritas

  • Expanding the leakage and validation checks to cover more data types and domains beyond tabular data
  • Adding support for auditing existing ML pipelines/models directly, not just datasets
  • Building out collaborative review, so teams can annotate and resolve audit findings together
  • Polishing the evidence-scoring methodology and publishing it openly so the audit criteria are as transparent as the reports themselves

Built With

Share this project:

Updates

Submission history