Themis

DataHub X OpenTelemetry. A green AI run can still be wrong. DataHub explains the data. Replay checks the answer. OpenTelemetry proves what happened. Themis decides if the release can move.

Comment

About the project

Inspiration

AI agents can produce valid SQL and still return the wrong business answer.

An agent may use a deprecated field, the wrong dataset, or an old business definition. Normal CI can check whether the code runs. It cannot prove that the answer still has the right meaning.

Themis combines DataHub context, real-data replay, independent correctness checks, and runtime evidence before an AI release moves forward.

What it does

Themis is a release gate for AI-generated analytics.

It reads DataHub through MCP. It selects relevant replay cases. It runs the baseline, candidate, and independent oracle against the same verified dataset. It records OpenTelemetry evidence and returns:

  • PROMOTE
  • REVIEW
  • QUARANTINE

Themis writes a redacted receipt back to DataHub. An Analytics Agent can propose SQL, but Themis owns the release decision.

Themis also can be more than this such as:

  • Explain agent behavior
  • Investigate an agent failure
  • Coordinate agent repair
  • Manage agent trust over time
  • Semantic contract checker
  • and many more

How we built it

We built Themis with:

  • Python and FastAPI
  • DataHub OSS and the DataHub MCP Server
  • DuckDB for read-only SQL replay
  • OpenTelemetry and Jaeger for runtime evidence
  • Official NYC TLC Parquet data
  • YAML contracts for scenarios, SQL, oracles, and policies
  • A vanilla JavaScript workspace
  • Optional HTTP, stdio, and native DataHub Analytics Agent integrations

Challenges we ran into

The hardest part was keeping every evidence source honest.

DataHub proves what the data means. OpenTelemetry proves what the system did. The oracle checks correctness. None of these signals can replace the others.

We also had to handle missing query usage, local DataHub feature limits, agent failures, write-back failures, and human repair states without turning them into false green results.

Accomplishments that we are proud of

Themis now demonstrates all three release outcomes:

  • A repaired candidate reaches PROMOTE.
  • A deprecated-field candidate reaches QUARANTINE.
  • A DataHub-disabled run reaches REVIEW.

The project also includes:

  • Real DataHub OSS and MCP evidence
  • Independent oracle evaluation
  • Verified receipt write-back
  • Real OpenTelemetry trace IDs
  • A receipt-backed evidence graph
  • A human-approved repair flow
  • A bounded Analytics Agent evidence integration

What we learned

Metadata alone does not prove that an AI answer is correct.

A trace proves that work ran. It does not prove that the answer was right. A passing SQL query does not prove that the query used the correct business meaning.

We learned that missing evidence must be visible. An AI agent should propose work, but a deterministic gate and a human boundary should control the release.

What is next for Themis?

Next, we want to support:

  • Generic model and agent release evaluation
  • RAG context and tool-calling agent checks
  • Agent Registry impact fan-out
  • Stronger evidence confidence and freshness checks
  • Human-approved repair-to-PR workflows
  • Native DataHub Assertions, Contracts, and ML lineage
  • Canary release and rollback support

These are future capabilities. The current project does not claim automatic deployment, rollback, or pull request creation.

Built With

Share this project:

Updates