TrustSQL

Inspiration

It started with a number that looked right.

A revenue chart went to leadership. The SQL was green. The dashboard was beautiful. Nobody questioned it — until someone noticed that "average monthly revenue" was actually averaging line-item prices, not monthly totals. The join used o.id instead of o.order_id. A filter for completed orders was missing entirely.

The query didn't fail. It succeeded at being wrong.

That moment crystallized our thesis for the Emergence AI × Nebius Token Factory Hackathon:

AI is getting very good at writing SQL.
Nobody is verifying whether the answer deserves trust.

We set out to build the missing layer between "query runs" and "number is true" — not another SQL generator, but a truth layer for analytics.

What it does

TrustSQL is an AI agent that verifies whether a SQL query's result can be trusted — not just whether it executes without syntax errors.

You give it:

A business question ("What was average monthly revenue from active customers in 2025?")
A SQL query (pasted or generated via CRAFT)

It returns:

A weighted confidence score across 7 dimensions
Verification cards explaining what passed, warned, or failed
Live evidence from validation queries run against real data
A corrected SQL suggestion when issues are found
Policy compliance checks against your org's governance rules

Two input modes:

Paste your own SQL — sanity-check analyst-written queries before they ship
Generate with CRAFT — generate SQL, then immediately stress-test it

The confidence score is computed as:

C = \sum_{i=1}^{7} w_i \cdot s_i

where each dimension (s_i \in [0,100]) covers intent alignment, join correctness, aggregation logic, business rules, policy compliance, data quality, and performance.

\text{label}(C)=
\begin{cases}
\text{High Confidence} & C \ge 80 \\
\text{Moderate Confidence} & 50 \le C < 80 \\
\text{Low Confidence} & C < 50
\end{cases}

CRAFT handles the what (governed data). Nebius handles the why (reasoning over evidence).

How we built it

Architecture

flowchart LR
    A[Business Question + SQL] --> B[Intent Analyzer]
    B --> C[SQL Analyzer]
    C --> D[Alignment Checker]
    D --> E[Validation Runner]
    E --> F[CRAFT MCP]
    F --> G[Live Snowflake Data]
    G --> E
    E --> H[Report Builder]
    H --> I[Nebius LLM]
    I --> J[Confidence Score + Corrected SQL]

The 9-phase agent pipeline

Phase	What happens
1	Resolve business terms via CRAFT `resolve_term`
2	Extract structured intent (metric, grain, filters, time range)
3	Parse SQL structure with `sqlglot`
4	Detect joins, aggregations, and structural risks
5	Compare SQL vs. business intent — surface gaps
6	Generate falsifiable verification hypotheses
7	Execute validation SQL via CRAFT `execute_query`
8	Reason over evidence with Nebius LLM
9	Compute weighted confidence score + export report

Stack

Layer	Technology	Role
Data	CRAFT MCP on Snowflake	Governed schema, term resolution, query execution
Reasoning	Nebius Token Factory (Llama 3.1 70B)	Intent extraction, gap analysis, evidence interpretation
UI	Streamlit	Interactive verification, OAuth sign-in, live progress
Fallback	DuckDB	Offline mode when CRAFT is unavailable

Key design decisions

Hypothesis → SQL → Execute → Evaluate — the LLM never declares a query wrong without data to prove it.
Configurable weights in confidence_weights.yaml — teams define what "trust" means for them.
Policy packs in data_policies.yaml — governance rules plug in without code changes.
Schema helpers for THELOOK_ECOMMERCE — canonical column names, epoch timestamps, corrected reference SQL.

Challenges we ran into

OAuth in Streamlit ≠ OAuth in Cursor

CRAFT MCP works seamlessly inside Cursor. Replicating that in standalone Streamlit meant implementing the full OAuth 2.0 PKCE flow — authorization URL, code exchange, and token refresh.

The surprise killer? localhost vs. 127.0.0.1.

Streamlit session cookies break across redirect URIs. We persisted PKCE verifiers to disk and forced http://127.0.0.1:8501. Hours of debugging for two characters of difference.

Real schema quirks break demo queries

Our flagship example joined on o.id = oi.order_id.

It runs.

It returns numbers.

It's catastrophically wrong — ORDERS uses order_id, not id. Timestamps are microsecond epoch integers, not TIMESTAMP literals. Snowflake wants "created_at" quoted and lowercased.

We built thelook_schema.py and validation checks that probe join cardinality:

\frac{\text{rows after join}}{\text{rows before join}} \gg 1
\Rightarrow
\text{inflated metrics}

LLM reasoning without hallucinated evidence

Early prototypes let the LLM declare a query wrong without running anything.

Impressive prose.

Zero proof.

The fix: every hypothesis compiles into validation SQL, executes via CRAFT, and maps to pass / warn / fail before the LLM narrates the results.

Scoring fairness across heterogeneous checks

Not every failure is equal.

A wrong join key is worse than a missing LIMIT.

We mapped check types into seven weighted dimensions so teams can calibrate what matters most.

Hackathon scope vs. production ambition

We wanted semantic lineage, cross-dataset reconciliation, and real-time dashboard monitoring.

We shipped one thing completely:

Verify a SQL answer against a business question, on live data, with a confidence score you can act on.

Accomplishments that we're proud of

End-to-end agent pipeline — 9 phases from business question to confidence score, report, and corrected SQL.
Live data verification — validation queries execute against real Snowflake via CRAFT MCP, not mocked results.
CRAFT OAuth in Streamlit — standalone app with the same governed data access Cursor enjoys.
Policy verification layer — organization governance rules evaluate every query automatically.
Compelling demo — flawed THELOOK queries that run perfectly and mean nothing, exposed in seconds.
Offline fallback — DuckDB adapter keeps the pipeline demoable without live CRAFT access.
Exportable reports — JSON and Markdown output for sharing verification results with stakeholders.

The demo that proves it:

Load Monthly Revenue 2025, click Run Verification, watch the confidence score crater, inspect the failure cards, and export the corrected SQL.

The query executed.

The insight didn't.

Until TrustSQL.

What we learned

Syntax correctness ≠ semantic correctness.

A query is a function:

f : \text{Schema} \rightarrow \text{Result}

Syntax checkers validate that (f) is well-formed.

TrustSQL validates that (f) answers the right business question.

The best verification is empirical.

LLMs reason brilliantly and guess dangerously.

Evidence must come from executed SQL on real data.

The model interprets.

It doesn't invent.

Real enterprise data is messier than tutorials.

Primary keys aren't always id.

Timestamps aren't always TIMESTAMP.

Governed schema access (CRAFT) is essential — you can't verify what you can't query safely.

Trust must be quantified.

Stakeholders don't want a wall of warnings.

They want a score, thresholds, and actionable recommendations.

Generate fast, verify before you trust.

As AI writes more SQL, the verification layer becomes the bottleneck — and the opportunity.

What's next for TrustSQL

CI/CD SQL review gates — block low-confidence queries before they reach production dashboards.
Slack & Microsoft Teams integrations — notify teams when scheduled reports fall below confidence thresholds.
Organization-wide policy packs — reusable governance templates shared across teams.
Semantic lineage — trace how metric definitions propagate through joins, transformations, and aggregations.
Cross-dataset reconciliation — verify that the same KPI computed across different pipelines agrees within acceptable tolerance.
Real-time dashboard monitoring — continuously score trustworthiness for production BI dashboards.

TrustSQL demonstrates that the next generation of data tooling isn't about generating more SQL.

It's about earning trust in the answers.

Because in analytics, the most expensive bug isn't the one that crashes your query.

It's the one that ships.