DBsensei

Project Story

About the project

We built a self-improving analytics agent that turns natural-language questions into safe, correct ClickHouse SQL—and gets better automatically over time.

What inspired us

Text-to-SQL demos usually fail on real edge cases (wrong joins, wrong grouping, wrong column order). We wanted an agent that can prove correctness and learn from mistakes, not just “try again.”

How it works

Ask: user asks a question → agent outputs JSON only with SQL.
Safety gate: SQL must pass a strict read-only validator (single SELECT, no writes).
Evaluate: we compare the agent’s result to a gold query using a stable result signature:

same columns (and order)
same row count
same sample hash [ \text{pass} \iff \text{sig(agent)} = \text{sig(gold)} ]
1. Repair: if it fails, we generate a repair prompt that includes the mismatch type (cols/rows/hash), the gold SQL, and the agent SQL.
2. Learn: if the repair succeeds, we store the fixed (question → SQL) as a trusted exemplar (deduped by hashes).
3. Retrieve: next queries pull the most relevant exemplars (token similarity) and inject them into the prompt—so the agent avoids repeating past mistakes.

What we built (dashboard)

Ask: query → SQL + results
Evaluations: benchmark runs + pass/fail breakdown
Improve: run the loop and see success rate increase
Overview / Schema / Settings: monitoring + schema card + configuration

Challenges & learnings

“Looks correct” isn’t enough—signatures made correctness measurable.
Repairs needed strict constraints (exact columns + aliases).
We added semantic dedupe to prevent learning the same pattern repeatedly.

The result

The agent self-improves through a closed loop: measure → repair → store → retrieve → repeat, and we visualize progress across runs in the dashboard.