Inspiration
Most text-to-SQL demos show one happy path: a question goes in, a correct query comes out. But real agents fail — and they fail in patterns: a missing JOIN, a hallucinated column, a "youngest / highest" question answered with a subquery instead of ORDER BY ... LIMIT 1. We wanted an agent that doesn't just answer questions, but notices how it fails, rewrites itself, and proves the fix worked — using its own production traces as the evidence. Arize Phoenix and the Phoenix MCP server made that loop possible: the agent can literally read its own telemetry back.
What it does
SQLoop turns natural-language questions into SQL over the Spider benchmark databases, executes them, and answers from the real rows — never invented numbers. More importantly, it improves itself: an Optimizer agent reads the task agent's execution traces through the Phoenix MCP server, clusters the failures by mode, proposes prompt / few-shot changes, A/B-tests each candidate on a held-out set, and keeps a change only if it's significantly better. The result is a rising execution-accuracy curve produced entirely by the agent.
How we built it
Two planes, one closed loop.
Task plane — every step is an OpenInference span in Phoenix:
Router → Schema Linker (with value linking) → SQL Generator (Gemini on Vertex AI) → Executor (sandboxed SQLite) → conditional ReAct Repair (triggers on SQL error / empty result / LLM self-check, with a loop guard).
Improvement plane:
Optimizer agent (Phoenix MCP server as its toolset) → pull failed spans → cluster by failure mode → mine few-shots from successful traces + append reflective generator guidance → validate candidates → held-out A/B with a commit-if-better significance gate → commit only the winner. Each round is logged as a Phoenix Experiment.
Built with Google ADK (code-owned agents), Gemini gemini-3.5-flash on Vertex AI for generation, Spider for data + gold SQL, and execution accuracy (result-set match, column-order-insensitive, with 95% confidence intervals) as a pure code-eval metric. The whole thing is deployed as a Gradio dashboard on Google Cloud Run, traced end-to-end to Arize Phoenix Cloud.
A real failure report the Optimizer produced from live Phoenix traces, for example, clustered runtime errors and a WRONG_COLUMN case (the model wrote a non-existent Name instead of song_name), and proposed concrete fixes: add ORDER BY ... LIMIT 1 few-shots for superlatives, and a strict column-selection rule. Those became the committed candidate.
The evidence (not vibes)
Submission backend — Gemini flash on Vertex, held-out 100 questions across 20 databases, seed 13, greedy decoding:
| round | 0 (baseline) | 1 | result |
|---|---|---|---|
| gemini-3.5-flash (weak) | 60% (CI 50–69) | 77% (CI 68–84) | +17% |
| gemini-3.1-pro (strong) | 87% (CI 79–92) | 87% | +0% |
The weak model gains +17 points from self-improvement. The strong model is already near the ceiling, so the commit gate honestly commits nothing rather than faking a rise — single-point noise never gets committed. Cross-checked on DeepSeek (dev): flash 65→79 (+14), pro 63→78 (+15) — same pattern. Improvement comes from transferable learned rules and retrieved few-shots, so it generalizes across databases rather than memorizing one.
Challenges we ran into
- Phoenix auth from a real exporter.
phoenix.otel.register()doesn't auto-applyPHOENIX_CLIENT_HEADERSto the OTLP span exporter, so large span batches 401'd. We parse the key out of the env and passapi_key=explicitly, with aBatchSpanProcessor+force_flush()to also dodge the genai instrumentor's "client has been closed" on short-lived eval runs. - Memory reuse vs. tracing. Cached answers skip the whole pipeline (zero spans) — great for latency, confusing in a trace demo. We made the distinction explicit in the UI.
- Region ≠ model location. Cloud Run runs in
us-central1, butgemini-3.5-flashis served from Vertex'sgloballocation; the deploy sets these independently. - Lean deploys. Spider is 1.7 GB; we stage a slim ~110 MB build context (only the 20 dev DBs the dashboard executes) and export an index-free
requirements.txtso Cloud Build installs cleanly from PyPI. - One config flips dev ↔ submission.
GOOGLE_GENAI_USE_VERTEXAIswitches a free AI Studio key (development) to Vertex AI (submission) on the same, identically traced code path.
What we learned
Self-improvement only means something if it's measured. Proposing changes is easy — an LLM will happily generate endless "improvements." The hard part is a gate honest enough to reject most of them: a significance bar over a held-out set, with confidence intervals, is what turns "the agent rewrote its prompt" into "the agent made itself measurably better." And MCP is what closes the loop — the agent doesn't need a human to paste its failures back to it; it reads its own traces and acts.
What's next
Larger held-out sets and full Spider-dev evaluation; failure-mode-targeted schema-linking fixes; and an automatic nightly loop that proposes, validates, and opens a pull request with the winning config — self-improvement on a cron.
Log in or sign up for Devpost to join the conversation.