QuantSentinel

A multi-agent Gemini research team that backtests hypotheses, fact-checks its own memos, and rewrites its prompts overnight — fully traced in Arize Phoenix.

Comment

Inspiration

Every week, social media is full of confident claims like "buy when RSI drops below 30 — guaranteed money." Almost none of these are backtested, none report statistical significance, and none mention the assumptions baked in. That gap between confident-sounding claims and actual evidence is something every quant-curious builder runs into.

At the same time, the Google Cloud Rapid Agent Hackathon's Arize track pushed us toward a question more interesting than "can an LLM write a trading strategy?" — namely, can an agentic system grade its own work, and then get measurably better at the job over time, the same way a junior analyst improves after rounds of feedback from a senior researcher?

QuantSentinel is our answer to both: a multi-agent research desk that turns a hunch into a rigorous, falsifiable memo — and then uses Arize Phoenix traces and DSPy to rewrite its own prompts overnight, keeping only the changes that provably score higher on a held-out benchmark.

What it does

You type a hypothesis in plain English — e.g. "Buying SPY when RSI(2) closes below 10 and selling when it crosses above 70 beats buy-and-hold from 2010–2024" — and QuantSentinel turns it into a fully-reasoned, statistically-tested research memo in real time.

The pipeline — 5 agents, built on Google ADK + Gemini:

  1. Orchestrator (gemini-3.1-pro-preview) decomposes the hypothesis into a structured plan: tickers, date range, signal logic, which statistical tests are required, and whether FOMC event data is needed.
  2. Data Agent (gemini-2.5-flash) pulls daily OHLCV bars via yfinance and FOMC meeting dates via FRED, caching them server-side and returning a lightweight cache_key.
  3. Backtester Agent (gemini-2.5-flash) writes Python signal code on the fly — entries, exits, position sizing — and executes it inside an AST-validated sandbox (only pandas / numpy / scipy / stdlib allowed). It returns the equity curve, trade log, and headline metrics.
  4. Statistician Agent (gemini-2.5-flash) runs the actual hypothesis test. For strategy returns $r_t$ vs. benchmark returns $b_t$:
  • Sharpe ratio: $\text{Sharpe} = \dfrac{\bar{r}}{\sigma_r}\sqrt{252}$
  • Welch's t-test (unequal variances) for $\bar r \neq \bar b$ at $p < 0.05$
  • A bootstrap 95% confidence interval on mean strategy return
  • Max drawdown: $\text{MDD} = \max_t \dfrac{\text{Peak}_t - V_t}{\text{Peak}_t}$

…then renders the equity curve vs. benchmark as a chart.

  1. Critic Agent (gemini-3.1-pro-preview) writes an 8-section memo — Hypothesis, Data Sources, Methodology, Results, Statistical Analysis, Risk Caveats, Conclusion (with an explicit high / medium / low confidence verdict), and a standing disclaimer.

Then the self-grading kicks in. An LLM-as-judge Evaluator scores the memo against the underlying numbers on four axes — faithfulness ($F$), statistical correctness ($S$), hallucination ($H$), and risk-caveat coverage ($R$) — combined into an overall score:

$$ \text{overall} = 0.30F + 0.30S + 0.25(1-H) + 0.15R $$

A Suggestion Engine then proposes the next hypothesis to test — tightening a parameter, testing a different regime — automatically skipping anything already tried.

Everything streams live to a dashboard: a real-time agent timeline showing each agent's "thoughts" and tool calls as they happen, the equity curve / drawdown chart, p-value and confidence-interval bars, the eval scorecard, and an improvement chart tracking the eval score across runs over time — visual proof the system is getting better, not just busier.

How we built it

Backend — Python 3.13 + FastAPI, with agents defined using Google's Agent Development Kit (ADK) and served through a single /run SSE endpoint. The orchestrator and critic run on gemini-3.1-pro-preview for reasoning quality; the data, backtester, and statistician agents run on gemini-2.5-flash for speed and cost.

To keep five agents from blowing through context limits, every tool that returns large data (OHLCV bars, equity curves, FOMC dates) writes to a server-side cache and hands back only a cache_key + summary stats — agents pass keys to each other, never raw arrays.

Tooling layer — yfinance + FRED for market/macro data, pandas / numpy / scipy for the quant math, matplotlib for base64-encoded equity-curve charts, and a custom AST-validated sandbox for executing LLM-generated backtest code safely.

Evaluation & self-improvement — A Gemini-based LLM-as-judge scores each memo, with deterministic heuristic scorers as a fallback if the judge call fails. A 20-hypothesis golden dataset, grounded in real academic literature (Moskowitz et al. on time-series momentum, Faber on SMA timing, Jegadeesh & Lehmann on short-term reversal, Lucca & Moench on FOMC drift, and more), is split 15 / 5 into train / held-out sets. A nightly DSPy BootstrapFewShot job mines low-scoring Arize Phoenix traces plus the training split, recompiles the critic's prompt, evaluates the candidate on the held-out set, and only promotes it to Phoenix if it beats the incumbent — that gate is what makes the improvement chart meaningful.

Observability — Arize Phoenix + OpenInference instrumentation wraps every agent and tool call from the first request, providing both debugging traces and the training data the optimizer learns from.

Frontend — Next.js 14 (App Router) + React 18, with an SSE proxy route that keeps long-running streams alive, hand-rolled SVG charts (equity curve, drawdown, p-value bar, bootstrap CI bar) plus Recharts for the improvement trend, and a monochrome "glass terminal" design system — black backgrounds, glowing white accents, frosted-glass cards.

Deployment — Dockerized and deployed to Google Cloud Run, with the nightly optimizer running as a separate Cloud Run Job.

Challenges we ran into

ADK's "one tool call per turn" rule, the hard way. Early on, the orchestrator occasionally emitted two function calls in a single turn — and the entire ADK run loop died with a fatal error mid-session. There's no parallel_tool_calls=false flag in the Google GenAI SDK, so we enforced this entirely through prompt engineering: the orchestrator instructions open with an all-caps "ABSOLUTE RULE — ONE TOOL CALL PER TURN" block, with every step explicitly saying "wait for the result before continuing."

Context overflow from market data. A year of daily OHLCV data, an equity curve, and a returns array are each thousands of floats — multiply that across 5 agents passing data back and forth and you blow past context limits fast. We solved this with a cache-key indirection layer: tools write large payloads to a server-side cache and return only a cache_key + summary.

Phoenix MCP on Windows. We initially wired up Arize Phoenix via its MCP server (npx @arizeai/phoenix-mcp), but the stdio transport raced with asyncio cancel scopes on Windows and reliably hung the session. We replaced it with direct Python calls to Phoenix's query API — more reliable and one less moving part.

Gemini rate limits and timeouts during demos. A live multi-agent run can take well over a minute, and Gemini occasionally returns 429s. We built a dispatcher that retries on 429 with backoff, but on a hard timeout it immediately falls back to a deterministic Python pipeline — same SSE event names, same UI, real backtest numbers, just without the LLM narration. The demo never shows a blank screen.

DSPy + LiteLLM provider prefixes. DSPy's LM() fails silently (or errors confusingly) if the model string doesn't have the right provider prefix — vertex_ai/gemini-3-flash-preview for Vertex, gemini/gemini-3-flash-preview for the direct API. google/... looks correct but isn't recognized at all.

Making "self-improving" actually mean something. It would have been easy to fake an improvement chart with random noise. Instead we insisted on a held-out golden set the optimizer never trains on, plus a strict promote-only-if-the-candidate-beats-the-incumbent gate — so the number on that chart reflects a real, validated gain.

Accomplishments that we're proud of

  • A closed-loop self-improvement system that's real, not cosmetic: Phoenix traces → DSPy BootstrapFewShot → held-out golden-set evaluation → promote-only-if-better → visible improvement chart.
  • A 20-hypothesis golden dataset spanning momentum, mean-reversion, seasonality, FOMC/macro, factor anomalies, and volatility regimes — each grounded in a real academic citation, giving us a credible answer to "how do you know this agent is any good?"
  • A demo-proof dual-path architecture: whether the ADK/Gemini path succeeds, hits a 429, or times out entirely, the user always gets a complete, numerically-grounded research memo through the same UI.
  • A live agent-thought stream that turns an opaque multi-agent backend into something a non-technical judge can watch and understand — you can see the orchestrator plan, hand off to each specialist, and watch the critic write the memo section by section.
  • Observability from line one — every agent and tool call is traced in Arize Phoenix, which is also the data source the optimizer learns from, so observability is load-bearing infrastructure, not a side feature.

What we learned

  • Context-window economics shape multi-agent architecture as much as prompts do. The cache-key pattern is the difference between a 5-agent pipeline that works and one that times out on the second hop.
  • ADK's runtime has sharp edges that aren't always documented — like the one-tool-call-per-turn constraint — and the fix is often prompt-level discipline, not code.
  • LLM-as-judge is genuinely useful, but needs a deterministic safety net. Our heuristic scorers — checking that key numbers appear, verifying CI/p-value interpretation, counting risk-caveat terms — catch cases where the judge call fails, and agree with it often enough to be a credible fallback rather than a stub.
  • DSPy's BootstrapFewShot is a lightweight, surprisingly effective way to demonstrate self-improvement without standing up a full RL loop, as long as it's paired with held-out evaluation and a promotion gate.
  • Resilience is a feature, especially for live demos. The fallback pipeline started as a safety net and became one of the things we're proudest of — a complete, independent implementation of the same research workflow in pure Python.

What's next for QuantSentinel

  • Optimize more than the critic. The nightly DSPy job currently tunes only the critic's prompt; the orchestrator's planning prompt and the backtester's signal-generation prompt are equally good optimization targets.
  • Portfolio- and multi-asset hypotheses — cross-sectional strategies across baskets of tickers, not just single-ticker signals.
  • Walk-forward / out-of-sample validation as a standard part of the methodology, directly addressing the overfitting caveat the critic already flags.
  • Multiple-testing correction across a user's history — once someone has tested 50 hypotheses, the $p < 0.05$ bar for any individual one needs to move (Bonferroni / false discovery rate).
  • Persistent accounts and research history, replacing the local run_store.jsonl with a real database so a user's hypothesis history, suggestions, and improvement curve persist across sessions.
  • Paper-trading hookup — feed the suggestion engine's recommended hypotheses into a paper-trading account and track forward performance, closing the loop between backtest and reality.

Built With

Share this project:

Updates