Financial AI Copilot

An autonomous AI data scientist for finance: investigates datasets, trains credit & fraud models in a secure sandbox, and remembers every insight with MongoDB Atlas vector memory.

Comment

Inspiration

A typical bank runs every dataset through a chain of specialists: data analysts profile it, fraud investigators hunt anomalies, ML engineers train models, risk officers audit them, and reporting teams write the summaries. That chain takes weeks, and the institutional knowledge it produces evaporates into slide decks. We asked: what if one autonomous agent could run that whole chain — and remember every investigation it ever did?

Working in credit scoring ourselves, we also knew the failure mode of "AI data scientists" that generate ML code on the fly: they demo well and fail in production. So we built the opposite — an agent whose reasoning is free but whose hands are bound to validated, pre-written pipelines.

What it does

Financial AI Copilot is an autonomous financial data scientist. You upload a dataset (or ask it to generate one) and it:

  1. Investigates — profiles schema, detects target leakage, class imbalance, high-risk segments
  2. Decides — picks the credit-scoring (7 steps) or fraud-detection (6 steps) pipeline and the target column on its own
  3. Executes — runs real ML (XGBoost, LightGBM, CatBoost, Optuna, SHAP) inside an isolated E2B sandbox
  4. Shows its work live — streaming reasoning, a pipeline timeline, a model leaderboard, matplotlib charts and downloadable artifacts (model.pkl, input schema, scoring script, governance model card) appear in the UI as each step completes
  5. Remembers — every finding, experiment and report is embedded and stored in MongoDB Atlas, so a brand-new conversation can ask "have we investigated something like this before?" and get a real answer

MongoDB as the agent's entire cognitive substrate

This is not "an app that uses a database." Every form of state the agent has lives in MongoDB Atlas — six distinct roles on one cluster:

Role Mechanism
Working memory (conversation state) LangGraph checkpointer (MongoDBSaver) — every super-step of the agent graph
Procedural memory (resumability) pipeline_state collection — each step writes its outputs; the agent resumes crashed investigations from the last completed step
Episodic memory (what happened) findings, experiments, reports, model_registry collections
Semantic memory (what it means) Atlas Vector Search over 3072-dim Gemini embeddings
Binary memory (files) GridFS — datasets, inter-step DataFrames (parquet), chart PNGs, trained model pickles
Inter-process bus Pipeline scripts running inside the E2B sandbox write directly to Atlas; the UI reads the same collections — the database is the interface between sandbox, agent and frontend

Semantic recall uses cosine similarity over embedding space: a query $q$ retrieves past investigations $d_i$ ranked by:

$$\mathrm{sim}(q, d_i) = \frac{\mathbf{e}q \cdot \mathbf{e}{d_i}}{\lVert \mathbf{e}q \rVert \, \lVert \mathbf{e}{d_i} \rVert}, \qquad \mathbf{e} \in \mathbb{R}^{3072}$$

served by an Atlas vectorSearch index, with text search as a fallback.

The pipeline itself is checkpointed as a state machine: an investigation is a sequence \( s_1 \rightarrow s_2 \rightarrow \dots \rightarrow s_7 \) where each transition persists \( (\text{investigation_id}, s_k, \text{data}k) \) to Atlas before \( s{k+1} \) begins — kill the process at any point and the orchestrator reconstructs exactly where it was.

Model quality gates are enforced numerically, not rhetorically. Each training step computes

$$ \mathrm{AUC} = \int_0^1 \mathrm{TPR}\,(\mathrm{FPR}^{-1}(x))\,dx, \qquad \mathrm{Gini} = 2\,\mathrm{AUC} - 1, \qquad \mathrm{KS} = \max_t \lvert \mathrm{TPR}(t) - \mathrm{FPR}(t) \rvert $$

and the pipeline blocks (exit code 2, CRITICAL finding written to Atlas) if AUC falls below threshold — the agent cannot talk its way past a bad model.

How we built it

Architecture: one orchestrator, zero hallucinated ML. A single DeepAgents orchestrator (Gemini 3 Flash) holds 25+ tools. Its only job is deciding — which pipeline, which step next, what the results mean. All ML runs through validated, pre-written Python scripts executed via a persistent per-thread E2B sandbox (with local fallback). The LLM never writes modeling code.

Custom LangGraph-compatible backend. We replaced langgraph dev with a hand-rolled FastAPI server implementing the full LangGraph API surface (threads, runs, SSE streaming with replay buffers for reconnection) on MongoDB — the official TypeScript SDK works against it unmodified.

Live, observable agency. The Next.js UI reads stream.values on every SSE chunk: reasoning text types out as the model thinks (GSAP), tool calls appear turn by turn, the pipeline timeline advances, charts and downloadable artifacts materialize mid-run, and a terminal panel streams raw sandbox stdout.

Challenges we ran into

  • Real-time streaming was our hardest fight. The agent framework buffers messages until end-of-turn; we rebuilt the flow around per-node updates events, token-level messages events, and SSE replay buffers — then discovered the final bug was a UI branch wiping streamed content during a 6-second Atlas history fetch. Lesson: instrument the browser before blaming the backend.
  • The sandbox is another machine. Scripts ran locally but failed in E2B: host env vars don't exist there, shared modules don't travel, and pymongo wasn't installed. We now embed config literally into composed scripts and materialize shared modules next to them at runtime.
  • Silent state loss. Our framework version dropped Command updates to undeclared state channels — pipeline/leaderboard panels were silently dead until we registered custom channels via middleware.
  • Gemini latency. Time-to-first-token with a large system prompt was ~5s; async checkpoint durability, request-time DB writes moved off the hot path, and thinking_level="low" cut it to ~2.5s.

What we learned

Reliability is an architecture choice, not a prompt. Binding the LLM to validated pipelines while persisting all cognition to MongoDB gave us an agent that is simultaneously autonomous, observable, resumable — and honest, because every claim it makes is backed by a number written to a collection.

What's next

Fraud relationship graph visualization (the NetworkX analysis already runs), experiment lineage trees, multi-dataset cross-investigation reasoning, and Cloud Run deployment hardening.

Built With

Share this project:

Updates