Litterman.ai

The AI voice co-pilot for asset managers: speak a market headline, get an institutional-grade portfolio rebalancing in seconds.

Comment

Litterman.ai — Voice-Powered Portfolio Optimization for Asset Managers

Inspiration

I came to this idea through a university investment challenge. My team spent hours debating market news: Fed decisions, jobs reports, earnings surprises. Trying to agree on what each headline meant for our portfolio weights. The debate was valuable, but the process was painfully slow: read the news, form a view, open a spreadsheet, run the math, decide. By the time we had a number, the market had already moved. That experience planted a question I couldn't shake: what if an AI agent could do that entire workflow in seconds? Speak the news. Hear the recommendation. Act. The name comes from Fischer Black and Robert Litterman, who developed the Black-Litterman model at Goldman Sachs in 1990, a quantitative framework that lets portfolio managers blend their own views with market equilibrium mathematically. It's the industry standard for institutional portfolio construction, and it was the perfect engine for this agent.

What It Does

Litterman.ai is a voice-powered portfolio optimization agent for fund managers. The workflow is simple:

  1. The manager speaks a market headline out loud
  2. The agent immediately acknowledges: "Understood. Stand by for analysis."
  3. Gemini 2.5 Flash classifies the news and with Google Search Grounding extracts structured Black-Litterman views
  4. The Black-Litterman engine optimizes the portfolio
  5. The agent responds verbally with the new allocations and the Sharpe ratio
  6. The live dashboard on Cloud Run updates in real time via Firestore

The manager can interrupt mid-response, ask follow-up questions, submit news via text or URL, and confirm or reset the rebalancing, all from a single interface.

How I Built It

Architecture

Voice Input (PyAudio · 16kHz PCM)
        ↓
Gemini 2.5 Flash — Native Audio (Live API)
 Bidirectional session · Input transcription · Voice response
        ↓
Gemini 2.5 Flash — Market News Classifier
  Binary YES/NO: "Is this a financial market event?"
        ↓
Gemini 2.5 Flash — View Extractor + Google Search Grounding
  Extracts P matrix + Q vector · Live yields · Central bank data
        ↓
Black-Litterman Engine (numpy · scipy · SLSQP)
  Market equilibrium prior · Bayesian blending · Turnover ≤20pp
        ↓
Output: Voice Response + Cloud Firestore + Live Dashboard (Cloud Run)

The Black-Litterman Model

The core quantitative engine implements the full Black-Litterman framework:

Equilibrium returns (reverse optimization):

$$\Pi = \lambda \Sigma w_{mkt}$$

Posterior expected returns (Bayesian blending):

$$\mu_{BL} = \left[(\tau \Sigma)^{-1} + P^T \Omega^{-1} P\right]^{-1} \left[(\tau \Sigma)^{-1} \Pi + P^T \Omega^{-1} Q\right]$$

Where:

  • \(\Pi\) = implied equilibrium returns
  • \(\lambda\) = risk aversion coefficient
  • \(\Sigma\) = covariance matrix
  • \(P\) = pick matrix (which assets the view is about)
  • \(Q\) = view vector (expected returns for each view)
  • \(\Omega\) = view uncertainty matrix

The optimizer then solves a constrained mean-variance problem via SLSQP, with a 20pp max turnover constraint per rebalancing event and 75% max concentration per asset.

Voice Pipeline — Barge-in Architecture

One of the key engineering challenges was implementing true barge-in — the ability to interrupt the agent mid-response and have it stop and listen immediately. This required a two-flag architecture:

  • _bl_running — blocks duplicate BL pipeline execution while one is already running
  • _bl_cooldown — prevents re-triggering after a response, but never blocks conversation

A single-flag approach caused conversation lockouts where the agent would stop responding entirely. The two-flag pattern solved this cleanly.

Results are injected back into the Gemini Live session using send_realtime_input(text=prompt) — the only method that maintains session context continuity and produces audio output.

Google Search Grounding

The view extractor uses Google Search Grounding to anchor its analysis in live market data — current yields, inflation figures, central bank positioning. This grounds the Black-Litterman views in real-world numbers rather than relying solely on the model's training data.

In testing, the grounding consistently retrieved 7–17 live sources per analysis call, materially improving the quality and specificity of the extracted views.

State Management

Cloud Run is stateless, so all shared state — portfolio weights, BL results, event log — lives in Cloud Firestore (default database). The dashboard polls Firestore every 2 seconds and updates in real time. The (default) database naming is critical — custom names cause 404 errors with the Python SDK.

Challenges

Barge-in without lockouts. Getting the agent to stop mid-sentence and respond to a follow-up question, without either ignoring the interruption or locking the conversation entirely, took significant iteration. The two-flag architecture was the solution.

Stateless deployment on Cloud Run. The voice agent needs microphone access and runs locally, while the dashboard runs on Cloud Run. Firestore as the shared state layer was the right architectural decision, but it required migrating away from a local state.json file mid-development.

CRLF corruption on Windows. Dockerfiles and requirements.txt built on Windows with CRLF line endings caused silent failures in Cloud Build. Fixed with .gitattributes enforcing LF for all relevant file types.

Module import path conflicts. Local development uses from bl_engine import ... while Cloud Run expects from core.bl_engine import .... Resolved with a try/except import pattern that works in both environments.

Thinking budget in voice responses. With thinking_budget unset, Gemini would output markdown "thinking" headers as audio — producing garbled responses. Setting thinking_budget=0 eliminated this entirely.

What I Learned

  • The Gemini Live API's Native Audio model is genuinely capable of real-time bidirectional interaction — barge-in is possible with the right architecture
  • Google Search Grounding adds measurable value for time-sensitive financial analysis, not just factual lookups
  • Cloud Run + Firestore is a clean pattern for stateless agents that need shared state
  • The Black-Litterman model is more intuitive to implement than its mathematical notation suggests, it's fundamentally just Bayesian updating

What's Next

For institutional clients, the biggest challenge is MNPI compliance — sending a portfolio manager's speech to a cloud API creates real legal exposure. The natural next step is a private deployment tier: local LLM inference (Ollama/llama.cpp) for the classification and view extraction layers, keeping sensitive data on-premise, while retaining Gemini Live for voice I/O.

Other directions:

  • Custody integrations (execute rebalancing directly, not just recommend)
  • Regulatory audit trail export (PDF/CSV for compliance teams)
  • Prediction market integration (Polymarket) as a probabilistic source for BL view confidence

Built With

gemini-live-api · gemini-2.5-flash · google-search-grounding · cloud-firestore · cloud-run · cloud-build · python · numpy · scipy · flask · plotly · pyaudio · docker

Built With

Share this project:

Updates

Submission history