NorthRiver Banking Agent

NorthRiver Bank: meet River, your AI voice banker. Check balance, transfer money & pay bills by snapping a photo - all by voice commands, in seconds. No taps, no forms, no branch. Just speak to River.

Comment

Inspiration

Banking apps are cluttered with menus, taps, and forms — yet the most natural thing a person can do is simply talk. We were inspired by the idea that your voice should be enough to manage your finances. No navigation, no waiting on hold, no hunting through dashboards. We asked: what if your bank just knew you, and all you had to do was speak?

What it does

NorthRiver Banking Agent introduces River — a voice-first AI banking assistant powered by Google's Gemini Live API. Once signed in, customers interact entirely through natural conversation:

  • "How much did I spend on groceries last month?" — River queries your transaction history and answers in plain speech
  • "Send €50 to David" — River looks up the contact, reads back the IBAN and amount for confirmation, then executes the transfer
  • "Pay this bill" + photo — River reads the SEPA EPC QR code embedded in the image, extracts the merchant name, amount and reference, confirms with you, and pays it — all within the same voice conversation

No typing. No forms. No branch visits.

How we built it

Backend: FastAPI server running Google ADK's runner.run_live() over a bidirectional WebSocket. Binary frames carry raw PCM16 audio at 16kHz; JSON frames carry images and text. We built a multi-agent hierarchy — a root orchestrator (River) that greets the customer and silently routes to two specialist sub-agents: account_info_agent and payments_agent.

AI: Gemini Live 2.5 Flash Native Audio on Vertex AI, giving us real-time audio-in / audio-out with sub-500ms latency and native multimodal support for QR bill scanning.

Data: Firestore as a mock banking database with accounts, transactions, and contacts collections. Five tools handle all reads and writes.

Frontend: React + Vite, served as static files by FastAPI. A Web AudioWorklet captures microphone input as PCM16; a custom AudioPlayer streams 24kHz PCM output without gaps. The UI shows a live transcript, tool-call toasts, and a bill photo uploader.

Infrastructure: Containerised with Docker (multi-stage Node + Python build), deployed to Cloud Run with a Cloud Build CI/CD pipeline.

Challenges we ran into

  • Image injection into sub-agents: ADK's get_fast_api_app helper doesn't support injecting image content mid-session. We had to drop down to raw runner.run_live() and manually construct multi-part Content turns — bundling the image and user intent into a single turn so both the orchestrator and the payments sub-agent could see it.
  • Output transcription buffering: Vertex AI sends transcription in unpredictable chunks — sometimes cumulative, sometimes independent deltas. We built a dual-strategy buffer that detects which mode the model is using per-turn and flushes only on finished=True or turn_complete, eliminating garbled partial text in the UI.
  • Session context limits: Long conversations hit Vertex AI's 10-minute session cap. We wrapped the agent in ADK's App with EventsCompactionConfig to summarise older history automatically, allowing indefinite sessions.
  • Barge-in handling: Detecting when the user interrupts mid-response and cleanly discarding the buffered agent transcript without leaving orphaned UI bubbles required careful state management across the async upstream and downstream tasks.

Accomplishments that we're proud of

  • Full QR bill payment by voice — the showstopper demo flow works end-to-end: snap a photo, River reads the SEPA EPC QR code visually using Gemini's multimodal capabilities, confirms the details, and executes the payment
  • Zero-silence conversations — River speaks first on session open, and filler phrases before every tool call mean the customer never hears dead air
  • Clean multi-agent routing — the orchestrator delegates silently; customers never know there are multiple agents running behind the scenes
  • Production-ready infrastructure — Cloud Run deployment, Cloud Build CI/CD, zero-downtime rolling updates, and smoke tests baked in from day one

What we learned

  • Gemini Live's native audio pipeline is remarkably capable but requires careful plumbing — the ADK abstractions help, but real-world features like image injection demand working at a lower level
  • Voice UX design is its own discipline: response length, filler phrases, confirmation flows, and barge-in behaviour all need deliberate thought that doesn't exist in text-based chatbots
  • Multi-agent architectures shine when domain boundaries are clear — keeping account queries and payments in separate agents made both prompts tighter and easier to debug
  • Firestore's real-time capabilities are well-suited to banking session state, but structuring queries to stay within ADK's tool context lifecycle requires discipline

What's next for NorthRiver Banking Agent

  • More banking actions: scheduled payments, standing orders, card freeze/unfreeze, and spending insights with natural language date ranges
  • Proactive alerts: River surfaces unusual spending or low balance warnings at the start of the session without being asked
  • Multi-language support: Gemini Live supports multilingual audio — River should too
  • Production auth: replace demo credentials with OAuth / OpenID Connect and a real core-banking API integration
  • Accessibility focus: voice-first banking is transformative for visually impaired customers — dedicated testing and refinement for this use case
  • Analytics dashboard: session transcripts → BigQuery → LLM-as-judge scoring to continuously improve River's response quality

Built With

  • audioworklet
  • docker
  • fastapi
  • firestore
  • gemini-live-2.5-flash
  • google-adk
  • google-cloud-build
  • google-cloud-run
  • javascript
  • pcm16
  • python
  • react
  • vertex-ai
  • vite
  • web-audio-api
  • websocket
Share this project:

Updates

Submission history