CADRE-AI

Voice-first agent with Gemini native audio. Speak naturally while it runs 38+ tools for finance, web search, and BIM via ADK + Cloud Run.

Comment

Inspiration

As architects and engineers, we spend hours toggling between tools — stock tickers, building models, web searches, research papers — all while on calls or at our desks. What if one voice-driven agent could handle all of it? Cadre-AI was born from the need for a single conversational interface that connects professional tools across domains.

What it does

Cadre-AI is a voice-first professional agent powered by Gemini's native audio model. You speak naturally, and it executes across 38+ tools spanning:

  • Financial Intelligence — Real-time stock quotes, technical analysis (RSI, MACD), fundamentals, portfolio tracking, sector performance, fear/greed index
  • Architecture & BIM — Query and modify Revit building models via named-pipe MCP bridge (levels, rooms, walls, doors, sheets, QA/QC)
  • Web Search & Research — Academic paper search, Wikipedia lookup, deep multi-source research, weather, image search, AI image generation
  • Multimodal Output — Inline images, YouTube video embeds, markdown rendering, follow-up suggestions

Users can ask compound questions and Cadre executes all tools in sequence, delivering a unified response.

How we built it

  • Google ADK (Agent Development Kit) with Runner.run_live() for bidirectional audio streaming
  • Gemini 2.5 Flash Native Audio (gemini-2.5-flash-native-audio-latest) — real-time voice-in, voice-out with automatic VAD
  • MCP Architecture — Each tool domain runs as a separate Model Context Protocol server (financial, web search, Revit proxy), connected via stdio
  • FastAPI + WebSocket server proxies audio between browser and Gemini Live API
  • Cloud Run deployment with custom Dockerfile
  • Custom voice client (vanilla JS) with AudioContext-based mic capture and PCM audio playback

Challenges we ran into

  • Gemini's google_search built-in tool conflicts with MCP function declarations on 2.5 models (400 INVALID_ARGUMENT). Had to replace with custom MCP web search.
  • Chrome's AudioContext autoplay policy silently blocks mic capture — required explicit resume() during user gesture.
  • outputTranscription from native audio is cumulative (full text each message), not incremental deltas — caused double-text display until we built a replace-based transcript handler.
  • Explicit VAD activity control conflicts with automatic VAD, causing WebSocket 1007 disconnects.

Accomplishments that we're proud of

  • True voice-first experience — speak naturally, get voice + rich text responses with images and video
  • 38+ tools accessible through a single conversational interface
  • Real-time bidirectional audio streaming with Gemini Live API
  • Clean MCP architecture that makes adding new tool domains trivial
  • Live at cadreai.dev on Cloud Run

What we learned

  • Gemini's native audio model requires careful handling of text vs. audio output channels
  • MCP (Model Context Protocol) is an excellent pattern for modular tool integration
  • Voice UX requires different design thinking than chat — concise responses, no URL reading, progress updates

What's next for CADRE-AI

  • Document generation (proposals, reports, invoices) via voice command
  • Calendar and email integration
  • Multi-agent orchestration for complex workflows
  • Mobile-optimized voice interface

Built With

Share this project:

Updates

Submission history