Coda

Coda is AI simulation platform designed to refine high-stakes communication skills by recreating realistic environments for users to practice and track measurable growth.

Comment

Inspiration

Every year, countless startup founders walk into investor meetings unprepared — not because they lack great ideas, but because they've never faced a skeptical CTO who interrupts mid-sentence to challenge their architecture, or a CFO who dismantles their unit economics on the spot. The same problem plagues professionals across domains: law students before mock trials, salespeople before critical negotiations, managers before difficult conversations. Traditional preparation — rehearsing in front of a mirror, reading books, or even practicing with friends — simply can't replicate the pressure, unpredictability, and emotional intensity of the real thing.

We asked ourselves: What if AI could recreate the room? Not a chatbot that asks scripted questions, but a living simulation — multiple AI personas with distinct personalities, expertise, and psychological motivations, all reacting to you in real-time through voice, pushing back when your answers are vague, and adapting their behavior based on how you perform. That vision became Coda.

What it does

Coda is an immersive, GenAI-driven simulation platform that recreates high-stakes communication scenarios — investor pitches, salary negotiations, crisis management meetings, press conferences — so users can practice, fail safely, and measurably improve.

Configure your arena. Users design custom scenarios by setting the context, difficulty level, and conversation tone. They engineer AI participant personas — a "Skeptical CTO" with 90% skepticism who interrupts frequently, a "Detail-oriented CFO" who probes financial assumptions — each with unique voices, expertise domains, and behavioral profiles. Users can upload their actual pitch decks, and the system generates contextually grounded questions from the materials.

Step into the simulation. The platform delivers real-time voice-to-voice interaction with AI personas who listen, react, and challenge. A live HUD displays a fluctuating Persuasion Score, individual attendee sentiment gauges, speaking pace, and filler word counts. Real-time coaching tips appear as you speak — "Try to pause before your key statistic" — while the AI personas maintain distinct emotional states that evolve based on your performance.

Get actionable insights. After each session, a comprehensive analytics suite breaks down your performance across dimensions like persuasion, clarity, confidence, empathy, and pacing. Timeline views show exactly when your resilience dipped, timestamped transcript replays highlight moments where you faltered, and the system generates prescriptive coaching roadmaps.

Track your growth. Side-by-side session comparisons let you visualize your journey from Attempt 1 to Attempt 4. A skill radar chart tracks competency growth over time, streak counters gamify daily practice, and the dashboard surfaces recurring patterns — like a filler word count that's finally trending downward.

The platform supports three interaction modes — Presentation (uninterrupted delivery with Q&A phase), Discussion (balanced back-and-forth), and Interview (agent-driven probing) — and simulations can run in 13 languages including Chinese, Japanese, Korean, Spanish, French, and Arabic.

## How we built it

Coda is built on a real-time, event-driven architecture with three distinct layers.

Frontend: React 19 with TypeScript, styled with Tailwind CSS v4 and Shadcn UI. Zustand manages the complex streaming state required for the live HUD, and Framer Motion handles animations. MediaPipe Face Landmarker runs in the browser at 5 FPS, capturing facial expressions (visual confidence, engagement, eye contact, nervousness) and streaming the metrics to the backend via WebSocket.

Backend: FastAPI (Python 3.13+) serves REST endpoints and WebSocket connections. The core of the system is the SimulationOrchestrator, which coordinates several components:

  • A TurnController that uses deterministic rules (direct address detection, topic-to-expertise keyword matching, round-robin balancing) for speaker selection — no LLM calls needed.
  • A ContentRequestEngine that manages question pools and content generation.
  • An OAI (Observe-Ask-Intervene) Framework that gives users approval control over when AI personas speak.

AI Layer: Google's Gemini ecosystem powers the intelligence:

  • Gemini 2.5 Live (gemini-live-2.5-flash-native-audio) for real-time bidirectional voice streaming with distinct persona voices (Puck, Charon, Kore, Fenrir, Aoede).
  • Gemini 3 Flash/Pro for coaching analysis, Q&A generation, and the Adaptive Question Mode's Simulation Brief generation.
  • Cloud Speech-to-Text V2 (Chirp 3 model) listens and transcribes — separated from persona agents to prevent unwanted autonomous responses.
  • Google ADK (Agent Development Kit) for multi-agent orchestration.

The system supports two agent modes: SingleAgentManager (one Gemini Live session embodying all personas with lower latency) and PersonaAgentManager (separate Gemini Live sessions per persona with unique voices).

Data: PostgreSQL with async SQLAlchemy for persistent storage, in-memory asyncio for millisecond-latency session state, and Google Cloud Storage for audio/video assets. Deployed on Google Cloud Platform with Cloud Run, Cloud SQL, Vertex AI.

Challenges we ran into

Making LLMs respect timing. LLMs have no concept of time — they can't "wait 2.5 seconds" or "let the user finish speaking." Our first iterations had agents talking over users or responding instantly without natural pause. We solved this by inverting the control model: Python code controls all timing through the ConversationStateMachine, while LLMs only decide what to say when explicitly asked. This was a fundamental architectural insight that shaped the entire project.

Multi-agent chaos. When multiple AI personas are in a room, the naive approach creates a cacophony — everyone wants to respond to everything, questions pile up, and the same topic gets asked by different personas. We built the TurnController with strict rules: one persona per turn, one question at a time, domain-based speaking (technical topics go to the CTO, financial to the CFO), and a turn budget that forces the user to speak after at most 2 agent turns.

Audio streaming reliability. Real-time bidirectional audio over WebSocket is inherently fragile. Gemini Live sessions have ~10-minute timeouts, Cloud STT streams expire after 5 minutes. We implemented automatic session resumption via ADK's RunConfig and graceful degradation to text-mode if WebSocket connections fail entirely.

Persona depth vs. mechanical behavior. Early personas felt robotic — their questions were formulaic, mapped mechanically from config sliders ("skepticism: 0.8 → skeptical, probing"). We developed the Adaptive Question Mode with LLM-generated Simulation Briefs: rich 5-section character documents covering psychological profile, communication style, knowledge boundaries, and specific claims to probe from pitch materials. The result is personas that feel three-dimensional — a CTO who says "Show me, don't tell me" and defers financial questions to the CFO, not because of a rule, but because the character naturally would.

Audio quality. Frontend audio processing introduced crackling artifacts that broke immersion. Debugging real-time audio pipelines across browser APIs, WebSocket binary streaming, and AI-generated audio playback required careful attention to sample rates (16kHz input, 24kHz output), buffer management, and the AudioWorklet API.

Accomplishments that we're proud of

The architecture is the product. The ConversationStateMachine + TurnController + ContentRequestEngine pattern creates simulations that genuinely feel like being in a room with real people. When "Marcus" interrupts your pitch to say "Wait — your CAC seems unsustainably low for this industry. How are you not being outspent by incumbents?", and you can hear the skepticism in his voice, and your Persuasion Score dips on the HUD — that's not a chatbot, that's a training environment.

Real-time coaching that doesn't break flow. The coaching agent operates as a silent observer in parallel with the simulation, delivering HUD updates and tips without disrupting the natural conversation rhythm. You get feedback while you're performing, not just after.

Session comparison that shows growth. Seeing your Persuasion Score climb from 55 to 84 across four attempts at the same scenario, watching your filler word count drop from 23 to 5, reading the AI's analysis that "Your opening statement in Attempt 4 showed steady eye contact and effectively commanded the room's attention" — that's the feedback loop that drives real skill development.

13-language support from the ground up. A user in Tokyo can practice a pitch in Japanese with AI personas who speak, think, coach, and report entirely in Japanese — including STT, TTS, real-time coaching tips, and post-session analytics. The language directive system threads through every component from the database to the prompt templates.

The OAI Framework. Giving users control over when AI personas speak — with previews of what they're about to say and an approval/reject mechanism — was a UX breakthrough. It transforms the simulation from something that happens to you into something you're in control of while still being challenged.

What we learned

Separate what LLMs are good at from what code is good at. LLMs excel at generating natural dialogue, embodying characters, and understanding context. They are terrible at timing, turn-taking, and following strict protocols. The moment we stopped trying to make LLMs manage conversation flow and built a proper state machine, everything clicked.

Pre-computation beats real-time complexity. Generating Simulation Briefs and Q&A pools at scenario save time (not session start time) means zero latency and zero API calls when the simulation begins. The same principle applies everywhere — invest computation upfront to keep the real-time loop fast.

Deterministic routing beats LLM routing. Using regex patterns for direct address detection and keyword-to-expertise mapping for speaker selection is faster, cheaper, and more predictable than asking an LLM "who should speak next?" The LLM's job is content, not control flow.

Audio is hard. Sample rate mismatches, buffer underruns, browser API inconsistencies, WebSocket binary framing — every layer of the audio pipeline is a potential source of bugs that are nearly impossible to reproduce in testing. Build resilience at every layer.

Character depth creates engagement. The difference between a persona with "skepticism: 0.8" and one with a full Simulation Brief including hidden biases, key phrases, and knowledge boundaries is the difference between a quiz and an experience. Users practice more — and learn more — when the AI feels real.

What's next for Coda Coach

Multiplayer sessions. Allowing human mentors and coaches to join live simulations as observers — watching the trainee interact with AI personas in real-time and providing human coaching alongside AI feedback.

Emotional intelligence. Using the facial expression data (visual confidence, nervousness, eye contact) we already capture to make AI personas dynamically respond to the user's emotional state — pressing harder when they detect confidence, backing off or redirecting when they sense frustration.

Collaborative agents. Moving beyond independent questioning to personas that build on each other's lines of inquiry — a CTO who references a concern the CFO raised, or a panel that organically converges on the weakest part of a pitch.

Video analysis. Full body language analysis through video stream processing — posture, gestures, movement patterns — to complement the facial expression metrics we already track.

VR integration. Extending the immersive experience to Virtual Reality headsets, placing users in virtual boardrooms and auditoriums for an even more realistic practice environment.

Community & marketplace. A scenario marketplace where communication coaches, industry experts, and experienced professionals can share their simulation templates — complete with calibrated personas and curated question pools — so best practices scale across the platform.

Built With

Share this project:

Updates

Submission history