Understudy

A live meeting agent that turns talk into done - it extracts action items as they're spoken, executes them in parallel while you talk, and keeps chasing every commitment for days after.

Comment

Inspiration

We say things in meetings. We just don't do them.

Think about the last meeting you were in. Somebody said "I'll send that over." Somebody else said "let's book the review." A third person promised to file the bug. Everyone nodded. Then the call ended, the tabs closed, the day swallowed everyone whole, and three days later half of it had quietly never happened. Not because anyone was lazy, but because the moment a meeting ends, the work falls into the gap between deciding and doing, and no one is standing in that gap.

I wanted to build the thing that stands in it. Not another note-taker that hands you a tidy summary and leaves the actual work to you. Something that does the work. So I named it Understudy: the one who quietly learns every line and every cue, and steps in to perform when the lead can't. The promise I set for it was simple:

By the time you say goodbye, the work is already done.

There was a night I almost scrapped it. I asked myself, honestly, is this good enough to win? And the answer stung: "meeting to action items" is the most crowded corner of AI, and I could build the cleanest version of a thing everyone has already seen. Then the reframe came from the hackathon's own words, run in the background, asynchronously, hold context across weeks. The crowded part is the live moment. The part nobody nails is what happens after. That was the night Understudy stopped being a note-taker in my head and started being a teammate.

What it does

Understudy listens to your meeting and does the work while you are still talking.

  • It acts live, in parallel. A watcher agent pulls out each commitment the second it is spoken, and an orchestrator fans the work out concurrently, so items are finishing while the conversation is still going. Each one shows up as a card that flips from queued to running to done in a floating companion window in the corner of your call.
  • It operates your tools, not just words. It creates a real Gmail draft and sends it on one tap. When a meeting says "the pricing page still shows the old numbers," it opens a real GitHub issue and a pull request with the actual code diff, waiting for you to merge. It files a real Plane ticket and walks it from Todo to Done. It also books real Google Calendar events and writes real Google Docs, and posts to Slack.
  • It keeps working after the meeting. Every commitment becomes a tracked item on a standing board. A scheduled scanner nudges owners, flags what has stalled, and escalates what is blocked, so things actually close.
  • It remembers across meetings. Decisions and commitments are embedded and recalled semantically, so "three weeks ago you decided X" surfaces when it matters.
  • Audio never leaves your device. No bot joins your call. A local model transcribes on your machine and discards the audio. Privacy is the architecture, not a checkbox.

How we built it

I built it solo, from zero on the Google agent stack, over about six days, in the Antigravity IDE.

The reasoning engine is Gemini 3.5 Flash on Vertex AI, orchestrated with Google's ADK (an LlmAgent watcher with structured output, a ParallelAgent fan-out to tool sub-agents). I proved the intelligence on typed transcripts first, then built the real input layer: a local listener that captures the mic, segments speech with voice-activity detection, and transcribes with faster-whisper, plus an opt-in screen-watcher that sends only changed, downscaled frames to Gemini for multimodal understanding of slides and docs.

Everything runs through a FastAPI server on Cloud Run, backed by Firestore for realtime state (the dashboard and companion read it live via onSnapshot, no websocket code). A Cloud Scheduler job drives the async follow-through. Along the pipeline sit an opt-in Gemma first-pass filter, a Model Armor style guardrail, and OpenTelemetry-style reasoning traces. A MOCK_AGENT mode lets the whole system run with zero model calls for fast, deterministic testing and demos.

Challenges we ran into

  • Real-time transcription without dropping words. Whisper is slow enough that it blocked audio capture and lost speech. I moved transcription onto a worker thread behind a queue, so capturing never waits on thinking, and tuned Whisper to stop hallucinating on silence.
  • Keeping the event loop responsive under load. The tool handlers made blocking network calls (GitHub, Plane, Gmail, Slack, embeddings) that starved the async server when several actions ran at once, so live transcript and cards stuttered. I pushed every blocking call onto worker threads, which kept capture and the UI smooth while the work ran in parallel.
  • A quota wall mid-build. The free tier's daily cap stopped me from testing. I answered it by building the full mock mode, so almost all development needed no live calls.
  • Parallel work that fails gracefully. Each item runs isolated with backoff on throttling, so one failed tool never crashes the batch, and a card that gets cleared can never resurrect stale state.
  • Making the buttons real. The dashboard's Approve first only wrote to the database. I routed clicks through the server's real endpoints (with CORS) so Approve actually sends and answering a clarification actually resumes execution.

Accomplishments that we're proud of

  • It genuinely does things in the real world, not mockups: a real email sent, a real GitHub pull request with a real diff, a real Plane ticket moving through its states, real calendar events and docs.
  • The privacy stance: a live meeting agent where the audio never leaves your machine and no bot ever joins the call.
  • Hitting three of the enterprise pillars the hackathon named (an opt-in Gemma filter, a Model Armor style guardrail, and agent observability), plus confidence-gated clarification, in a solo build.
  • Watching the whole thing work end to end: talk, and the work appears, finished, before you have said goodbye.

What we learned

Autonomy is a feeling before it is a feature. The moment that lands, every single time, is watching a card flip to done while you are still mid-sentence. And the right amount of autonomy is the trustworthy amount: Understudy does the reversible things on its own, asks when it is genuinely unsure, and always keeps a human finger on anything it cannot take back. I also learned that the differentiator was not the clever live moment everyone builds, but the boring, patient follow-through that closes the loop over days.

What's next for Understudy

Auto-join and system-audio capture so it hears everyone on a remote call, calendar-triggered auto-start, upgrading cross-meeting memory to Firestore vector search at scale, and per-vertical playbooks such as sales-call to CRM follow-through. Because we say things in meetings, and Understudy should make sure they actually happen.

Built With

  • cloud-run
  • cloud-scheduler
  • electron
  • fastapi
  • faster-whisper
  • firestore
  • gemini
  • gemini-3.5-flash
  • gemma
  • github-api
  • gmail-api
  • google-adk
  • google-calendar-api
  • google-docs-api
  • google-genai
  • opentelemetry
  • plane
  • python'
  • react
  • slack-bolt
  • typescript
  • vertex-ai
  • vertex-embeddings
  • vite
  • webrtcvad
Share this project:

Updates

Submission history