Huddle

Inspiration

We're Yurny Labs, a product design and engineering studio, and we kept hitting the same problem on our own projects. The real state of a project, what's decided, what's blocked, who's doing what, never lives in one place. It's scattered across conversations and stuck in people's heads. Nobody has time to keep it current, so it rots. The same questions get asked twice. Blockers sit for days before anyone notices. And because we all work flexible hours, someone is always the one catching up on a decision they missed.

It wasn't something we could fix with more process. We'd tried standup notes, status docs, a shared board, and keeping any of them current was its own job, the one nobody wanted to own.

We didn't want another bot that transcribes meetings and files notes. We wanted a teammate that listens, keeps the project's truth accurate on its own, and speaks up, out loud, mid-conversation, when something contradicts what it already knows.

What it does

Huddle is a single AI teammate shared by your whole team, with two ways in:

• a personal room: your always-on 1:1 with the team's agent, which writes through to the shared team context by default;
• a team huddle: the same agent, live, with everyone present. It mostly stays quiet and only interjects on a real conflict or a genuine ambiguity the conversation is skating past.

Either way, when someone states something durable, whether a decision, a blocker, an open question, or work being committed to, the agent reconciles it against current state and records it as a task, an open item, or a knowledge note. At the end of a huddle it re-tiers every knowledge note (evergreen / active / faded), folds faded context into a project digest, and narrates exactly what the session changed.

How we built it

Two independent services:

• Web app: FastAPI backend, Vue 3 frontend. It handles Google sign-in, teams, invites, and streams live project state to the browser over SSE, pushed straight from Firestore on_snapshot watches. The browser never touches Firestore directly.
• Voice worker: a Python process that joins a LiveKit room the moment anyone connects, wiring room audio into Google ADK's run_live, which streams straight to Gemini's Live API and speaks the reply back.

State lives in one Firestore document tree per team. The agent itself is a root persona that delegates to specialists (knowledge, open items, tasks, notifications, artifacts), plus two passes (a relevance curator and a session summarizer) that run once at the end of a gathering, when they can see the whole session rather than just the last utterance.

We kept a hard architectural line: the agent graph knows nothing about LiveKit. The room-facing worker hands it plain data (team id, roster, current speaker) and never a transport object, so the connection method stays swappable.

Challenges we ran into

Voice latency. Our first version was a classic pipeline: speech-to-text, then the LLM, then text-to-speech, with LiveKit carrying the audio. Turn latency was multiple seconds. We assumed LiveKit was the cost and nearly dropped it. Live testing proved the opposite. The tax was the pipeline itself. Every turn paid for four stages in a row: speech-to-text, turn detection, the LLM, then text-to-speech.

Moving to Gemini Live's native audio collapses all of that into a single streaming exchange, and we kept LiveKit purely as transport. Its client SDK (echo cancellation, jitter buffering, reconnect) was worth more than the hand-rolled WebSocket path was becoming. One transport now serves both room types.

Who is speaking. On a shared huddle stream, every participant's mic feeds one audio queue, and LiveRequestQueue.send_realtime() carries audio only, with no speaker field. The model can't tell who said what. We solved it by re-sending the room roster and current speaker as an interleaved text turn on every speaker change. The same channel carries the team/project briefing the agent leans on each turn.

Event-loop-bound clients. The Firestore client binds to whatever event loop is running at import time, which isn't the loop a LiveKit job runs on. Every first Firestore read from inside a job crashed with "attached to a different loop" until we rebound the client at the top of each job's entrypoint.

Keeping the spec honest. Midway through, our written spec had drifted so far from the running code that we rewrote it from scratch as a description of what actually exists, not a wishlist.

What we learned

• With modern realtime voice models, the pipeline is the latency. Native audio isn't just simpler, it's the difference between a conversation and a walkie-talkie.
• Persistent truth belongs in the data model, not the conversation. We wipe conversation history on every room join, and what the team has established always carries over because it was never stored in the transcript.
• A shared audio stream needs an explicit identity channel. The model won't infer it.

What's next

The directions we're most excited about all serve the same goal: truth that reflects reality more faithfully, with less effort from anyone.

• Let anyone correct the agent directly. When Huddle gets something wrong, you should be able to fix it in place, the way you'd correct a teammate, instead of working around it.
• Keep truth current even when the team goes quiet. Huddle should keep the project's state sharp on its own during the stretches between gatherings, so nothing goes stale while everyone is heads-down.
• Greet each person with what moved while they were away. A short, personal catch-up on return: here's what changed, and here's the part that's yours.

Further out, we want reconciliation to scale to a real project's worth of history, and roll-ups that hold truth together across large, nested projects.

Built With

• cloud-run
• fastapi
• google-adk-(agent-development-kit)
• google-cloud
• google-cloud-firestore
• google-gemini-live-api
• google-oauth
• livekit
• livekit-agents
• livekit-client
• pinia
• pydantic
• python
• server-sent-events
• silero-vad
• uvicorn
• vertex-ai-/-google-genai
• vite
• vue-3
• vue-router
• webrtc