you had to be there...

A real-time voice AI hidden inside a musician's album campaign. You dial in. She picks up. She knows the story. She remembers you. Built with Gemini Live API in one day.

Comment

Inspiration

My album You Had to Be There is about leaving places you loved before you understood what they meant to you. A few months ago I built youhadtobethere.life — a Japanese payphone-inspired site where fans dial numbers to hear exclusive songs and voicemails. The music doesn't exist anywhere else.

But I kept running into the same problem: the album is deeply personal, and I didn't want the experience to feel self-indulgent. I wanted listeners thinking about the music in the context of their own lives, not mine. The solution was an ARG — a puzzle where the listener has to engage to get the full picture, discovering the story on their own terms. The problem with an ARG is that without a guide, it's just chaos. Half the people who find it have no idea what they're looking at. They leave.

I needed someone to answer the phone. A voice that could orient a new caller, hand over the right numbers, ask the right questions, and remember where they left off. A guide that collected what it needed to know about you naturally — the way a person would, not a form.

The Gemini Live Agent Challenge gave me the deadline I needed to stop going in circles and ship it.

What It Does

The YHTBT Phone Booth adds a live voice agent to the existing phone directory at youhadtobethere.life. Every number in the directory (101–125) plays pre-recorded audio. Number 100 is different. You dial it, and Aza picks up.

Aza Fre is a fully developed character from the Surf Noir world — she found out about Cam through his travel videos, has been reaching out to connect, and now recruits callers into the directory experience. She's not a bot you type at. She's a phone call you have.

She guides callers through 11 narrative cycles, each unlocking specific numbers to dial. The cycles branch based on what the caller felt when they listened — there are no wrong answers. Her branching logic responds to interpretation, not comprehension. She remembers returning callers across sessions: which cycles they've completed, which branch they took, where they left off. Call back the next day and she skips the intro and picks up the conversation where it ended.

The entire experience runs in real time. Interruptible. Conversational. No noticeable lag.

How We Built It

Stack: Python, Google Agent Development Kit (ADK), Gemini Live API (native audio model gemini-2.5-flash-native-audio-preview), FastAPI, Google Cloud Run, Google Cloud Firestore, Docker. Frontend integration via Base44 (the existing phone site).

Agent architecture: Aza's character is loaded as a continuous prose system prompt — no bullet points, no structured headers. This was a deliberate choice based on prior testing: structured prompts flatten voice quality. The model executes rules but loses rhythm. Prose produces character. The character layer, navigation layer, and voicemail knowledge are loaded from separate .txt files and combined at runtime, making each layer independently iterable.

Session memory: A Firestore integration (caller_store.py) tracks each caller by anonymous UUID — cycles completed, branches taken, numbers dialed, whether they've heard the ending. This context is injected into the ADK session before Aza speaks, so she knows who she's talking to before the first word.

Server: A custom FastAPI server replaced the ADK dev tool. It accepts WebSocket connections, fetches Firestore history, injects session context, and manages bidirectional audio streaming via two concurrent async tasks — one upstream (mic audio in), one downstream (agent audio out).

Deployment: Containerized with Docker, deployed to Cloud Run with session affinity to keep each caller's WebSocket on the same instance. Infrastructure-as-code deploy script (deploy.sh) automates the full deployment: enabling GCP APIs, granting IAM roles, creating Firestore, deploying the service.

Frontend integration: A LiveCallManager component on the Base44 phone site handles mic capture via AudioWorklet, WebSocket streaming, and audio playback queue. Number 100 routes to the live agent. All other numbers continue as pre-recorded audio.

Challenges We Ran Into

Latency vs. accuracy tradeoff. Setting thinking_budget=0 to reduce response latency made navigation worse — the model couldn't reliably track the cycle map without any reasoning time. Calibrated to 256 tokens as the sweet spot: enough to reason through the navigation layer, not enough to add perceptible delay.

Navigation reliability. The first navigation layer used prose descriptions of each cycle. The model kept mixing up which numbers belonged to which cycle. Rewrote it as a strict ordered list with explicit rules ("CRITICAL RULE: cycles in exact order, never skip, never combine"). Fixed completely after the rewrite.

Three frontend integration bugs:

  1. ADK event payloads use camelCase (inlineData, mimeType) not the snake_case Python conventions — the event handler had to be updated to check both formats.
  2. Modern browsers suspend AudioContext until a user gesture. Aza's audio wouldn't play until an explicit AudioContext.resume() call was wired to the dial button click.
  3. Gemini Live API sends URL-safe Base64 (using - and _ instead of + and /). JavaScript's native atob() can't decode it. Added a conversion step before decoding.

Voice quality. Initial audio was slightly crunchy. Tracked it to a sample rate mismatch — the playback AudioContext was set to 24kHz but Gemini's native audio model outputs at 16kHz. Setting new AudioContext({ sampleRate: 16000 }) fixed it immediately.

Accomplishments That We're Proud Of

The character. Aza was designed by a narrative team before a single line of code was written. She has a specific voice, specific knowledge, specific things she will and won't say. When the first real test call came through the phone site and she responded in character — oriented, warm, already recruiting — it felt less like a demo working and more like a person picking up.

The fact that this is a real campaign with a real audience. Most hackathon voice agents are standalone demos asking judges to imagine what the product would feel like in the world. This one is already in the world. People are dialing in. The agent is the live layer on top of something that's already running.

Shipping the whole thing — agent, server, Cloud Run deployment, Firestore integration, frontend wiring, infrastructure-as-code script — in one day.

What We Learned

Prose beats structure for voice quality. Structured prompts produce compliant agents. Prose produces characters. If you want a model to sound like a specific person, describe that person to it. Don't instruct it.

The thinking budget is a real dial. Turning it to zero for latency savings had a direct, measurable cost to navigation accuracy. Understanding the tradeoff and calibrating it intentionally matters more than optimizing for one axis.

Ship the bridge, not the perfect version. I'd been going in circles on this campaign for six months trying to make it complete. The agent didn't complete it — it closed one loop and opened the next one. That's how it's supposed to work.

What's Next for You Had to Be There...

The album drops April 3, 2026. The full ARG runs the length of the campaign — two floors of puzzles, the second of which won't resolve until the final track releases. Aza is the live spine of it.

After the campaign, the agent architecture becomes infrastructure for Surf Noir Studio — the transmedia world the album lives inside. Every character in that world has a fully developed POV document. The phone booth proved the technology works. The next step is putting the rest of the cast on the line.

Built With

  • base44
  • docker
  • fastapi
  • gemini-live-api
  • google-agent-development-kit-(adk)
  • google-cloud-firestore
  • google-cloud-run
  • python
  • websockets
Share this project:

Updates

Submission history