Akira — project overview

An AI phone agent that calls companies and negotiates on the user's behalf: lower a bill, waive a fee, cancel a service. The user states a goal, Akira places a real outbound call, talks to a real human rep, and streams the whole thing to a live dashboard.

Hackathon project. ~2,150 lines of hand-written code across a Python backend and a Next.js frontend.

The one-sentence pitch

Akira is a relay: it sits between a live phone call and a speech-to-speech model, and forks everything interesting to a browser so you can watch the negotiation happen.

Why it's technically interesting

Three things are genuinely hard here, and all three are solved and verified on real calls:

Barge-in. Twilio buffers audio you send it. When the rep interrupts, a naive agent keeps talking into their ear for seconds. Akira clears the buffer and tells the model how much of its sentence was actually heard, so the conversation doesn't silently drift.
Phone-tree navigation. On a bidirectional Twilio stream there is no "press a key" API. Akira synthesizes the actual DTMF dual-tone signal from scratch, mu-law encodes it, and sends it as ordinary audio.
Watching a speech-to-speech model think. There's no visible reasoning step in speech-to-speech. The tool-call stream is the reasoning display.

Architecture

Browser (Next.js)
    |  WebSocket  /ws/dashboard/{call_id}      (transcript, tools, barge-ins, kill switch). We hosted our Websocket on render instead of exposing it on the local Zoho network. We did this because there was a lot of traffic interfering with the websocket, and use render to communicate to twilio.
    v
FastAPI backend  <----- REST -----> Twilio REST API (place call, hang up)
    ^                                        |
    |  WebSocket  /ws/twilio  <--------------+  (Twilio Media Stream, bidirectional)
    |
    +--- WebSocket  wss://api.openai.com/v1/realtime   (GPT Realtime, speech-to-speech)

The backend owns three sockets per call. Audio flows Twilio → OpenAI → Twilio with no transcoding in either direction, because both ends speak G.711 mu-law at 8 kHz.

There is no separate STT or TTS service. No Deepgram, no Whisper-as-transcriber in the audio path, no ElevenLabs. One Realtime session does speech-to-speech end to end. That's what keeps latency conversational.

Call lifecycle

POST /calls — creates an in-memory CallSession, calls the Twilio REST API.
Twilio dials out; on answer it fetches POST /twiml/voice, which returns pointing at our public wss:// URL. The internal call_id rides along as a so the socket can find its call record.
Twilio opens /ws/twilio. The start frame carries streamSid — required on every outbound frame or Twilio drops it silently.
The bridge opens the OpenAI Realtime socket, configures the session, and asks the model to speak first.
Two asyncio tasks run concurrently under asyncio.wait(FIRST_COMPLETED). Whichever finishes first tears down the other.
On teardown, a markdown transcript is written to call_logs/.

Stack

Layer	Choice	Notes
Backend	Python 3.11+, FastAPI, uvicorn	Fully async; nothing slow ever awaits in the audio path
WebSockets	websockets 14.1	Client to OpenAI; FastAPI's own for Twilio + dashboard
Telephony	Twilio Programmable Voice + Media Streams, twilio 9.4.1	Bidirectional
Voice model	OpenAI Realtime, gpt-realtime-2.1, voice cedar	Speech-to-speech, semantic_vad turn detection
Audio codec	G.711 mu-law audio/pcmu), 8 kHz, 20 ms frames	Same on both ends — zero transcoding
Research	OpenAI Responses API, gpt-5-search-api	Built, not yet wired into the call flow
Frontend	Next.js 16.3, React 19.2, TypeScript, Tailwind v4	App Router, two pages
HTTP client	httpx 0.28	Async, used by research
Tunnel	ngrok	Twilio requires a public HTTPS/WSS endpoint
Storage	An in-memory dict	No database. It's a hackathon.

Dev ports are 8001 (backend) and 3001 (frontend) — 8000 and 3000 are taken by unrelated services on the dev machine.

Repo layout

backend/
  main.py               FastAPI app, CORS, /health, route registration          40 loc
  config.py             Env loading, URL builders, missing-config check         49
  state.py              CallSession dataclass + in-memory registry + fan-out    88
  transcript.py         Markdown transcript writer, per-call + master log       97
  routes/
    calls.py            POST /calls, TwiML webhooks, status callbacks, hangup  185
    ws_twilio.py        Twilio media stream socket, echo + ai modes            112
    ws_dashboard.py     Dashboard fan-out socket                                52
  realtime/
    bridge.py           Audio relay, barge-in, playback clock, tool dispatch   298
    session.py          OpenAI WS connect + nested GA session.update           117
    tools.py            Tool schemas and handlers                              106
    dtmf.py             DTMF synthesis + G.711 mu-law codec, written by hand   109
    prompts.py          Disclosure, style, opening + greeting instructions      70
  research/
    brief.py            Pre-call research via Responses API                    120
frontend/
  app/page.tsx          Goal input + call launcher                              83
  app/calls/[id]/page.tsx  Live transcript, phase pills, tool feed, kill switch 228
  lib/ws.ts             Dashboard socket client + REST helpers                  47
scripts/
  check_realtime.py     Preflight: does OpenAI accept our session shape?         84
  check_dtmf.py         Goertzel-decodes generated tones, all 12 digits          86
  fake_twilio.py        Impersonates a Twilio media stream — test for free      144
call_logs/              Auto-written markdown transcripts, one per call

The four pieces worth demoing

1. Barge-in and truncation — realtime/bridge.py

When input_audio_buffer.speech_started fires, three things happen in this order:

Send {"event": "clear", "streamSid": sid} — Twilio drops its buffered audio, so Akira stops talking into the rep's ear.
Send conversation.item.truncate with audio_end_ms set to how much audio actually played. Skipping this is the subtle failure: audio stops so it looks fixed, but the model still believes it finished the sentence and the conversation quietly desyncs.
Reset the local playback tracker.

The playback clock is the non-obvious part. Counting bytes sent to Twilio overestimates playback position, because Twilio buffers. So the Playback class instead uses the timestamp on inbound Twilio frames as a real-time wall clock, and caps that by bytes actually sent (truncating past the end of real audio is an API error):

elapsed = self.latest_media_ts - self.response_start_ts   # real time
sent_ms = self.sent_bytes // 8                             # mu-law: 8 bytes per ms
return max(0, min(elapsed, sent_ms))

Verified on a real call: 5 clean truncations from 140 ms to 1860 ms, zero errors.

2. DTMF synthesis — realtime/dtmf.py

The constraint: on a bidirectional , the only frames you may send back to Twilio are media, mark, and clear. There is no outbound DTMF event. You cannot ask Twilio to press a key.

So Akira generates the tone itself — two summed sine waves (a row frequency and a column frequency off the keypad), 100 ms of tone plus a 50 ms gap, hand-rolled G.711 mu-law encoding, sliced into 20 ms frames and sent as ordinary media. Amplitude is kept at 8000 of full scale, because two summed sines clip easily and a clipped tone fails detection.

scripts/check_dtmf.py runs a Goertzel detector over the output — the same algorithm an IVR uses — and confirms all 12 digits decode back correctly.

The model decides when to press, via a press_digits(digits, reason) tool. No regex on the transcript, no keyword-matching for menus.

3. The dashboard fan-out — state.py + routes/ws_dashboard.py

Each connected browser gets its own asyncio.Queue (depth 256). The audio relay writes with put_nowait and never awaits a browser. If a queue fills, frames are dropped:

A dashboard that cannot keep up loses frames; the phone call does not.

A late-joining browser gets a snapshot message replaying the call so far, so the page is never blank. The frontend reconstructs streaming turns by searching backwards for the open partial — the rep can start talking while Akira is still streaming, which puts their bubble in between.

4. Kill switch — routes/calls.py

Routes through the Twilio REST API calls(sid).update(status="completed")), not by closing our own socket — on a bidirectional stream, ending the call is the only way to stop it.

Measured at 0.261 s round-trip. Target was under 1 s.

Critical API details (the things that break builds)

The Realtime session config uses the nested GA shape. The old beta format with flat input_audio_format / output_audio_format / modalities keys is rejected outright. Most tutorials online still show the old shape.

{"type": "session.update", "session": {
    "type": "realtime", "model": "gpt-realtime-2.1",
    "output_modalities": ["audio"],
    "audio": {
      "input":  {"format": {"type": "audio/pcmu"},
                 "turn_detection": {"type": "semantic_vad", "eagerness": "low"},
                 "transcription": {"model": "whisper-1"}},
      "output": {"format": {"type": "audio/pcmu"}, "voice": "cedar"}},
    "instructions": ..., "tools": [...], "tool_choice": "auto"}}

scripts/check_realtime.py sends the real payload function, not a copy of it, so the preflight check and the live call can never drift apart.

Two settings that took tuning:

eagerness: "low" — at the default, a cough, a line click, or the model's own echo triggers an interrupt, and a greeting that keeps getting cut restarts from the top forever.
transcription: {"model": "whisper-1"} — without it the model still hears the caller fine, but conversation.item.input_audio_transcription.completed never fires and the rep's half of the dashboard transcript stays empty. (This is transcription for display only; it is not in the audio path.)

Event names that changed: audio arrives on response.output_audio.delta. response.audio.delta is the beta name and never fires. response.done carries transcripts and tool calls, never audio bytes.

voice is locked* once the model emits audio. Set it up front. Everything else in the session can be updated mid-call — which is how phase switching will work.

Design decisions worth defending

Disclosure is built in at the prompt level and non-negotiable. Akira identifies itself as an AI in its first turn. But the phrasing was tuned: "I'm an automated assistant calling on behalf of the account holder" is legal boilerplate that sounds like a robot reading a disclaimer. The prompt instead asks for a casual, in-passing heads-up — "quick heads up, I'm an AI assistant calling for the account holder" — said once, not repeated every turn. If asked directly whether it's human, it answers immediately and without hedging. Built in from Stage 3 so it could never become an afterthought.

Prompt style is engineered against formality. Speech-to-speech models drift formal, and formal reads as robotic on a phone call. The style block enforces contractions, one idea per turn, everyday words, and — importantly — "if you're interrupted, do not restart your sentence."

An echo mode is kept in the Twilio socket even though it's obsolete. It bounces inbound audio straight back with zero AI involved. If audio breaks later, re-running one call in echo mode says immediately which half is at fault.

POST /dev/calls creates a call session without dialing.* Combined with scripts/fake_twilio.py, the entire AI bridge can be exercised end to end for free. Realtime audio runs about $4.61/hr of output and debugging against real calls burns it fast.

Every mutation of shared session state is under a per-call asyncio.Lock. Races here show up as garbled audio, which is not debuggable live on stage.

Tool failures never kill the call. dispatch() catches everything and hands the model the error as its tool result.

Build status

Built in nine stages, each demoable on its own, each verified on a real phone call before starting the next.

Stage	What	State
0	Accounts + tunnel	done — Twilio full account + voice number, OpenAI key, model reachable, GA shape accepted
1	Place a call, static TwiML	done, real call — full initiated → ringing → in-progress → completed
2	Media stream echo	done, real call — 7,402 frames echoed over 149 s, caller heard themselves
3	OpenAI speaks into the call	done, real call — model greeted with disclosure, clean audio, zero errors
4	Full duplex conversation	done, real call — real back-and-forth, both transcript directions
5	Barge-in and truncation	done, real call — 5 clean truncations, 140–1860 ms, zero errors
6	Phone tree navigation	done, with a caveat — see below
7	Dashboard	built, verified against a simulated call, not yet a real one
8	Phases and tools	partial — tool plumbing is live end to end; only press_digits is implemented
9	Research and post-call report	research written, not wired in; report not started

What is honestly not done yet

Worth knowing before a judge asks:

Phase switching is not implemented. The three-phase design (opening → midgame → endgame) is specced, and the dashboard renders phase pills, but the backend never emits a phase event and never swaps instructions mid-call. The pills always read "opening".
Only one tool exists. press_digits is live and works. advance_phase, lookup_competitor_offer, record_concession, and end_call are designed but not built. The dispatch machinery is done, so each is a schema plus a handler.
Pre-call research is written but not called. research/brief.py works — it finds the company, its customer-service number, an IVR hint, and leverage bullets, and to_prompt_block() compacts it for prompt injection. Nothing in routes/calls.py calls it yet. It uses the OpenAI Responses API with gpt-5-search-api, not Perplexity or Bright Data as originally planned.
No post-call report. Transcripts are written to call_logs/ as markdown; nothing summarizes them yet.
Stage 6 caveat. The model chooses press_digits correctly and the tones go down the line — verified on a real call. The tones are verified as well-formed DTMF offline by a Goertzel decoder. What has not been tested is a real IVR consuming them, because the project rule forbids dialing a real company. POST /twiml/test-ivr is a menu built for exactly that test: point the Twilio number's voice webhook at it, dial your own number, and Twilio reports the digit it actually detected.

Known issue

Speakerphone causes a feedback loop. The model hears its own voice, semantic_vad fires, and it interrupts itself mid-greeting. Use the handset or headphones when testing. This is not a bug in the relay — but it will ruin a live demo if someone puts the phone on speaker.

Running it

python3 -m venv .venv
.venv/bin/pip install -r requirements.txt
cp .env.example .env      # then fill it in

.venv/bin/uvicorn backend.main:app --port 8001 --reload
ngrok http 8001           # put the host, no scheme, in PUBLIC_HOST
cd frontend && npm run dev -- --port 3001

GET /health reports which settings are still missing.

Preflight — both run with no phone call and no Twilio:

.venv/bin/python scripts/check_realtime.py                              # session shape
.venv/bin/python scripts/fake_twilio.py --url ws://localhost:8001/ws/twilio   # full bridge
.venv/bin/python scripts/check_dtmf.py                                  # tone correctness

Place a call:

curl -X POST localhost:8001/calls -H 'content-type: application/json' \
  -d '{"to":"+1555...","goal":"lower my internet bill","mode":"ai"}'

curl localhost:8001/calls/<call_id>                    # status + recent events
curl -X POST localhost:8001/calls/<call_id>/hangup     # kill switch

Modes: say (static TwiML, Stage 1), echo (audio path only, Stage 2), ai (full bridge).

Test numbers only. Do not point this at a real company's support line.

API surface

Method	Path	Purpose
POST	/calls	Place a call. Body: {to, goal, mode}
POST	/dev/calls	Create a session without dialing — for free testing
GET	/calls	List all sessions
GET	/calls/{id}	Status + last 50 events
POST	/calls/{id}/hangup	Kill switch, via Twilio REST
POST	/twiml/voice	Twilio fetches on answer; returns or
POST	/twiml/status	Twilio status callbacks
POST	/twiml/test-ivr	A phone menu we own, for testing DTMF
GET	/health	Which config is missing
WS	/ws/twilio	Twilio media stream
WS	/ws/dashboard/{id}	Browser event feed

Dashboard event kinds: snapshot, agent_delta, agent_said, rep_said, tool_call, tool_result, barge_in, press_digits, status, realtime_connected, realtime_error, bridge_failed, transcript_written, killed.

Constraints and costs

60-minute cap on a Realtime session. Hold music eats it.
128K context — a long call fits without compaction.
~$1.15/hr audio in, ~$4.61/hr audio out. Testing a voice agent means dozens of calls.
ngrok URL churn — the free tier reissues a URL on every restart, invalidating the webhook config.
Audio format mismatch is the single most common failure mode. Symptoms: chipmunk pitch, static, or silence. Always check audio/pcmu on both ends first.