Bee Call

Say a half-formed idea to Bee. Bee Call phones you to think it through, then saves what you agree to back into Bee.

Comment

Inspiration

I think best out loud, usually on a walk, and research points the same way (on a phone, speaking is about three times faster than typing, and walking helps people come up with more ideas). I use Bee on my Apple Watch (the Bee wearable ships only in the US). It keeps what I say with a summary, facts and suggested to-dos, but a note doesn't talk back. What I wanted was to talk my ideas through out loud with someone who already knows what Bee knows.

Before building my own, I tried two existing voice agents I could have connected to Bee. OpenAI's GPT-Live is a full-duplex voice model, so it sounded very fluent, but its answers often missed the point. OpenClaw's Talk mode did not feel finished enough for daily use. So I built my own voice agent on Claude, a frontier model. It thinks things through with me over a call that rings like a normal phone call, starts from what Bee knows about my day, and, with my OK, saves what we worked out back into Bee.

What it does

Bee Call architecture

Say a half-formed idea to Bee during the day, and Bee Call may offer a call about it. A meta agent reads each new Bee record and decides whether a call would help. If so, it sends one notification to the Bee Call app on my iPhone, with buttons for the suggested time, now, or not now. Nothing rings without a yes, offers stop at four a day, and none arrive at night. I can also call from the Bee Call app whenever I want.

The phone rings like any other call, on the iPhone's own call screen, even when it is locked. The AI's first words are about why it called, from a short prep note written in the seconds before the call.

On the call, the AI keeps its turns short and stops when I talk. It waits through my pauses and can look things up in Bee. Slow work such as research or a comparison goes to a background agent with web search, page reading and code execution while we keep talking. The AI tells me the answer when it arrives, and the report and charts appear in the Bee Call app, even if I hang up first. When I ask, it also books a call back for later.

Near the end, the AI asks once whether to put what came up into Bee. After a clear yes, the new facts and to-dos are saved, and "take that back" undoes a change made in the call. When I ask, it can also correct or delete a fact, complete a to-do, or go through the last few days of Bee records and save what matters as facts or to-dos.

The Bee Call app shows each call as a thread with live captions, cards for background tasks, report pages and a card for every Bee change. Each person picks English or Korean, a voice and the models. Bee Call also works without Bee, and then it reads and writes nothing there.

How I built it

The iPhone app is native Swift, using CallKit to ring and answer, VoIP pushes to ring while the app is closed, and the LiveKit Swift SDK for the call itself. Where iOS already had a control, such as the call screen or notification buttons, I used it instead of drawing my own.

The server is Python on LiveKit Agents, on one AWS EC2 instance in Sydney. LiveKit Cloud carries the audio, Amazon Transcribe turns speech into text, and ElevenLabs speaks, with Amazon Polly when no ElevenLabs key is set. The models run on Anthropic's API, since Bedrock would not serve Claude to my account (see Challenges), and GLM 5 on Amazon Bedrock takes over when Anthropic's API fails.

Each agent gets the model that fits its job. The call AI has to answer quickly, so it runs on Claude Haiku 5.5 with only light thinking. Turning thinking off was a little faster, but then Haiku sometimes said it had booked a call without booking it. Claude Sonnet 5.5 is an option for deeper answers. The prep note is written by Claude Haiku 5.5 with thinking off, so the first words come quickly. The meta agent decides between calls, when nobody is waiting, so it runs on Claude Sonnet 5.5 with more thinking. The background agent does research that can take minutes, so it runs on Claude Sonnet 5.5 with the most thinking, or on GLM 5 in an Amazon Bedrock AgentCore harness.

Bee is reached only through the official Bee CLI, so Bee Call works the same whichever device does the recording. Each person's Bee login sits in a folder of its own. To read, a watcher (the Bee link in the figure) follows bee stream. Once a burst of events settles, it asks bee changed with a saved cursor for everything new and wakes the meta agent. The meta agent, the call prep and the call AI share twelve read tools, each a single CLI read with Bee's own parameters, covering today, conversations, transcripts, search, daily summaries, facts, to-dos, voice notes and insights. The watch also records my calls with Bee Call, so the read tools leave out what Bee recorded during them. To write, Bee Call uses the CLI's fact and to-do commands, only after a clear yes or a clear request in a call. Each write is logged with the AI's question and my answer, word for word. Bee has no restore, so undoing a delete recreates the fact from the copy Bee Call kept before deleting it.

The token server is the app's API, for sign-in, call tokens, offers and call pages. One server serves several people, each signed in with Apple and kept in a folder of their own. So far Bee Call runs as a TestFlight build. I use it for my own calls, and a few other people have tried it too. I built it with Claude Code as my coding agent, with fresh review agents checking larger changes before I deployed them.

Challenges I ran into

Claude on Bedrock. My new AWS account could not call Claude Sonnet 5.5 on Bedrock. Even after its model agreement went through and every availability check said yes, every call returned "not available for this account". I moved Claude to Anthropic's API and kept GLM 5 on Bedrock as the model standing by.

Making it feel like a real phone call. Speed came first. Besides the fast model, Bee Call now learns how long I usually pause mid-sentence and waits that long before treating my turn as over. With a fixed short wait, one of my answers was cut into five turns. Then the voice. I chose ElevenLabs for voices that sound like a person, and the AI may add a sound such as [laughs] or [sighs] where a person would, which the voice performs instead of reading out. The instructions tell the model that everything it writes is spoken on a phone, so it keeps turns short, says at most three things, rounds numbers and never reads out lists. Last, noise. Sounds around me sometimes reached speech recognition as a short word like "um", and the AI stopped talking mid-sentence. A noise filter now cleans my audio before recognition, and transcripts from audio quieter than my real speech are ignored.

Long, loosely labeled records. One watch recording stayed open as a single conversation for 16 days, and when I was alone in a room, Bee labeled some of my lines "Unknown", so its labels cannot tell my words from someone else's, and Bee Call asks me before it saves anything about me. The watch also records my side of every Bee Call call, and Bee Call must not read those lines back as new notes. My first filter dropped any Bee conversation that overlapped a call, and because that conversation stayed open for 16 days, that threw away all 16 days. Now the read tools drop only the lines spoken during a call.

Accomplishments

Every part runs on real infrastructure. Bee records on my watch, the server runs on AWS, and I talk to Bee Call on my iPhone through a TestFlight build. I use it for my own think-out-loud calls and tuned it by play-testing it on real calls, such as how long it waits through a pause, what makes it stop talking, and when it asks to save something in Bee.

The yes-and-undo loop holds up. In scripted test calls against a stand-in Bee, the AI wrote to Bee after all 18 clear yeses and after none of 27 answers that were not a yes, such as "hold off on that for now", and undo worked in 18 of 18 tries.

I also built two things for other Bee developers. A pull request to the official Bee CLI (https://github.com/bee-computer/bee-cli/pull/26) makes bee status finish a --no-wait login once the user approves it in the Bee app. Without it, an agent that polls bee status, as the CLI's own hint suggests, never sees the login finish. Several of its new tests fail on the current main branch, which shows the bug, and all of them pass with the fix. And I rewrote Bee Call's Bee layer as a new, general Python package, bee-agent-tools (https://github.com/leeminseung/bee-agent-tools), with an asyncio wrapper for the Bee CLI, twelve read tools for a model, logged writes with undo, and a watcher for new records. Apart from the Bee CLI, it needs only the Python standard library.

What I learned

A phone call is less forgiving than a chat. A slow answer, a wrong guess about when I finished, or one word misheard from noise stands out at once, so much of the work went into timing and voice. Deciding whether something I said counts as a yes or a goodbye also works better in the model's instructions, with examples, than in code that matches words, and undo is what makes trusting the model affordable. Wearable memory is messier than it looks, too. Conversations run for days, labels can be wrong and the wearable hears the agent's own calls, so Bee Call asks me to confirm what it keeps.

What's next

I plan to keep using and improving Bee Call after the hackathon. These are the next items on my list.

  • Making calls feel even more like talking to a person, with short "mm-hm" responses and natural interjections. One path is a full-duplex voice model. The other is more engineering around a frontier text model, so it keeps its reasoning and still sounds like a person on the phone.
  • Getting the agent better at asking about and drawing out what matters most to me, such as lasting facts about my goals, habits and preferences, through better instructions.
  • Waking the meta agent once when a long conversation ends, instead of each time Bee updates its summary.
  • Finding facts that overlap or conflict and asking in the next call whether to merge them.
  • Transcribing sensitive calls on the iPhone, so their audio never leaves it.

Built With

Share this project:

Updates

Submission history