Inspiration
Bee is great at remembering what happened. But in everyday conversations, the important part is often not the event itself. It's the thing you still need to do afterward. We say things all the time like "I need to send that later," "I promised I'd follow up," or "they said they'd get back to me." Most of those things never make it into a task manager.
RecallRay started with a simple question:
What if Bee could tell you not just what happened, but what still needs to happen?
What it does
RecallRay reads conversations captured by Bee and looks for unresolved commitments, or what we call "open loops."
It sorts them into a few simple states:
- Open - something I still need to do
- Waiting - something another person owes me or I'm waiting on
- Resolved - something that was clearly completed
- Ignore - normal conversation that is not actionable
For example, if I say:
"I promised Sarah I'd send her the article."
RecallRay creates an open item.
If I say:
"I already finished updating the documentation."
RecallRay recognizes that it is already done and doesn't add it to the active list.
If I say:
"Mike said he would send me the contract, so I'm waiting on him."
RecallRay puts it under Waiting On.
Every item keeps the original Bee utterance as evidence. If you wonder why something showed up, you can see exactly what you said.
How we built it
RecallRay uses Bee as the source of conversational memory.
We use the Bee developer API and local proxy to retrieve completed conversations along with utterances, timestamps, speakers, and conversation IDs.
Each utterance is then evaluated using TypeSafe AI's Jev model. Instead of asking a general-purpose LLM to rewrite or summarize the whole conversation, we use Jev to answer a few specific questions:
- Is this actionable?
- Is it open, waiting, resolved, or irrelevant?
- Who is responsible for the next action?
- How confident are we about that classification?
We keep the original Bee utterance as the source of truth and store the extracted items locally in SQLite.
The web interface has two main sections:
Open Loops shows things the user still needs to do.
Waiting On shows things that depend on somebody else.
The current version uses manual synchronization with Bee because it keeps the processing pipeline simple and easy to debug. Later, we can use Bee's streaming events to update RecallRay much closer to real time.
Challenges we ran into
The hardest part was figuring out what actually counts as an unresolved commitment.
Simple keyword matching doesn't work very well. These two sentences look very similar:
"I need to update the documentation."
"I already finished updating the documentation."
But one is a task and the other is something that is already done.
We also had to figure out who owns the next action. "I need to send Mike the contract" should go into my open tasks. "Mike said he would send me the contract" should go into my waiting list.
Another challenge was resisting the urge to summarize everything. We found that RecallRay works better when it stays close to the exact Bee utterance instead of generating a cleaner version that might accidentally change the meaning.
Accomplishments that we're proud of
We built the whole pipeline using real Bee conversations captured from an Apple Watch instead of mocked transcript data.
RecallRay can distinguish between open, resolved, and waiting commitments in normal speech.
In our first test conversation, it correctly found four open tasks, ignored a task that had already been completed, and identified one item that was waiting on somebody else.
We're also happy that every RecallRay item can be traced back to the original Bee conversation. It isn't just an AI-generated task list. You can see exactly why each item exists.
What we learned
We learned that conversational memory and task management are related, but they are not the same thing.
A transcript tells you what was said. A summary tells you what seemed important. Neither one necessarily tells you what is still unresolved.
We also found that a bunch of small, structured AI decisions can be more useful than asking one large model to generate an entire answer. Keeping classification separate from presentation helps RecallRay stay grounded in what the user actually said.
Bee's developer tools also made us realize that real-world conversations can become a useful source of structured data. Once you have speech broken into speakers, timestamps, and events, you can build a lot of interesting personal software on top of it.
What's next for RecallRay
The next step is real-time processing using Bee's event stream, so open loops can show up shortly after a conversation happens.
We also want to add lifecycle tracking so RecallRay can notice when something said later resolves an earlier commitment.
Other things we want to add:
- reminders for old or forgotten open loops
- grouping commitments by person or project
- detecting deadlines and dates mentioned in conversation
- turning selected RecallRay items into Bee todos
- identifying recurring commitments
- questions like "What am I still waiting on from Mike?"
- tracking how long an open loop has been unresolved
Long term, we want RecallRay to be a lightweight memory layer for commitments:
Bee remembers what happened. RecallRay remembers what still needs to happen.
Built With
- codex
- python
