RecallRay

Inspiration

“I’ll send you that link.” “I’ll call them tomorrow.” “Mike said he’d get back to me.”

Conversations are full of small commitments. Getting them into a task manager takes another step, and it’s easy to forget. We wanted to see whether ordinary conversation could create a useful record of what still needs to happen—and whether a later conversation could close those commitments automatically.

What it does

RecallRay records a short conversation, transcribes it locally, and finds unfinished commitments. It separates things you need to do from things you’re waiting on someone else to do.

Later conversations can update that memory. Saying “I sent Sarah that article” can resolve an earlier promise to Sarah, with the new evidence saved alongside it.

For commitments that need current information, Help me finish lets Nemotron decide whether to search. Tavily retrieves sources, and Nemotron uses them to draft a response. The commitment stays open until a later conversation confirms it was completed.

How we built it

RecallRay is a Rust application with a localhost web UI and SQLite storage. The browser handles microphone recording, and local faster-whisper converts the audio into an editable transcript.

We send the transcript, active commitments, and recent conversation context to NVIDIA Nemotron through Nebius Token Factory. Nemotron proposes structured operations to create, update, keep, or resolve commitments. Rust validates those operations before saving them. A separate Nemotron call checks proposed resolutions.

We kept the memory system small: SQLite stores the conversations, commitments, evidence history, and assistance results. Tavily is the only external search tool.

Challenges we ran into

The live demo exposed how much depends on transcription. Whisper heard “I sense Sarah” instead of “I sent Sarah,” and the commitment stayed open. It also struggled with the name “Jetson Orin Nano.”

We had to slow down the spoken audio, fix recording timing, and review the transcript before submitting it. That review step matters: a small transcription error can change whether something sounds finished.

We also had to distinguish preparing an answer from completing a promise. Finding a price and drafting a message doesn’t mean the user sent it.

Accomplishments that we're proud of

We demonstrated the full loop with real microphone recordings and live API calls. One conversation created Sarah’s commitment. A later conversation resolved it while Mike’s remained waiting.

We also ran live search for Ben’s request, saved the draft and sources, and resolved his commitment from a later spoken update. The evidence history makes those changes inspectable.

What we learned

For this version, sending Nemotron all active commitments was enough. We could test the central idea without embeddings or a vector database.

We also learned to test the whole path early. Correct reasoning depends on the words that reach the model, so microphone timing and transcript review deserve as much attention as the reasoning prompt.

What's next for RecallRay

The next step is improving transcription reliability, especially for names and product terms, and making corrections easier to carry into existing commitments.

We’d also like to test less scripted conversations: changed plans, vague references, partial completion, and contradictory updates. Before adding more tools, we want stronger evidence that RecallRay consistently keeps the right commitments open and closes the right ones.

Built With

• codex
• javascript
• python
• rust