Inspiration

This year's theme is time, and a voice call lives in time. So does its cost: on every turn the agent resends everything said so far, so the longer you talk, the more tokens, money and waiting each reply needs. The usual fix is to drop old messages, but then the agent forgets what matters, like an allergy mentioned at the start. We wanted memory that forgets well over time: let the small talk fade, never lose the facts that matter, and don't spend another LLM call to decide.

What it does

It sits between an ElevenLabs voice agent and the language model and decides, on every turn, what the model gets to see.

Each message is scored with plain maths: similarity to the current question, how recent it is, how important it looks, and whether later messages refer back to it. As a message ages it is kept in full, shrunk to a one-sentence summary, cut to a short gist, or left out of that turn's prompt. If a later question matches it, it comes back. Critical facts (health, allergies, safety) go into a pinned lane on top of the token budget, so they are never squeezed out. A live dashboard shows what was kept, shrunk and forgotten for every message.

There is no language model anywhere in the memory loop, so it is free, repeatable and runs locally.

How we built it A FastAPI proxy with an OpenAI-compatible streaming endpoint that ElevenLabs calls as a custom LLM, in front of Gemma on the Gemini API. A memory manager in Python: MiniLM sentence embeddings for similarity, pattern matching for importance, and a small logistic regression that spots personal facts by meaning. A token budget that grows with the length of the call, with a protected lane for pinned facts. Several copies of each model request raced, so a hung request does not stall the call. A Streamlit dashboard, a benchmark harness with planted facts and recall questions, and a test suite. Challenges we ran into The model API was overloaded and hung at random, and Gemma "thinks out loud" by default, which broke voice replies. Racing requests and turning thinking off fixed it. A fixed token cap squeezed out the facts we most wanted to keep. We added the pinned lane, then a budget that grows with the call but eases into a ceiling, so we never send more than we save. Exact-word rules missed paraphrases such as "shellfish puts me in hospital". A small learned fact detector fixed most of that. Measuring honestly. Time to first token was lower on average in our 50-turn runs, but the effect is small and weakened in a longer 107-turn test, so we call it suggestive. Accomplishments that we're proud of A working live voice demo: we say we have a heart condition, talk about unrelated things for many turns, and the agent still uses it when we ask about exercise. On 208 real ChatGPT conversations of 51 or more turns, each with a critical fact (an allergy or a medication) and an ordinary fact planted in it: 88 percent fewer tokens on average (95 percent interval 85 to 90 percent), and 97.6 percent of critical facts kept, against 1.9 percent for a recent-message window at the same token cost. Ordinary facts were kept 77.9 percent of the time, which is the trade-off. On a 50-turn conversation with a real model, three runs per mode: 58 percent fewer tokens than sending everything, and the same 5 out of 5 recall (also 5 out of 5 on reworded questions). A plain last-20-messages window scored 0 out of 5 in our own run. On a 107-turn conversation of 100 real questions, about half the tokens: roughly 175,000 against 350,000. In offline tests, we kept 247 of 247 critical facts. A plain recent-messages window at the same token cost kept 4 percent, 0.5 percent and 0 percent of planted facts at 50, 100 and 200 prompts, against our 100, 100 and 93.5 percent. At 200 prompts the learned detector kept 100 percent of critical facts, against 33 percent for exact-word rules alone. No LLM in the loop, about 100 ms per turn, and every decision visible on the dashboard. What we learned The system prompt counts towards the budget, so a voice agent's budget is much tighter than it looks. Deterministic memory is easier to test, explain and trust than memory that depends on another model call. Report what you can back up, including results that are small or neutral. What's next for Deterministic Memory for Voice Agents Our large tests check that a fact is in the prompt, not that the model then answers correctly, and the real chats are typed, not voice. About 1 in 40 real chats lost its critical fact. Live recall at longer lengths, and on real voice transcripts, is the next test. Run on conversations we did not tune against, including other phrasings and languages. At 200 prompts, about 1 in 4 paraphrased facts is still never recognised as a fact. Learn the scoring weights from data instead of setting them by hand. Run the proxy in the cloud instead of on a laptop.