Cortex

full app view
full app view
brain map

Inspiration

Every phone agent we tried had the same problem: once the call ended, it forgot everything. The next call started from scratch. But computers are good at exactly the opposite. They can store huge amounts of information, keep context over time, and learn from thousands of conversations.

That led us to a simple question: what if an outbound call agent could build its own memory from previous calls, remember individual callers, and improve as it handled more conversations?

We decided to test the idea in a safe and practical setting: pharmacy and clinic medication-adherence check-ins. The agent does not diagnose conditions or provide medical advice. It only listens, records information, and flags relevant patterns for staff.

What it does

Cortex is an outbound CALL-E agent built around two levels of memory.

The first is a private sub-brain for each caller. It stores a short summary of the person, their open follow-ups, previous conversations, and notes such as requests to call back at a later time.

The second is a shared master brain. It stores general, anonymized information learned across callers. This allows something learned from one conversation to potentially help with another caller without sharing personal information.

During a run, Cortex:

Calls a patient and conducts a short adherence check-in.
Converts the conversation into structured memory.
Looks for recurring patterns across different callers.
Requires independent corroboration before treating a new pattern as reliable.
Asks a human to review and approve a learned pattern before using it proactively.
Remembers callback context, such as a previous conversation about being at a wedding, and can follow up on it later.

Why it is more than a call database

A normal database stores information that someone explicitly enters into it. Cortex also manages how much confidence it should have in that information.

When a new fact is discovered, it starts as a candidate. It does not immediately become shared knowledge. The fact must be independently confirmed by two different callers before it becomes canonical.

The same caller repeating something does not count as independent confirmation. This prevents one incorrect or misleading conversation from influencing what the agent tells everyone else.

The system takes information from speech, converts it into structured memory, compares experiences across different callers, turns repeatedly confirmed information into shared knowledge, and then uses that knowledge to improve future conversations.

Without this loop, the system is essentially a call log. With it, previous calls can influence future calls.

Human-in-the-loop

We made an important distinction between learning from conversations and changing what the agent says.

Learning information from callers is relatively safe. Automatically changing the agent's questions or behavior based on that information is not.

When Cortex identifies a potential pattern, it creates a proposed prompt change. A human can review and approve it before the agent starts asking about that pattern.

Strong patterns supported by many independent callers can be configured to apply automatically. Weaker patterns can remain pending until someone approves them. An administrator can also require manual approval for every change and revoke previously approved changes.

The agent only asks proactive questions about patterns that have been approved. If a patient says the pattern does not apply to them, the agent simply reassures them and directs them to contact the pharmacy if they ever have concerns.

How we built it

CALL-E handles the actual outbound calls, including planning, running the call, and providing the transcript.

Gemini processes the transcripts and generates embeddings. We use it for understanding the conversations, while the stored facts and signals rely on deterministic keys. This means two differently worded conversations about the same issue can still be recognized as the same pattern.

SQLite acts as the underlying storage for the system. It contains the caller sub-brains, facts, signals, call history, and system settings.

The Streamlit operator console provides a view of the master brain, individual caller sub-brains, staff alerts, the administrator approval panel, and a live brain map. The brain map shows the master brain, individual caller brains, and the connections between them.

We also built a fail-closed orchestrator around every call. Before a call can be placed, it checks consent, quiet hours, idempotency to prevent duplicate calls, and the overall budget limit.

Challenges we ran into

One of the biggest challenges was getting the system to recognize the same idea when different callers described it in completely different ways.

Initially, allowing the language model to freely describe facts meant that two conversations about the same issue could produce different stored facts. They would then fail to corroborate each other.

We solved this by having Gemini map information to a predefined vocabulary and then generating the stored facts using deterministic templates.

Speech-to-text was another challenge. Live calls are not always transcribed perfectly. Instead of trying to fill in missing information or guess what a caller meant, the system records only what it can confidently understand. This made the memory less complete in some cases, but much more reliable.

We also designed the safety controls so that they fail closed. If a required safety check fails, the call does not happen. Personal information stays inside the individual caller's sub-brain, which also makes deleting someone's data straightforward.

Accomplishments

We tested the complete system using real phone calls.

During the demo, a candidate fact became canonical as soon as a second independent caller confirmed it. The system then generated a staff alert, an administrator approved the resulting pattern, and a completely new caller was later asked about it proactively.

We also demonstrated that the system could remember information from an earlier conversation, including a callback about a wedding.

The total cost of the demo was $0.

What we learned

The most valuable part of the system is not the underlying model. It is the curated memory that builds up over time.

Models will continue to improve, but a collection of corroborated, auditable knowledge can continue to be useful as those models change.

Our main takeaway is that persistent memory, combined with a process for checking and approving what the system learns, can make agents more useful over time without simply relying on larger prompts.

What's next

The memory system is designed to be independent of the specific use case, so the same approach could be applied to areas such as collections, logistics, customer support, and callback management.

Our next steps are to improve how information is consolidated, support additional use cases, and build a hosted version of the operator dashboard.