Recall

AI agent with persistent semantic memory that never forgets.

Comment

Recall: The AI Agent That Never Forgets

Inspiration

Most AI agents have excellent short-term intelligence but surprisingly poor long-term memory. A conversation can contain an important preference, goal, decision, or piece of context, yet that information is often lost once the conversation ends.

We wanted to build an agent that could remember what matters.

The inspiration for Recall came from a simple question:

What if an AI agent could build a persistent understanding of a person over time, instead of starting from zero every time?

Human memory isn't just a transcript of everything we've experienced. We remember important facts, connect related experiences, update old beliefs, and forget information that is no longer useful. We wanted Recall to follow a similar principle.

What We Built

Recall is an autonomous AI agent with persistent semantic memory.

Instead of treating every conversation as isolated, Recall continuously extracts useful information from conversations and turns it into structured memories:

  • Facts — information about the user or their world
  • Preferences — things the user likes or dislikes
  • Goals — objectives they are working toward
  • Events — important things that happened
  • Relationships — connections between related memories

Every memory is converted into a vector embedding and stored directly in CockroachDB. When a new message arrives, Recall performs semantic retrieval to find the memories most relevant to the current context.

The result is an agent that can connect today's conversation with things it learned yesterday, last week, or months ago.

How It Works

Recall's memory pipeline has four major stages.

1. Extract

The agent analyzes each conversation and identifies atomic pieces of information worth remembering.

Instead of storing an entire conversation as one giant blob, Recall breaks it into meaningful memories.

2. Embed & Store

Each memory is converted into a 1536-dimensional vector embedding using Amazon Bedrock's Titan Text Embeddings v2 model.

Those embeddings are stored in CockroachDB alongside the memory metadata.

We use a distributed HNSW vector index for approximate nearest-neighbor search:

CREATE INDEX idx_memories_embedding
ON memories
USING hnsw (embedding vector_cosine_ops);

This lets Recall search for semantically similar memories without requiring a separate vector database.

3. Recall & Link

When the user sends a new message, Recall embeds the query and searches the memory index for relevant information.

Retrieved memories are then connected through semantic relationships, creating a growing memory graph.

This allows the agent to reason not only about individual memories, but also about how different memories relate to one another.

4. Consolidate

Memory shouldn't grow forever without organization.

Recall periodically runs a consolidation process that identifies duplicate or overlapping memories, updates importance scores, strengthens useful relationships, and prunes low-value information.

The idea is inspired by how human memory consolidates information during sleep.

Architecture

Recall combines CockroachDB's distributed database capabilities with AWS serverless AI infrastructure.

Agent Layer

The agent is responsible for:

  • Memory extraction
  • Semantic retrieval
  • Response generation
  • Memory graph linking
  • Consolidation and reflection

AWS

Amazon Bedrock generates embeddings using Titan Text Embeddings v2.

AWS Lambda handles the serverless memory pipeline, including embedding generation, retrieval, storage, consolidation, and graph operations.

Amazon S3 provides durable archival storage for conversation transcripts, memory snapshots, and agent reflections.

CockroachDB

CockroachDB acts as the persistent memory layer.

It stores:

  • Memory records
  • Vector embeddings
  • Importance metadata
  • Semantic relationships
  • Graph connections

Its distributed vector indexing allows semantic search to remain part of the same database that stores the underlying memories.

This was particularly important to us: we didn't want to bolt a vector database onto the side of the application.

Why CockroachDB?

CockroachDB was a natural fit because Recall needs both durable relational storage and semantic search.

A memory isn't just a vector. It also has metadata, timestamps, types, relationships, and importance.

Keeping these pieces together means the agent can perform traditional database operations and vector similarity search within the same system.

The distributed architecture also gives Recall a path toward scaling its memory layer as the number of users and memories grows.

We also explored the CockroachDB Cloud MCP Server, which gave an agent direct, auditable access to the database for inspecting schemas, debugging queries, and understanding the memory store.

The ccloud CLI further allowed us to manage and inspect the CockroachDB environment in an agent-friendly, machine-readable way.

Challenges

Designing useful memories

One of the hardest problems wasn't storing information — it was deciding what deserves to be remembered.

If we stored every sentence, the memory system would quickly become noisy. If we stored too little, the agent would forget important context.

We therefore focused on extracting small, meaningful memories rather than entire conversation histories.

Semantic retrieval

Finding the right memory isn't always a keyword-matching problem.

A user might say something completely different from the original conversation while referring to the same underlying concept. Vector embeddings and cosine similarity allow Recall to retrieve memories based on meaning rather than exact wording.

Avoiding memory duplication

Repeated conversations can produce multiple versions of the same fact.

This motivated the consolidation engine, which compares memories, identifies duplicates or related entries, and keeps the long-term memory store useful instead of allowing it to become an ever-growing transcript.

Building the memory graph

Retrieval tells us which memories are similar, but we also wanted Recall to understand relationships between memories.

Designing a lightweight graph layer on top of the vector relationships required balancing useful connections against excessive linking.

What We Learned

Building Recall taught us that agentic memory is fundamentally a data problem as much as it is an AI problem.

Generating an embedding is easy. Building a memory system that remains useful as information accumulates is much harder.

We learned how to combine:

  • LLM-based information extraction
  • Vector embeddings
  • Distributed HNSW indexing
  • Relational database design
  • Semantic similarity
  • Graph relationships
  • Serverless AWS infrastructure
  • Memory consolidation

We also learned that persistence changes the design of an AI agent. Once an agent can remember, every new interaction has the potential to improve future interactions.

What's Next

Recall is designed as a foundation for agents that become more useful over time.

Future improvements could include stronger temporal reasoning, confidence-aware memories, explicit memory updates when information changes, richer graph traversal, and more sophisticated forgetting and consolidation strategies.

Our long-term vision is simple:

An AI agent shouldn't just answer your questions. It should learn from your relationship with it.

Recall is our step toward that future — a persistent memory layer where every meaningful interaction can compound into a smarter, more contextual agent.

Built With

Share this project:

Updates

Submission history