AgentMemory-OS

The Persistent Memory Layer for AI Agents

Comment

Inspiration Every AI agent today has a fatal flaw: complete amnesia. Close a chat session, and the agent forgets who you are, what you know, and what you've already discussed. The next time you return, you're a stranger. Teams hack around this by stuffing 10–20 past messages into prompts, which is expensive, slow, and fundamentally broken. We experienced this pain firsthand. We built a coding assistant that gave brilliant advice on Monday, but by Friday it couldn't remember the user's tech stack, their career goals, or the debugging pattern we had just solved together. That frustration became our mission: build the persistent memory layer that AI agents should have had from day one. We drew inspiration from human cognitive science — how our brains store episodic (experiences), semantic (facts), and procedural (skills) memory — and asked: why can't agents do the same? AgentMemory OS was born from that question.

What it does AgentMemory OS is a persistent memory infrastructure framework that gives any AI agent three types of long-term memory through a Model Context Protocol (MCP) server: Episodic Memory — Complete conversation history across all sessions, stored with vector embeddings for semantic retrieval. Semantic Memory — Distilled facts about the user: their skills, goals, preferences, and restrictions, each with confidence scores. Procedural Memory — Reusable skill patterns and problem-solving procedures that the agent learns from successful interactions. When a user says "I know Node.js and want a fintech job," the agent stores that as semantic facts. One week later, when the same user asks "What should I learn today?" the agent retrieves those facts in under 50ms and responds: "Since you know Node.js and are targeting fintech, let's explore JWT authentication patterns today." The system is framework-agnostic — any agent built on any platform can connect via MCP and instantly gain persistent memory. We built a live chat demo with a Memory ON/OFF toggle so anyone can see the difference between a generic agent and a memory-powered agent.

How we built it Core Stack: Google Cloud Agent Builder for agent orchestration and tool calling Gemini 3 (via Vertex AI) for reasoning and response generation MongoDB Atlas as the unified database — operational data + vector search in one platform Model Context Protocol (MCP) for standardized agent-database communication Python FastAPI for the MCP server backend Google Cloud Run for serverless hosting of both MCP server and frontend Gemini Embedding API (text-embedding-004) for 768-dimensional vector generation Architecture: We built a FastAPI-based MCP server that exposes five tools: store_episodic, retrieve_context, store_semantic_fact, store_procedural_skill, and get_user_profile. The Agent Builder agent is configured with a strict system prompt that always checks memory before responding, stores new facts when detected, and saves every conversation turn. The MongoDB Atlas cluster stores all three memory layers. We created vector search indexes on the episodic_memory and procedural_memory collections for cosine similarity search. The retrieve_context tool performs three parallel queries — episodic vector search, semantic fact lookup, and procedural vector search — and returns combined context in under 50ms. We deployed everything to Cloud Run with public HTTPS endpoints, created a vanilla HTML/CSS/JS chat frontend, and seeded comprehensive demo data for a user named "Raj" to prove cross-session persistence.

Challenges we ran into Vector Search Configuration: Setting up MongoDB Atlas vector search indexes with the correct 768 dimensions and cosine similarity metric required careful tuning. We initially mismatched the embedding dimensions and had to rebuild the index. Embedding Latency: The first implementation called the Gemini embedding API synchronously for every memory operation, creating noticeable delay. We solved this by batching operations and adding lightweight caching for repeated queries. Context Overload: Early versions retrieved too many episodic memories, overwhelming the agent's context window. We implemented a hybrid scoring system that balances semantic similarity with recency, ensuring only the most relevant and recent memories are returned. MCP Protocol Integration: This was our first time building an MCP server from scratch. Debugging the tool schema format that Agent Builder expects took multiple iterations, especially getting the JSON parameter definitions exactly right for the Gemini function-calling mechanism. Memory Conflicts: When a user contradicts a previous fact (e.g., "I actually prefer Python now"), the agent needs to handle confidence decay gracefully. We built a conflict resolution system that reduces confidence on outdated facts and flags them for user verification.

Accomplishments that we're proud of True Cross-Session Memory: Our agent remembers users after days, weeks, or even months. We proved this with a demo where "Raj" mentions his Node.js background on Day 1, and the agent references it accurately on Day 7 without any prompt stuffing. Sub-50ms Retrieval: We achieved an average memory retrieval latency of 45ms across all three memory layers. This proves that persistent memory doesn't have to be slow — MongoDB's unified vector + operational architecture makes it possible. Framework-Agnostic Infrastructure: We didn't build a chatbot. We built infrastructure. Any agent — LangChain, LlamaIndex, ADK, or custom — can connect via our MCP server and instantly gain memory. That's a platform, not a product. Three-Layer Cognitive Architecture: Implementing episodic, semantic, and procedural memory separately, each with its own retrieval strategy, was complex but scientifically grounded. Judges and testers immediately understand the analogy to human memory. The "Memory Toggle" Demo: Our frontend's ON/OFF switch is the most powerful 10 seconds of our demo. Flip it OFF: generic responses. Flip it ON: personalized, contextual, intelligent responses. The contrast is visceral.

What we learned MCP is the Future of Agent Integration: Building with Model Context Protocol taught us that standardized tool interfaces are game-changing. Instead of custom integrations for every database, MCP lets any agent speak to any tool. This protocol will define the next era of AI development. MongoDB Atlas is Uniquely Positioned for AI: Before this hackathon, we saw MongoDB as a document store. Now we see it as an AI-native platform. The ability to run vector search, operational queries, and semantic retrieval on the same data in the same database — with sub-50ms latency — eliminates the fragmentation that plagues most AI stacks. Memory Architecture is Undervalued: The AI industry obsesses over model size and reasoning, but memory is equally critical. An agent with a 1B parameter model and perfect memory outperforms a 100B parameter model with amnesia. We believe memory infrastructure will be as important as the models themselves. User Verification Matters: Not all remembered facts are true forever. Building confidence scoring and conflict detection taught us that memory systems need skepticism — they must know what they know, and know when they might be wrong.

What's next for AgentMemory-OS Memory Compression: Long-term episodic memories become noisy. We will build an auto-summarization pipeline that compresses old episodic events into semantic facts, keeping memory relevant while reducing storage costs. Multi-User Shared Procedural Memory: Currently, procedural skills are either private or public. We want to build team memory — where a dev team's agents share learned debugging patterns, and a company's agents share institutional knowledge. Memory Decay & Forgetting: Real brains forget strategically. We will implement intelligent TTL and confidence decay so agents naturally forget irrelevant details while strengthening important facts through repetition. Enterprise Compliance Layer: For regulated industries, we will add audit trails, memory export, and GDPR-compliant memory deletion — turning AgentMemory OS from a developer tool into an enterprise-grade memory platform. Open Source Community: We are open-sourcing the MCP server under MIT license. We want AgentMemory OS to become the standard memory layer for the AI agent ecosystem, with community-contributed connectors for LangChain, CrewAI, AutoGen, and beyond.

Built With

  • arize
  • cloud-run-database:-mongodb-atlas
  • css3
  • fastapi
  • gemini-3
  • gemini-embedding-api
  • gitlab
  • google-cloud:-agent-builder
  • javascript-devops:-docker
  • mongodb-vector-search-protocol:-model-context-protocol-(mcp)-backend:-python
  • motor-frontend:-html5
  • pymongo
  • vertex-ai
Share this project:

Updates