Hipocampus

A biologically-inspired, four-tier memory system that transforms stateless LLMs into persistent AI assistants with sleep consolidation, semantic vector retrieval, and real-time voice interaction.

Comment

The Inspiration

Every time I opened a new chat with an AI assistant, I had to start from scratch. "I'm a data scientist." "I use Python." "I prefer concise answers." Over and over, in every session, with every tool. The AI was brilliant in the moment and completely amnesiac the next day.

That bothered me. A real assistant, a human one, doesn't need you to re-introduce yourself every morning. They remember. They learn. They get better at helping you over time.

I started reading about how the human brain actually handles this: the hippocampus transfers short-term experiences into long-term cortical memory during sleep. That biological process quiet, scheduled, working while you rest became the architectural blueprint for Hipocampus.

How I Built It

I designed a four-tier memory hierarchy mirroring the brain's own structure:

  • Working memory (Redis, 1-hour TTL): the active conversation buffer
  • Episodic memory (PostgreSQL + pgvector): raw experiences with 1024-dimensional semantic embeddings from Qwen's text-embedding-v3
  • Semantic memory: distilled facts extracted by qwen-max during consolidation ("User prefers Python", "Team uses Kubernetes with Helm only")
  • Procedural memory: recurring behavioural patterns detected across sessions

Every night at 3AM, a Celery Beat scheduler triggers the sleep consolidation pipeline clustering episodes by cosine similarity, extracting structured facts using qwen-max, and resolving conflicts when preferences change. Users can also trigger it manually with a single button.

Retrieval uses pgvector cosine similarity across all tiers simultaneously, ranked by a custom importance scoring formula that weights recency, frequency, novelty, and explicit user commitment signals.

For MCP integration, I implemented the full OpenAI-compatible tool-calling loop with qwen-max. The model autonomously invokes a web_search tool when it determines real-time data is needed, no prompting required. When it fires, a globe badge appears: "Searched the web · Powered by Qwen MCP".

The voice pipeline chains three Qwen Cloud APIs: qwen3.5-omni-flash for speech-to-text via the native DashScope multimodal endpoint, qwen-max for reasoning with full memory context and MCP web search, and qwen-omni-turbo for text-to-speech via streaming SSE. I built a Voice Activity Detection system using the Web Audio API that auto-stops recording after 1.8 seconds of silence, no button holding needed in auto mode.

The stack: FastAPI (async Python 3.12), React + Vite, PostgreSQL 16 with pgvector, Redis, Celery, all orchestrated in Docker Compose and deployed on Alibaba Cloud Simple Application Server.

What I Learned

  • pgvector is production-ready. Running cosine similarity over 1024-dimensional vectors with sub-100ms latency on a 2-core machine is genuinely impressive.
  • Biology is a great API. Modelling sleep consolidation as a nightly Celery task wasn't just a metaphor, it was the right architectural decision. Offline processing keeps inference latency low and gives the system time to think.
  • MCP is a real paradigm shift. Letting the model decide when to call a tool rather than wrapping every response in a search produces dramatically better answers. The model knows when it doesn't know.
  • Audio APIs are deceptively hard. qwen-omni-turbo streams PCM16 audio, not MP3, despite the format parameter. The browser can't play raw PCM. The fix, prepending a 44-byte RIFF/WAV header took longer to diagnose than anything else in the project.
  • Alibaba Cloud's DashScope International is a genuinely powerful inference platform. Five distinct model types from one API key, one endpoint, one billing account. The qwen-max model's reasoning quality is exceptional, and text-embedding-v3 at 1024 dimensions produces rich semantic vectors that make retrieval feel almost magical.

The Challenges

The ECD disaster. I bought an Alibaba Cloud ECD (Elastic Cloud Desktop) thinking it was a server. It's a virtual desktop. It requires a Chinese enterprise workspace account to access. It cost $40. It taught me to read the product description more carefully. I eventually deployed on Alibaba Cloud Simple Application Server, which worked perfectly.

DuckDuckGo rate limits. The duckduckgo-search library v6 switched from httpx to primp for browser impersonation. On my server, primp couldn't find its browser profiles and fell back to random fingerprints which DuckDuckGo flagged immediately. The fix: replaced the library entirely with direct httpx calls to html.duckduckgo.com, the no-JS endpoint. Zero rate limits since.

TTS audio format mismatch. The text2audio/generation endpoint is CosyVoice's voice-cloning API, it requires a reference audio URL. Every other TTS model in the Qwen ecosystem uses a WebSocket endpoint via the SDK. The solution was qwen-omni-turbo with stream: true, which sends PCM16 chunks over SSE then wrapping those bytes in a WAV header client-side for browser playback.

Base64 chunk concatenation. When concatenating base64 SSE chunks directly ("AAAB==" + "CCCC"), the padding characters in the middle make the combined string invalid. The fix: decode each chunk independently to bytes, then concatenate the bytes.

Memory without context. Early versions injected all retrieved memories as raw text. The AI would reference things in confusing, out-of-context ways. Building the ranked context assembly algorithm tier weights, importance decay, cosine similarity scoring, token budget folding was the most intellectually satisfying part of the project.

What's Next

Hipocampus is built to be the memory layer for any AI application. The next step is exposing the memory system as a proper MCP server, so any Qwen agent, any LLM client, can call search_memory(query) as a tool and get semantically relevant user context back. The four-tier architecture scales horizontally; the consolidation pipeline is stateless; the retrieval is sub-100ms. It's ready to be a platform, not just a demo.

Built With

Share this project:

Updates