-
-
Architecture
-
Alibaba workbench
-
-
-
-
-
-
-
-
-
-
-
Inspiration
Every new "AI memory" framework I tried had the same failure mode: it stored more, then hallucinated more. RAG under the hood means the model still fills gaps with invention whenever retrieval is noisy — and retrieval always gets noisier as you store more. I kept watching agents confidently answer questions from evidence that had been contradicted three sessions ago, because retrieval surfaced the wrong sentence and nothing in the pipeline pushed back.
The realization: memory isn't a storage problem, it's an epistemology problem. A fact needs a source, a confidence, and a lineage — not just an embedding. And a memory system that can't tell "sustained preference change" apart from "one-off stray claim" will always oscillate on noisy input, no matter how big the context window gets.
MemoryOS is my answer: treat memory as a chain of evidence, gate every action by earned confidence, and — when evidence can't decide — ask a human instead of guessing fluently.
What it does
MemoryOS is an evidence-based memory agent for Track 1. Every stored fact carries its source, its confidence, and its lineage across four memory layers:
- Episodic — raw events with full provenance, never interpreted.
- Semantic — duplicate facts merge, and their sources merge with them. Disagreeing values coexist as competing facts until audited (they are not overwritten).
- Pattern — five deterministic detectors promote a hypothesis to trusted knowledge only when ≥3 sourced episodes across ≥2 sessions agree. This is where unprogrammed discovery happens ("meetings get rescheduled after long weekends" — nobody ever told it that).
- Decay — single-source memories fade with a 6-month half-life; corroborated ones, 18 months. Uncorroborated claims fade; corroborated ones endure.
Confidence is a documented formula — 0.40·corroboration + 0.30·recency + 0.20·verification + 0.10·user-confirmation — so every score in the UI can be recomputed by hand from the evidence chain. Confidence gates what the agent may do (policy-as-code):
- ≥ 0.80 → act autonomously
- ≥ 0.40 → answer with sources
- below → ask a clarifying question instead of guessing
- competing values within 0.15 of each other → ask, regardless of absolute score
An Evidence Auditor runs after every ingest: it detects contradictions, resolves what evidence can decide (its timeline-aware rule distinguishes a genuine preference change from a one-off noisy claim), and escalates the rest to the user — whose answer becomes the strongest evidence in the formula.
A Next.js dashboard exposes all of this: an accuracy-over-sessions curve, every memory with its expandable evidence chain and term-by-term confidence breakdown, a live auditor page for one-click human resolution, and a side-by-side "Ask" panel that contrasts a traditional last-wins memory (fluent, sourceless) against MemoryOS (gated, cited, honest about conflicts). Everything streams live over SSE.
Two real-world ingesters ship out of the box: python -m evals.ingest_markdown --path ~/notes folds an Obsidian vault into episodic memory, and python -m evals.ingest_ics --path ~/calendar.ics does the same for a Google Calendar export. Both go through the same engine as the API, so corroboration, contradictions, confidence, and decay all apply to real user data.
Raw event (email · chat · note · task · calendar)
│
▼
┌──────────────┐
│ Qwen extract │ ── structured assertions ──┐
└──────────────┘ │
▼
┌──────────────────┐
│ Episodic layer │
│ raw + provenance│
└────────┬─────────┘
▼
┌──────────────────┐
│ Semantic layer │
│ dedupe, merge │
│ sources — losers│
│ coexist │
└────────┬─────────┘
▼
┌──────────────────┐
│ Evidence Auditor │
│ contradiction? │
└──┬────────┬──────┘
timeline-decidable │ │ cannot decide
▼ ▼
┌──────────┐ ┌──────────┐
│ auto- │ │ ASK the │
│ resolve │ │ user │
└────┬─────┘ └────┬─────┘
│ │ user answers
│ │ = strongest evidence
└──────┬──────┘
▼
┌──────────────────────┐
│ Pattern layer │
│ ≥3 episodes across │
│ ≥2 sessions │
└──────────┬───────────┘
▼
┌──────────────────────┐
│ Decay — 6mo / 18mo │
│ half-life │
└──────────┬───────────┘
▼
┌──────────────────────┐
│ Confidence formula │
│ .4·corr + .3·rec + │
│ .2·verif + .1·conf │
└──────────┬───────────┘
▼
┌─────────────┐
│ Policy gate │
└──┬───┬───┬──┘
≥0.80 ────┘ │ └──── below 0.40
│ or gap <0.15
│
0.40 ─ 0.80 ─────┘
│
┌──────────────┼──────────────┐
▼ ▼ ▼
ACT ANSWER + ASK instead
auto cite sources of guessing
How we built it
MemoryOS is structured as an event-driven agent: every observation (calendar item, email, chat, note, task) enters through an ingest endpoint, is perceived by a Qwen-powered extractor, and passes through a deterministic control loop — episodic recording → semantic dedupe → Evidence Auditor (a specialized sub-agent for timeline-aware conflict resolution) → pattern promotion → decay recompute. The agent's action policy is policy-as-code (backend/config/confidence_policy.yaml): confidence ≥ 0.80 triggers act, 0.40–0.80 triggers cite sources, below or ambiguous triggers ask the user — the human-in-the-loop path whose answer becomes the strongest evidence in the ledger. Tool use: Qwen (DashScope) for reasoning, PostgreSQL + pgvector for state and retrieval, SSE for live agent-to-UI signaling.
Backend — FastAPI + SQLAlchemy on Python 3.11. Memory persisted in PostgreSQL with pgvector for the hybrid-retrieval path. The engine is a set of small, deterministic modules — app/memory/{episodic,semantic,pattern,decay}.py, app/confidence.py, app/evidence_auditor.py — each independently testable, so the fast path can be verified without touching an LLM at all.
Slow path — Qwen on Alibaba Cloud — All LLM calls go through DashScope's OpenAI-compatible endpoint on Alibaba Cloud Model Studio (see backend/app/qwen_client.py). Qwen is used for exactly four jobs: extracting structured assertions from raw event text, mapping free-text questions to memory keys, phrasing sourced answers, and wording proven patterns. A fallback chain degrades qwen-plus → qwen-turbo → deterministic rules; with zero working keys the system still records every event episodically and keeps every confidence score correct — it defers interpretation rather than inventing it.
Frontend — Next.js + TypeScript + Recharts, subscribed to backend SSE for live auditor updates and pattern promotions.
Deployment (Alibaba Cloud, end-to-end) — ECS runs the backend + frontend via Docker Compose (see deploy/); ApsaraDB RDS for PostgreSQL (with pgvector) is the memory store — only DATABASE_URL changes between local and cloud; Model Studio serves Qwen. Nginx enforces gzip and modern security headers (X-Content-Type-Options, X-Frame-Options, Referrer-Policy, Permissions-Policy). Containers run as a non-root user with a HEALTHCHECK.
Evaluation harness — a deterministic seeded harness asks the same 12 decision tasks after each of 20 sessions against a synthetic enterprise history (calendar, email, notes, tasks, chat) containing sparse early evidence, misleading one-off events, and two genuine mid-run preference changes. Zero LLM calls, so every reported number is bit-for-bit reproducible. Additionally, MemoryOS is run against LongMemEval (ICLR 2025, MIT-licensed, 500 questions across 6 memory categories) with a vanilla RAG baseline using the same Qwen embeddings and answer model — any delta is attributable to the fact layer, not retrieval quality.
Engineering hygiene — 83 tests, ruff-clean, GitHub Actions CI runs on every push, bash scripts/reproduce_all.sh verifies every claim in the README end-to-end in about five minutes.
Judge's browser
┌──────────────────────────┐
│ Next.js dashboard │
│ Recharts · SSE client │
└────────────┬─────────────┘
│ HTTP + SSE
┌──────────────────┴───────────────────┐
│ Alibaba Cloud · ECS (Singapore) │
│ ┌────────────────────────────────┐ │
│ │ nginx — gzip, security headers │ │
│ └───────────────┬────────────────┘ │
│ ┌───────────────┴────────────────┐ │
│ │ FastAPI · uvicorn · non-root │ │
│ └───────┬───────────────┬────────┘ │
│ │ │ │
│ fast path (~99%) slow path (~1%) │
│ ┌───────┴───────┐ ┌───┴────────┐ │
│ │ Episodic │ │ Qwen via │──┼──► Model Studio
│ │ → Semantic │ │ DashScope │ │ qwen-plus →
│ │ → Pattern │ │ (fallback │ │ qwen-turbo →
│ │ → Decay │ │ chain) │ │ rules-only
│ │ + Auditor │ └────────────┘ │
│ │ + Confidence │ │
│ └───────┬───────┘ │
└──────────┼───────────────────────────┘
│ SQL + vector search
┌──────────┴───────────────────────────┐
│ Alibaba Cloud · ApsaraDB RDS │
│ PostgreSQL 16 + pgvector │
└──────────────────────────────────────┘
Challenges we ran into
- Distinguishing preference change from noise. The naive rule ("newest evidence wins") whipsaws on stray claims; the strict rule ("majority wins") is blind to genuine changes. The fix was making the Evidence Auditor timeline-aware: a challenger wins only when it has sustained support from independent origins arriving after the incumbent's last support. Encoding that predicate cleanly took several iterations.
- The 80/20 fast/slow split is easy to claim and hard to hold. Every time we added a feature we had to actively resist reaching for the LLM. The dashboard's live fast-path counter (currently 99.6% on a 20-session state, 725 deterministic ops vs 2 Qwen calls) exists partly as a self-discipline device.
- Making "reproducible" actually mean reproducible. Every claim in the README maps to a script in
scripts/reproduce_all.sh— the accuracy curve, the 6-seed sweep, the docker stack, the demo CLI, the LongMemEval-vs-RAG comparison. Getting seeds, timestamps, and fallback paths to all agree bit-for-bit across environments was more work than the eval itself. - Fallback that degrades gracefully instead of failing loudly.
qwen-plus → qwen-turbo → rules-onlysounds simple, but the system had to keep confidence math correct on the rules-only path so evaluators could reproduce numbers even without a Qwen key.
Accomplishments that we're proud of
- Precision when acting: 1.00 on every seed. Across 6 seeds × 20 sessions × 12 tasks = 1,440 decisions, the acting gate never once fired on a wrong answer.
- Beats vanilla RAG on LongMemEval using the same Qwen models on the same data: +20 pts on knowledge-update, +10 pts on multi-session — the two categories the MemoryAgent track brief was explicitly written about.
- 99.6% fast-path share on a live 20-session state: 2 Qwen calls vs 725 deterministic ops. Every avoided call is a token judges aren't paying for — and the counter is live in the dashboard, not a slide.
- Unprogrammed pattern discovery — the pattern layer promotes hypotheses like "weekend avoidance", "late-night activity", "post-break reschedules", "Monday reschedules" purely from sourced episodes agreeing. Nobody hardcoded any of them.
- Complete engineering package: 83 tests, ruff-clean, green CI, docker-compose one-liner,
scripts/reproduce_all.shruns every claim in the README in five minutes, two real-world ingesters (Markdown + ICS), full data-portability endpoint (GET /api/export), and comprehensive docs including a security review and an evaluation-methodology doc.
What we learned
- Retrieval quality is a ceiling, not a solution. Once retrieval is decent, additional accuracy comes from validation — pushing back on evidence — not from more scale. Evidence chains are cheaper than better embeddings.
- Human-in-the-loop only works if the machine knows exactly when to defer. A vague "confidence is low" signal generates noise; a hard rule ("competing values within 0.15 → always ask") generates trust.
- Determinism is a feature, not a limitation. Making 80% of the work deterministic didn't reduce what the agent could do — it made everything else auditable. Users trust the LLM more when they can see how little of the pipeline actually depends on it.
- Fallback chains are product decisions. Choosing to keep the system usable with zero API keys shaped a lot of downstream design — and turned out to be what made the eval harness deterministic.
What's next for MemoryOS
- Enterprise connectors — real Google Calendar / Microsoft 365 / Slack / Notion ingestion on top of the two file-based ingesters that ship today.
- Multi-tenant isolation — one MemoryOS instance serving many users, with per-tenant confidence policies and per-tenant audit trails.
- Learned pattern detectors — the current five detectors are hand-written; the same evidence infrastructure can support pattern learning from co-occurrence in the episodic store.
- Embedding-based semantic clustering of near-duplicate keys ("meeting_pref" and "meeting_preference") so the semantic layer merges even when the extractor produced slightly different key names.
- Decay tuning from observed re-confirmation rates rather than the current fixed half-lives — the system already has all the data it needs.
Built With
- alibabacloud
- apsaradb
- dashscope
- docker
- dockercompose
- ecs
- fastapi
- githubactions
- modelstudio
- nextjs
- nginx
- pgvector
- postgresql
- pydantic
- pytest
- python
- qwen
- react
- recharts
- ruff
- sqlalchemy
- sse
- tailwindcss
- typescript
Log in or sign up for Devpost to join the conversation.