Inspiration

Engineering Memory OS

A shared memory layer for engineering teams and the AI agents they work with.

Inspiration

Every engineer now works with three or four agents — Claude Code, Codex, Cursor, ChatGPT — and every one of them starts cold. We compensate by hand-maintaining CLAUDE.md, .cursorrules, AGENTS.md, a Notion log, and an ADR folder that nobody has touched since Q2.

That has three failure modes, and we have lived all of them:

1. $N$ copies, $N$ drift rates. The same architectural fact gets written into four files that disagree with each other within a month.
2. Maintaining the log becomes a second job. The discipline required to keep it fresh is exactly the discipline nobody has at 6pm on a Friday.
3. The real rationale was never written down anywhere. It lives in a merged PR thread, a Slack argument from March, and one person's head. When that person leaves, so does the why.

The thing that finally pushed us to build was noticing that the knowledge already exists. It's generated continuously — in PRs, Slack threads, Jira tickets, design docs. It is captured but not retrievable, and it is completely invisible to the agents actually doing the work.

So the gap isn't storage. Storage is solved. The gap is a shared context layer that can tell a stable identity fact from a project constraint from a one-off session detail from a durable decision — and serve the right slice to whoever is asking, human or machine.

One more thing lit the fuse: watching an agent confidently cite a decision that had been reversed a year earlier. A memory system that can't represent "we changed our minds" is worse than no memory system, because it launders stale opinions into authoritative-sounding context.

What it does

Engineering Memory OS connects to the tools a team already uses, automatically extracts durable engineering knowledge into a typed, versioned, provenance-tracked memory store, and serves it back on demand.

Capture is fully automatic. No confirmation dialog, no "save this to memory" button. A message lands in Slack or an issue is edited in Jira, and within a few seconds it has either become a memory or been recorded as a documented rejection.

Every memory has exactly one of six types, and the taxonomy is load-bearing — retrieval weighting, decay rates, and write permissions all key off it:

Type	Holds	Lifetime
identity	who a person or team is; role, preferences, working style	very stable
project	what a system is; architecture, constraints, goals	slow-changing
convention	how we do things here	slow-changing
decision	a choice made, with rationale and alternatives	immutable; superseded, never edited
reference	pointer to an external artifact	stable
session	ephemeral working state	TTL, hours to days

convention earns its own type because it is precisely what CLAUDE.md and .cursorrules contain. Replacing those files is the core promise.

Provenance is mandatory. A memory with no source pointer cannot be written — the dataclass raises in __post_init__. There is no code path in the repo that produces an unattributable record. This is the thing that makes fully-automatic capture survivable: every claim is auditable back to the thread it came from, and one click removes it.

Decisions form a DAG, not an edit history. When a new decision contradicts an old one, the old record stays intact with status: superseded and a superseded_by pointer:

$$D_1 \;(\text{"use MongoDB", 2024-01})\;\xrightarrow{\text{superseded_by}}\;D_2\;(\text{"move to CockroachDB", 2026-03})$$

Retrieval returns only chain heads by default. The full chain surfaces when the intent is explain-why or history — because "why did we change our minds?" is a genuinely different question from "what do we do now?", and conflating them is exactly how agents end up citing reversed decisions.

Low-confidence extractions get quarantined, not discarded. Below the confidence threshold a record is persisted with status: quarantined — saved, with its provenance, but invisible to queries until a human asks for it by name. The alternative options are both bad: store it as fact and an agent eventually cites it, or drop it and lose coverage with no record that anything was ever there.

Today it ships with:

• Google sign-in, workspaces, and per-workspace tenancy
• Slack connector — OAuth install, signature-verified events, /ask slash command answered from memory
• Jira connector — 3LO OAuth, webhooks, plus a JQL backfill for the history webhooks can't reach
• Semantic retrieval over a workspace-partitioned vector index
• A web app — sources, timeline, memory browser, ask, settings — including the "what did you learn this week" audit view that automatic capture requires

How we built it

Stack: React + Vite + TypeScript + Tailwind on the front, FastAPI on the back, CockroachDB (with native vector support) for storage, Redis for cache, Gemini for embeddings and classification behind a provider abstraction that also has OpenAI and Anthropic implementations. Docker Compose for dev and prod, nginx in front of prod.

The capture pipeline

source webhook/poll
  → normalize to a common Event shape
  → prefilter        (free, no LLM)
  → classify         (one LLM call: store? which type? confidence?)
  → confidence gate  (active vs quarantined)
  → embed + persist with provenance

Normalization first. Slack, Jira, and GitHub payloads collapse into one Event via a Pydantic discriminated union, so FastAPI validates the right shape and errors name the offending field instead of dumping all three schemas. Everything downstream — filter, classifier, pipeline — has never heard of Slack.

The pre-filter is a cost control, not a quality control. This is the stage that makes Slack economically viable at all. If $N$ is inbound message volume, $p_{\text{keep}}$ the fraction that survives the filter, and $c_{\text{LLM}}$ the per-classification cost, then

$$C_{\text{total}} = N \cdot p_{\text{keep}} \cdot c_{\text{LLM}}$$

and $c_{\text{LLM}}$ is fixed by the model you picked. $p_{\text{keep}}$ is the only lever you own. So the filter drops bot messages, Slack noise subtypes, anything empty after stripping code blocks and mentions, bare links, whole-message acknowledgements, and anything under CAPTURE_MIN_LENGTH (40) characters.

But a pure length cutoff throws away "we're going with Postgres, not Mongo" — which is the single most valuable message in the channel. So a ~40-term signal lexicon (decided, instead of, because, convention, always, never, migrate, deprecat, adr, owns, …) rescues short-but-decisive text. And the ack regex is anchored: lgtm dies, lgtm, but the retry loop still double-counts survives, because the second one is a real convention signal.

The stage deliberately leans toward keeping. A false keep costs one cheap model call. A false drop loses the memory permanently and silently.

Classification is one LLM call that decides store/don't-store and the category together, answering in JSON over the plain-text chat interface rather than a provider-native structured-output mode — so switching LLM_PROVIDER doesn't touch that file. The prompt carries an explicit reject list: questions and speculation, proposals nobody agreed to, status chatter, transient facts, restatements of what's already obvious from the code. Parsing strips ``fences and then falls back to salvaging first-{through last-}`, because models prepend prose no matter what you tell them.

Every outcome is recorded, including the rejections. Four outcomes — dropped_prefilter, dropped_classifier, quarantined, stored — each with a reason, either from the check that fired or the model's own rationale. A silent drop is indistinguishable from a bug, and the audit view needs to show what was rejected and why. Rejections return 200: "nothing worth storing" is a normal connector outcome, not an error.

The connectors

Slack has a hard 3-second ack budget, which forces the shape of the whole handler: verify the signature, record the event, return 200, and only then do the LLM work in the background. A source_events table with a unique constraint on (workspace_id, external_id) makes Slack's retries idempotent — without it a slow classifier turns one message into three identical memories.

Jira looks like the same problem and isn't, in three ways:

1. Tokens expire. Slack's xoxb- token lives until revoked; a Jira 3LO access token lasts an hour. Every install stores a refresh token, and Atlassian rotates the refresh token on use — so the refreshed value has to be persisted before every outbound call. Dropping that write breaks the install exactly one hour later, which is the worst kind of failure to debug.
2. Webhooks are not signed. Slack HMACs the raw body; Atlassian doesn't sign deliveries for OAuth apps at all. So a high-entropy per-installation secret sits in the callback path and possession of it is the authentication. That URL is a bearer credential, and we document it as one.
3. Text arrives as a document. REST v3 returns descriptions and comments in Atlassian Document Format — a nested JSON tree, not a string — so adf_to_text flattens it before the classifier, which reads prose, ever sees it.

Jira also needed a gate Slack didn't: jira:issue_updated fires for every field touch — assignee, status, sprint, story points. None of those change the prose. Without a changelog check, one drag of a card across a board is one LLM call. Four gates run cheapest-first, and Atlassian's own JQL filter is the free one that runs before we're even involved.

Retrieval

Candidate generation is a cosine nearest-neighbour search over the embeddings table. The endpoint reports, per stage, what actually ran:

"pipeline_stages": {
  "candidate_generation": "vector_cosine",
  "supersession_collapse": "not_implemented",
  "rerank": "not_implemented"
}

That field exists so nothing downstream — MCP, the web app, the editor extension — mistakes today's output for supersession-safe. The response shape is the deliverable; the internals get replaced wholesale.

It also degrades rather than fails, in two places. No workspace_id on the request, or an embedding provider that's down, both fall back to lexical token overlap and say so in the response note. A retrieval endpoint that 500s is worse than one that answers less well.

Challenges we ran into

The vector index quietly did nothing. This was the big one. We built the embeddings table, wrote a vector on every capture, created a vector index, and queried it scoped to a workspace. It worked. It was also reading every row in the table:

EXPLAIN ... WHERE workspace_id = 'ws1' ORDER BY v <=> $1 LIMIT 3
  -> filter -> scan  table: t@t_pkey  spans: FULL SCAN

A vector index is only used when the query's equality filters match its prefix columns — and a column you reach through a join can't be a prefix. The fix was to denormalize workspace_id onto embeddings (redundant, but it cannot live behind a join) and build the index as (workspace_id, vector vector_cosine_ops). Same query, different plan:

-> vector search  table: t@t_workspace_id_v_idx
     prefix spans: [/'ws1' - /'ws1']

An index-backed search confined to one tenant's partition. The failure mode here is nasty precisely because nothing errors — you get correct results, at full-scan cost, forever.

Two more ways vector search fails silently. The opclass has to be cosine, not the vector_l2_ops default, because <=> is the operator the query uses and mismatching them falls back to a scan. And the ORDER BY must use the raw distance — writing 1 - (a <=> b) to sort by similarity drops the index — so the conversion to a 0..1 similarity happens in Python after the rows come back:

$$\text{sim}(q,m) = 1 - d_{\cos}(q,m), \qquad d_{\cos}(q,m) = 1 - \frac{q \cdot m}{|q|\,|m|}$$

Embeddings that were only 59% of a vector. text-embedding-004 was retired mid-build and started 404ing. Its replacement, gemini-embedding-001, returns 3072 dimensions natively; we ask for 768 to match the VECTOR(768) column. What the docs don't put in bold is that the model only returns normalized vectors at its native dimensionality. Truncated to 768, they come back at

$$|v_{768}| \approx 0.59$$

Cosine ignores magnitude, so ranking looked fine and nothing complained — but any future reach for L2 or an inner product would have been silently wrong. We renormalize on the way in.

Asymmetric embeddings. The same model projects a stored document and the question someone asks about it into different regions unless you tell it which is which. Indexing uses RETRIEVAL_DOCUMENT, searching uses RETRIEVAL_QUERY. Swapping them isn't an error — it just quietly costs recall, which is the hardest class of bug to notice because your test query still returns something.

Slack's 3-second ack versus an LLM call. You cannot classify inside the request. Restructuring around record-then-ack-then-process is what forced source_events into existence — and that table turned out to be the most valuable one in the schema, because it's simultaneously the idempotency key, the retry guard, and the audit trail.

ngrok, repeatedly. Both Slack and Atlassian must reach the backend over public HTTPS, and the free tunnel URL changes every restart. Each restart means updating BACKEND_URL, the OAuth callback in two developer consoles, the event subscription URL, and reconnecting — because the registered webhook still points at a hostname that no longer exists. We wrote it into the docs as a step rather than pretending it was a one-time setup.

Atlassian consent failing for a reason that isn't about consent. "Requires access to a Jira site which you don't have" turns out to mean the signed-in account has no Jira site at all. An Atlassian account is not a Jira licence, and the account that owns the developer-console app very often has no product attached to it. Incognito is the fastest test.

Nothing committed to the database. Early on, the chat service held a session and never called .commit(). Everything appeared to work in a single request and vanished on the next one.

Accomplishments that we're proud of

The provenance invariant holds at the type level. Not "we remember to add provenance" — the constructor raises without it. There is no path through the codebase that writes an unattributable memory. That single constraint is what makes automatic capture defensible instead of reckless.

We found the full-scan bug. Vector search that returns the right answers at the wrong cost is invisible until it's a production incident. Catching it during the build, writing the two EXPLAIN plans directly into the migration file as a comment, and encoding why the redundant column exists means nobody deletes it in six months during a tidy-up.

Two connectors, one pipeline. Jira and Slack differ in auth, signing, retry semantics, and text encoding, and they share every line downstream of normalization. Adding GitHub is a connector, not a pipeline change.

We designed for being wrong. pipeline_stages announcing not_implemented per stage, quarantined as a real state rather than a discard, rejections carrying reasons, the store behind an ABC so Postgres arrived as a second class rather than a rewrite of every caller. The parts we haven't built yet are labelled in the API response rather than in a TODO nobody reads.

The docs are honest. backend/docs/ has a known-limits section per connector that says plainly: tokens are stored in plaintext, the Jira webhook URL is a bearer credential, background processing dies with the worker, there's no ACL inheritance yet. Writing those down was uncomfortable and is the reason we know exactly what M5 is.

What we learned

Silent degradation is the enemy, not exceptions. Almost every hard bug in this project produced correct-looking output: the full scan, the un-normalized vectors, the swapped task types, the uncommitted session. We started defending against it explicitly — the response note field, pipeline_stages, recorded rejection reasons — because the systems that hurt you are the ones that don't complain.

Cost is an architectural constraint, not an optimization pass. "Embed every Slack message" doesn't survive contact with $C = N \cdot p_{\text{keep}} \cdot c_{\text{LLM}}$. The pre-filter isn't a performance tweak bolted on at the end; it's the reason Slack ingestion exists at all, and it shaped the Event abstraction that everything else hangs off.

Taxonomy is engineering, not documentation. Deciding on six memory types felt like a naming exercise for about a day. It's actually the schema for retrieval weighting, the decay rates, and the write permissions. convention existing as its own type is what makes "replace your CLAUDE.md" a real feature instead of a slogan.

Bias your filters toward recall, then measure. A false keep costs one cheap model call. A false drop loses information permanently, silently, with no record it ever existed. Asymmetric costs deserve asymmetric thresholds — and we wrote the reasoning into the module docstring so the next person tuning it knows which direction to be nervous in.

Connectors are 80% the platform's quirks, 20% your code. Token rotation, unsigned webhooks, ADF trees, 3-second acks, retries, changelog gates. The pipeline was the interesting part; the connectors were the work.

Self-reported confidence is not calibrated. The threshold of 0.6 is a guess. Models cluster on round numbers and skew overconfident. We know the number is unjustified — which is why the eval harness is the next milestone and not a stretch goal.

What's next for Engineering Memory OS

Dedupe and supersession detection. Both were blocked on a vector substrate; that now exists. An issue edited three times currently produces three near-identical memories. Dedupe is embedding similarity against existing memories in scope; supersession is entity overlap plus contradiction classification plus recency, with low-confidence supersessions routed to quarantined rather than silently rewriting history.

The intelligent retrieval pipeline. This is the differentiated half of the product and today it's one stage out of nine. The design:

query → intent classify → scope resolve → hybrid candidates → category weighting
      → supersession collapse → temporal decay → ACL filter → rerank → budget-pack

Intent classification routes to a category weight matrix — an explain-why query weights decision at 1.0 and session at 0.2; an implement query inverts that and pushes convention to the top. Combined with per-type temporal decay, a candidate's score becomes

$$s(m) = w_{\text{intent}}(\text{type}m)\cdot\text{sim}(q,m)\cdot e^{-\lambda{\text{type}}\,\Delta t}, \qquad \lambda_{\text{type}} = \frac{\ln 2}{t_{1/2}(\text{type})}$$

where session has a half-life measured in hours and identity in years. Vector search over a document dump is the commodity part — it is not the product.

An MCP server. The leverage play: one protocol, and Claude Code, Codex, Cursor, and Claude Desktop all work. Tools: memory.search, memory.get_context(scope), memory.write, memory.forget, memory.decisions(entity). This is the point at which the agents stop starting cold.

A GitHub connector. The densest source of real decisions with rationale, and the one with the most natural provenance links. Jira records what; PR review threads record why.

An eval harness, before more features. Retrieval precision@5 on a hand-labelled set of 100+ real queries; supersession correctness with a target above 99%, because one confidently-cited reversed decision costs more trust than ten misses; extraction precision, below which automatic capture is net-negative; and forget rate as the early warning that extraction is degrading. The product is unfalsifiable without these numbers.

Then the hardening that turns a demo into a product: a real job queue (background tasks currently die with the worker, leaving rows stuck at pending), envelope encryption for stored OAuth tokens, ACL inheritance at ingestion and enforcement at retrieval — because private-channel content in a shared store is the one failure mode that ends the product outright — a promotion path out of quarantine, and a VS Code extension over the same API.

Built With

• ai
• python
• react