Build Week scope: MemoryCustodian existed before OpenAI Build Week, with v0.7.0 as the pre-event baseline. During Build Week, I collaborated with Codex using GPT-5.6 to design, implement, test, and release versions 0.8.0 through 0.9.1. The work focused on reliability, privacy, safe mutations, protocol compatibility, and a clearer boundary between agent reasoning and deterministic enforcement.
Inspiration
Every new coding-agent session starts with a capable model, but often with project amnesia.
An agent can inspect the current code while still missing why architectural decisions were made, which constraints must remain true, and which approaches have already been rejected. Developers compensate by repeating context in prompts or adding more instructions to files such as AGENTS.md and CLAUDE.md.
Over time, those files become large, duplicated across platforms, and loaded into tasks that do not need them.
I wanted project memory to behave more like code: local to the repository, readable by humans, reviewable in pull requests, portable across agents, and small enough to use in normal development loops.
MemoryCustodian is built around one principle:
Memory can grow; context must stay small.
What it does
MemoryCustodian gives coding agents durable project memory without loading the entire project history into every task.
It stores knowledge as plain Markdown under docs/memory/:
brief.mddescribes the current projectdecisions.mdpreserves confirmed choicesconstraints.mdrecords requirements that must remain truedo-not-use.mdcaptures rejected approachesinbox.mdholds candidates awaiting semantic reviewmanifest.mddetermines which memory files apply to each task
Platform files such as AGENTS.md, CLAUDE.md, and GEMINI.md remain thin bootstraps. They point agents to the same repository memory instead of duplicating durable context for each platform.
Before substantial work, an agent reads the manifest and project brief, then loads only the files relevant to the current task.
The included NightNotes example demonstrates this workflow. Its memory records that session state should use human-readable local JSON files, the application must work offline using the Python standard library, and SQLite has already been rejected for the current storage requirements.
A completely new Codex session can recover those decisions from the repository without having them repeated in the prompt.
A lightweight Python CLI provides deterministic operations for initialization, task-specific reading, validation, updates, forgetting, repair, compaction, and protocol migration.
How I built it
MemoryCustodian has three main layers.
First, a repo-native protocol defines where memory lives, how it is structured, and which files apply to different task categories.
Second, an agent skill and thin adapters connect that protocol to Codex, Claude Code, Gemini-style agents, and generic shell workflows.
Third, a Python standard-library CLI handles operations that require predictable behavior, including routing validation, context budgets, safe file mutations, compaction, and migrations.
During Build Week, I used Codex with GPT-5.6 to audit the existing implementation, identify failure modes, turn them into concrete implementation plans, coordinate changes across the protocol, CLI, skill, adapters, documentation, and tests, and expand regression coverage.
GPT-5.6 was especially useful for reasoning about difficult boundaries:
- Which decisions require semantic understanding?
- Which operations can be enforced mechanically?
- What should happen when a multi-file mutation fails?
- How can information be forgotten without damaging unrelated memory?
- How should an older CLI respond to a newer project protocol?
The most important design change was making the semantic boundary explicit.
Earlier versions included limited keyword-based classification of inbox candidates. That appeared convenient, but it made a deterministic script responsible for judgments it could not reliably make.
Now, the agent or user decides meaning: scope, type, confidence, overlap with existing memory, and whether a candidate deserves promotion.
The CLI performs only mechanically verifiable work. It validates structure, previews exact mutation plans, detects exact duplicates and tombstone matches, and applies bounded operations safely.
I retained responsibility for the product boundaries, architecture, safety model, and final release decisions. Codex and GPT-5.6 helped build and validate MemoryCustodian; they are not runtime dependencies.
Challenges I ran into
Separating intelligence from enforcement
The hardest design question was deciding which work belongs to the agent and which belongs to the CLI.
A candidate such as “consider encrypting exported notes” might be a feature idea, a security constraint, a confirmed decision, or a temporary observation. A keyword cannot determine the correct interpretation.
The solution was to let capable agents or humans make semantic decisions while the CLI remains conservative and deterministic.
Making memory mutations safe
Forgetting, initialization, replacement, migration, and compaction may affect several files. A failure after only some writes can leave project memory inconsistent.
MemoryCustodian now computes mutation plans before writing, validates targets first, uses safe ordering where necessary, and reports partial completion explicitly if an unexpected filesystem failure occurs.
Initialization also distinguishes repair from replacement. init --repair creates missing components and refreshes recognized generated metadata without overwriting curated memory. Full replacement is separate, preview-first, and requires explicit application.
Markdown structure introduced another challenge. A top-level entry may contain continuation paragraphs, nested bullets, or fenced examples. Version 0.9.1 treats these as complete units so compaction and previews preserve their meaning.
The CLI also refuses repair or migration when project protocol metadata is newer than the installed version or cannot be parsed, preventing accidental protocol downgrades.
Accomplishments I’m proud of
The biggest accomplishment is that MemoryCustodian now has a clear trust model.
It does not claim that deterministic scripts understand project semantics. Instead, it combines agent reasoning with conservative, inspectable tooling.
The Build Week releases added safer multi-file mutations, complete-entry context packing, manifest-authoritative routing, privacy-safe forgetting, conservative repair, preview-first replacement, protocol compatibility guards, structure-preserving compaction, cross-platform CI, and expanded regression tests.
MemoryCustodian also dogfoods its own protocol. Its real project brief, decisions, constraints, and rejected approaches live in docs/memory/. Using the system on itself exposed lifecycle and mutation problems that synthetic examples did not.
The core remains lightweight and offline-first. Routine operation requires no embeddings, vector database, hosted memory service, API key, or automatic collection of private chat history.
What I learned
The biggest lesson was that agent memory is not primarily a storage problem. It is a governance problem.
A trustworthy memory system must decide what deserves to survive, which tasks should load it, how humans can inspect it, and how it can be safely changed, superseded, or forgotten.
I also learned that semantic intelligence and deterministic tooling should complement one another rather than imitate one another.
The agent understands meaning.
The CLI validates structure, presents exact plans, and applies bounded changes.
MemoryCustodian became safer when each layer focused on what it does best.
What’s next for MemoryCustodian
The next step is broader live evaluation across coding agents and long-running projects.
These evaluations will measure whether agents load the correct task-specific memory, preserve active constraints, avoid rejected approaches, make appropriate promotion decisions, and propose useful updates without creating context bloat.
I also plan to improve collaborative review, decision supersession, memory ownership, installation workflows, and team-level governance.
The long-term goal is not to remember everything.
It is to preserve the smallest amount of durable knowledge that helps the next agent make a better decision.
Durable memory. Minimal context.
Log in or sign up for Devpost to join the conversation.