The Memory

Every outage becomes a test. Every promise, a live check. Every AI agent earns the right to ship. A delivery pipeline that never forgets.

Comment

2:07 a.m.

Your phone buzzes. Orders are failing. You open the dashboard with one eye and see it: 503 database_busy.

You've seen this before. You know you've seen this before. There was a postmortem. Someone wrote "increase the pool size and add monitoring" under Action items. Someone said never again.

That someone left in June. The ticket is still in the backlog. It's 2:07 a.m., and you're rediscovering a fix your team already paid for once.

This isn't a made-up story. It's a real r/sre thread with almost exactly these words:

"We've written 23 postmortems this year and completed exactly 3 action items. [...] took us 2hrs to remember the fix because person who handled it in june left."

Twenty-three postmortems. Three action items. Teams don't lack tools. They lack memory.

And now we're adding AI agents that write code ten times faster. Every one of those changes is a fresh chance to bring back an old incident, quietly break an old promise, or ship something nobody fully trusts. This hackathon says code generation is solved. We think that makes the next question urgent:

When is it actually safe to let an agent ship on its own?

We answered it the way you'd answer it for a new teammate. Not on day one. Not everywhere. When they have a track record, in the parts of the system where they earned it, and only while they keep earning it.

What it does

The Memory is a delivery system that never forgets. It keeps three kinds of memory as plain files in the repository (.memory/), changed only through merge requests, so every memory is reviewable, versioned and auditable.

** Every wound becomes an antibody.** When production breaks, the Incident Responder checks past incidents first, diagnoses, and opens a fix with a regression test. Close the Loop writes the postmortem and turns every action item into a merge request the same hour (not a ticket that dies in a backlog), then turns the incident into an antibody: a replayable fault scenario. From then on, every MR that touches those files is replayed against the old incident on its own Cloud Run revision. If the incident would come back, the MR is blocked, with a link to the postmortem.

** Every promise becomes a living contract.** When an issue with acceptance criteria closes, Contract Keeper turns the criteria into executable checks that run on every review revision and against production every 5 minutes. Months later, if the promise breaks, the original issue reopens itself and Contract Breach finds the change that broke it.

** Every agent earns its autonomy.** Each agent has a trust score per code area. Survive a production canary: +0.05. Cause a rollback: ‚àí0.20. Get overridden by a human: ‚àí0.10. A policy file turns scores into hands-off, supervised or assisted. Agents never merge anything themselves: a deterministic gate merges only when all the evidence is green and the agent has earned hands-off trust in every area it touched. Auth and infrastructure are capped at supervised, whatever the score.

The Memory Console puts it all on one screen around a 3D memory core you can spin, and answers one question live: Is it safe to let the agents ship? Today the honest answer is not yet, and the console shows you exactly why.

Who it's for

  • The on-call engineer at 2:07 a.m., who should never rediscover a fix the team already paid for.
  • Platform and SRE teams bringing AI agents into their pipelines without handing over the keys blindly.
  • Engineering leaders asked "can we let the agents ship?", who need an answer backed by evidence, not vibes.

The morning we tried to break it

We didn't want a demo that only works in slides. So one morning we broke our own system and watched. Every timestamp below is real, and every step is in the public GitLab history.

07:55. We hit our staging Orders API with 40 users while the database crawled at 400 ms. 40.1% of orders failed. p99 latency: 15.6 seconds. A real outage, on purpose.

07:57. Incident #2 opened and went to the Incident Responder.

07:58. Seventy-four seconds after the mention. A diagnosis appeared on the incident. Before guessing, the agent had checked the immune memory for a matching past incident. None existed yet. Then it named the cause: a database pool of 5 with a 1-second timeout.

07:59. Two minutes and thirty-four seconds after the mention. A fix was waiting as a merge request (MR !1): pool 5 ‚Üí 20, timeout 1 s ‚Üí 5 s, plus a regression test that fires 40 concurrent requests at a slow database.

08:00. In parallel, the Contract Keeper (MR !2) turned issue #3, a closed feature ("hold orders above $500 for verification") into a four-check production contract. It added the $500.00 and $500.01 boundaries on its own.

08:10. The gate said no. Not because of anything we planted. pip-audit had found seven real known vulnerabilities in starlette, a framework dependency we had pinned (PYSEC-2026-161, -248, -249, -1941, -1942, -2280, -2281). Nothing merged until we upgraded to FastAPI 0.162 and starlette 1.10. That was the moment it stopped feeling like a demo. It felt like a colleague who had just caught our mistake.

08:20. Everything green: tests, replay, 4 of 4 promises, security. And the gate still held the fix. No agent had earned hands-off trust in infrastructure yet, so a human had to look. We'd built a system smart enough to fix the outage in two and a half minutes, and humble enough not to merge it alone.

09:39. A human merged the fix and the promise. Pool 20 rolled out through a 10% canary on Google Cloud Run, held, and was promoted; production's /health now reports pool 20. The trust ledger wrote its first lines: Incident Responder, 0.50 ‚Üí 0.55 in api and infra. A tiny number, and the most important one on the page, because it was earned. Later the Contract Keeper earned its first +0.05 too.

09:40. Close the Loop wrote the postmortem and the INC-2 antibody (MR !3, merged at 10:20), and turned every remaining action item into a merge request: a runbook (!4), an alert as code (!5) and a Retry-After header (!6). Twenty-three postmortems, three action items had become one incident, every action item a merge request within the hour.

We proved the antibody both ways before trusting it: replayed against the old pool of 5, 65% of requests fail; against the fix, zero errors, p99 1.2 s.

10:20. Then we tried to undo it. We opened the kind of MR anyone would approve on a busy afternoon (MR !7): lower the DB pool to 5 to save Cloud SQL connections. Tests passed. Security passed. Every promise held. Then the pipeline replayed incident #2 against it on its own Cloud Run revision, and the gate wrote, in plain words:

This change brings back incident #2. Replay failed: p95 3,766 ms (limit 2,500), p99 3,801 ms (limit 3,500). Postmortem: .memory/postmortems/INC-2.md (MR !7)

The pipeline went red and the MR can't merge. The system remembered what we would have forgotten.

An incident went in. A permanent memory came out. That's the whole idea, in one morning.

For the judges: the 90-second path

  1. Watch it (45 s): open the Memory Console in Judge Mode. It replays incident #2 in eight scenes, and every number links back to GitLab.
  2. See the memory refuse a regression (20 s): MR !7: all green except the replay of incident #2, so it's blocked.
  3. Read the incident end to end (15 s): issue #2: alert, diagnosis, fix, canary, postmortem, antibody.
  4. Open the memory itself (10 s): the .memory/ folder (antibodies, contracts, postmortems, trust ledger), all plain files, all changed through MRs.
  • Technological implementation: 5 custom Duo flows synced from the repo and validated in CI; a gate that reads commit authors, replay results, contracts, security and trust; 43 tests; visible pipeline history
  • Design: One unbroken loop: alert ‚Üí diagnosis ‚Üí fix ‚Üí gate ‚Üí canary ‚Üí postmortem ‚Üí antibody ‚Üí blocked regression
  • Potential impact: The r/sre "23 postmortems, 3 action items" problem: every action item of incident #2 became a merge request within the hour, and the incident became a test that blocks its return
  • Innovation: Incidents as replayable antibodies, promises as production contracts, and autonomy as an earned, losable trust score per agent per code area
  • Presentation: A 3-minute video, Judge Mode on the live console, and this real timeline

How we built it

GitLab Duo Agent Platform: the agents

  • Five custom flows in the AI Catalog, enabled with triggers:
    • Incident Responder: conditional routing, fix or escalate
    • Close the Loop: a postmortem writer, then an action-item agent
    • Contract Keeper
    • Contract Breach: an investigator, then a repairer
    • Release Council: a supervisor with security, SRE and product sub-agents that argue go or no-go from evidence
  • Memory Oracle, a custom chat agent that answers "why is this the way it is?" from the memory.
  • GitLab's foundational Code Review and Fix CI/CD Pipeline flows, plus SAST false-positive detection.
  • A read-only MCP server on the console, so IDE Duo Chat can ask the memory questions.
  • Every flow is created and updated from the repository through GitLab's GraphQL API (scripts/duo_sync.py), and validated in CI against GitLab.com's real tool list.

GitLab CI/CD: the nervous system, across all nine DevSecOps stages

  • Plan: acceptance criteria become contracts.
  • Create: agent fixes and action-item MRs.
  • Verify: immune replay, living contracts, unit and end-to-end tests.
  • Package: one image, copied by digest to Artifact Registry.
  • Secure: GitLab SAST, IaC, secret, dependency and container scanning, plus a pip-audit and bandit summary that the gate reads.
  • Release: a Cloud Run review revision per MR, staging, a 10% canary, automatic promote or rollback, and release notes that say which agent shipped what and why it was trusted.
  • Configure: Terraform, with alerts as code.
  • Monitor: Cloud Monitoring ‚Üí GitLab incident ‚Üí Incident Responder, and promises checked every 5 minutes.
  • Govern: policy as code, the trust ledger, caps and CODEOWNERS.

Google Cloud: where it lives Cloud Run (production with traffic-split canaries, staging with tagged per-MR revisions, the console, a contract-runner job), Cloud SQL for PostgreSQL, Artifact Registry, Secret Manager, Cloud Monitoring, Cloud Scheduler, and Workload Identity Federation, so CI never holds a single key. All of it in Terraform.

The Memory Console A live page on Cloud Run: a three.js memory core, the real incident loop on canvas, every gate decision with its reason, the trust matrix, and Judge Mode, an autoplaying replay of incident #2 for anyone with 90 seconds.

Challenges we ran into

  • Agents can't wake each other. GitLab ignores triggers fired by bots, so one flow can't call the next. We let CI drive the deterministic steps, and let a relay act on behalf of the on-call human.
  • "Who actually wrote this?" A flow's merge request is opened as the person who triggered it, while its commits belong to the flow's service account. If we'd trusted the author field, no agent would ever have earned a single point of trust. The gate now reads commit authors.
  • Config isn't code, until it is. The fix lived in Terraform, but CI only shipped images. So each commit's pool settings now flow into its review, staging and canary revisions. A config fix gets replayed and canaried like any line of code.
  • The runner hiccupped twice pulling its own helper image. The gate refused to call anything green without evidence, which is exactly what it should do. Annoying in the moment. Reassuring in hindsight.

Accomplishments we're proud of

  • An agent went from alert to tested fix in 2 minutes 34 seconds, and still didn't get to merge it alone. We're proud of both halves of that sentence.
  • The gate caught seven real vulnerabilities on its first day.
  • Trust moved because of real deploys: the Incident Responder and the Contract Keeper each earned their first +0.05 from changes that survived a production canary.
  • The first antibody is proven both ways: against the old configuration, 65% of requests fail; against the fix, zero errors and p99 1.2 s.
  • A postmortem whose action items all became merge requests within the hour, instead of tickets.
  • The first time someone tried to quietly undo the fix, the memory blocked it (MR !7).

What we learned

The most valuable thing an agent produces after the code isn't more code. It's evidence.

Every decision here (merge, hold, roll back, trust more, trust less) comes from evidence that is written down, versioned and replayable. We came in thinking autonomy was a setting. We're leaving convinced it's a balance you earn, and can lose, one deploy at a time.

What's next for The Memory

  • Trust that decays when an area goes quiet, the way your confidence in old code does.
  • Blast radii learned from incidents, not declared by hand.
  • Shared antibodies across services that use the same libraries, so one team's bad night protects every other team.

Somewhere tonight, someone's phone will buzz at 2:07 a.m. for an outage their team has already survived. We built The Memory so that, next time, the system remembers, and the pager stays quiet.

The system remembers every promise and every wound, so the people don't have to.

Built With

Share this project:

Updates

Submission history