agentic memory ci

Pay once to learn, answer forever from memory.

Comment

Inspiration

Every engineer has lived this loop: CI fails, you scroll through a wall of logs, you ask an LLM what went wrong, you fix it — and two weeks later the same root cause returns under a different commit SHA. The model gets paid again to rediscover what the team already knew.

We were inspired by a simple observation: most CI failures are not new. On public repos they repeat. A triage agent that forgets after every call is an expensive search bar. A triage agent that remembers — and only spends a model call on genuine novelty — felt like the missing piece between agent demos and something you would actually leave running overnight.

CockroachDB made that idea concrete. We did not want a bolt-on vector store glued to an app database. We wanted embeddings, operational rows, concurrent agents, and historical truth in one system — so “memory” was a database property, not a marketing label.

What it does

Agentic Memory CI is a triage agent that turns failed CI logs into durable, human-verified memory.

Ingest failed GitHub Actions logs from real repos. Fingerprint each log into a canonical failure signature. Embed it with Amazon Titan Text Embeddings V2 and store it as VECTOR(1024) in CockroachDB. Cluster near-duplicates into patterns. On triage: recall the nearest patterns. If a human-confirmed fix is close enough, reuse it for free. Otherwise ask Amazon Nova Pro, grounded in what memory already knows. A human Accept/Reject closes the loop — accepted answers become the pattern’s known resolution; ignored noise expires via row TTL. On our corpus of 236 real failures from pandas, home-assistant, next.js, prettier, and grafana, failures collapse into 51 patterns — about $4.6\times$ compression. Two confirmed fixes already cover 62 of those failures ($\approx 26%$), so more than a quarter of the corpus would need no model call today.

Live console: ci-failure-memory.onrender.com

How we built it

A short pipeline, each step owned by one module:

Stage Module Job Ingest ingest.py Pull failed job logs from public GitHub repos Fingerprint fingerprint.py Denoise → extract last real error → normalize Memory memory.py Embed, store, serializable transact() Cluster cluster.py Group by vector distance Triage triage.py Recall → propose → learn from verdict Console app.py + dashboard.html Failures, proposals, memory, audit, concurrency Distance policy was measured, not guessed. Failures sit a median $0.26$ from their own pattern and $0.89$ at the 90th percentile. Clustering admits up to $1.0$; reusing a remembered fix requires $d \leq 0.80$. We deliberately made reuse tighter than clustering: a wrong group is a reporting mistake; a wrong root cause is an engineering one.

CockroachDB features we actually leaned on:

Distributed vector indexes on failures and patterns Serializable transactions with client retry (memory.transact) so concurrent agents create one pattern, not two AS OF SYSTEM TIME for audits of past decisions Row-level TTL so unconfirmed patterns decay on their own AWS: Titan Embeddings V2 for vectors; Nova Pro for fresh triage; adaptive retry with a degraded memory-only path if Bedrock is unreachable.

Deploy: Render blueprint (render.yaml), least-privilege dashboard user from harden.py, public console read-only behind WRITE_TOKEN.

Challenges we ran into

Fingerprints that lied. Our first clustering looked amazing — roughly $11\times$ “compression” — until we noticed patterns like DEBUG pixi_config and fail-on-cache-miss: false. We were clustering boilerplate with boilerplate. Two bugs: an unanchored case-insensitive error: match, and scanning forwards (setup noise) instead of backwards (where real failures surface). Fixing fingerprinting was the highest-leverage work in the project.

Privilege reality check. Applying the CockroachDB hardening-user-privileges skill showed the app connected as an admin member, and PUBLIC held CREATE. Worse: revoking CREATE on the database changed nothing — the privilege that mattered lived on the schema. harden.py now creates a purpose-built role and proves the denials by connecting as that user.

Time travel has a wall. We configured a 14-day GC window on our tables, but AS OF SYSTEM TIME also touches a system range pinned at gc.ttlseconds = 4500. On a shared Cloud tenant we cannot raise it. Live probes showed roughly $-12\text{h}$ succeeds and $-14\text{h}$ fails. So audits record retrieval evidence (which patterns came back, and how far) — because MVCC can replay what a row was, never what the agent was looking at.

Concurrency without hope. Naive read-modify-write lost 24 of 32 increments under load. The serializable + retry path lost none. Correctness lived in the client’s willingness to retry, not in a cleverer SQL statement.

Accomplishments that we're proud of

A memory loop that is provable, not just demoable: fingerprint.py, selfcheck.py, triage.py --selfcheck, contention.py --selfcheck, audit.py --decay-proof. Real corpus math: $236 \rightarrow 51$ patterns, with confirmed memory already covering $\approx 26%$ of failures. A public demo that cannot burn Bedrock credits or run DDL — hardened role + write token. An operator console where you can see origin labels: model vs recalled vs degraded — so memory is visible, not a black-box cache. Honest limits documented in the README instead of papered over.

What we learned

Similarity search is not memory. Memory needs a verdict loop, confidence, expiry, and auditability. Fingerprint quality dominates model quality. A perfect LLM on noisy logs still clusters garbage. Serializable isolation only helps if you retry. The database refusing a bad interleaving is useless if the agent shrugs and moves on. Measure thresholds on your corpus. Reuse at $0.80$ and cluster at $1.0$ came from distance percentiles, not vibes. Least privilege is a product feature when your demo is on the public internet.

What's next for agentic memory ci

Dedicated CockroachDB cluster with control of default-range GC, restoring the full intended audit window (14 days, not $\sim 12$ hours). Periodic re-clustering so near-duplicate patterns can merge after single-pass assignment. Org / repo scoping and real auth beyond a shared WRITE_TOKEN. PR bot integration — comment the recalled fix (or “novel failure, asking the model”) directly on failing checks. Broader ingest beyond GitHub Actions (GitLab CI, Buildkite) with the same fingerprint → memory loop.

Built With

Share this project:

Updates

Submission history