Déjà Vu — on-call agent with incident memory

An on-call SRE agent with long-term incident memory. Same alert, twice: 31% confidence and "restart the pods" without memory, 97% and the exact fix with receipts when it remembers.

Comment

Inspiration

At 2 a.m. an alert fires in checkout-api. The engineer who fixed this exact failure six weeks ago is asleep. What's left is a runbook, a dashboard, and a stranger on call.

So they restart the pods. It holds for forty minutes. Then it comes back, because the real cause — a missing composite index on orders(created_at, status) — is still there.

This is the normal state of on-call. The knowledge exists. It just isn't reachable at the moment it's needed. A bigger model does not fix it, because the answer was never in the weights. It's in your own incident history.

I wanted to see whether an agent could answer the way the person who was there would: with the incident, the fix that actually worked, and the receipts to prove it.

What it does

Déjà Vu answers an alert, then shows you exactly what it remembered before it answered. Same alert, answered twice:

Memory off Memory on
Decision nothing to act on APPLY THIS FIX
Hypothesis "generic triage for checkout-api" missing composite index on orders(created_at, status)
First action restart the affected pods CREATE INDEX CONCURRENTLY, then verify the query plan
Evidence none INC-1006, INC-1001
Confidence 31% 97%
Owner — Priya Raman
Fix time — 6 minutes

Two details matter more than the headline number.

It names the look-alike it ruled out. INC-1010 has a near-identical alert and a completely different cause — a config service pushed a low pool ceiling. The agent shows the discriminator: pool size is low, but DB CPU and query latency are normal once a connection is acquired. A reviewer can check the reasoning instead of trusting it.

It says when it doesn't know. First-time failures can't be recalled. The agent returns low confidence and no recommended action rather than inventing one.

The demo also has a Memory tab (every stored incident), a Learning tab (the replay curve), and an Incidents tab (the raw corpus), because a memory system you can't audit is just a confident paragraph.

How I built it

The pipeline is deliberately boring:

alert → recall (Hindsight) → rank → contradiction guard → decide → narrate → critic → human

Memory is Hindsight. One bank per team (dejavu-oncall), one document per incident: symptom, signal fingerprint, failure class, root cause, fix, outcome, owner, duration, and a discriminator. retain fires when an incident resolves; recall fires on every alert.

The recall path returns its own provenance, which is what makes the UI honest:

def recall(query: str, k: int = 8, service: str | None = None) -> dict:
    ...
    return {
        "backend": used,          # "hindsight" or "local"
        "latency_ms": round(ms, 1),
        "query": query,
        "hits": hits[:k],
        "detail": detail,
    }

Ranking is deterministic. 50% signal-fingerprint similarity, 45% IDF text coverage, 5% coverage bonus, with discriminating signals weighted above generic ones (cert_age_days, dns_ttl_s, clock_skew_s, pool_size). It's unit-tested against the seeded corpus, so changing a weight shows up as a failing check rather than a vibe.

The model never owns the answer. Retrieval, ranking, the contradiction guard, the decision and the confidence are all deterministic. Groq (openai/gpt-oss-120b) only writes prose around facts it was handed, and a deterministic critic can block the output entirely.

Two structural choices. First, the unit of learning is the failure class, not the root cause string: "missing index on orders" and "missing index on settlement_date" are different strings but the same lesson. Second, the signal fingerprint is scored before the text, because two incidents can share every word and differ in one signal. Text search returns the nearest paragraph; Hindsight plus a fingerprint returns the right incident.

Human in the loop goes through a Band AI room. The agent posts the diagnosis and the proposed fix; a second participant, DejaVuCritic, checks that every claim has at least two current, successful receipts and can allow or veto before anything is applied.

Challenges I ran into

Separating "similar" from "same". Early ranking put the config-capped incident above the index incident because the alert text was closer. The fix was to weight discriminating signals above generic ones and to require the agent to state the discriminator in its answer, so the mistake would be visible instead of plausible.

Making confidence mean something. The first version was confident about everything. Confidence is now derived from evidence count, signal agreement and whether the discriminator is present, and it drops to a refusal when the guard trips.

Honest evaluation on a small corpus. With 34 incidents across a handful of services, one lucky hit moves the number. So the replay is graded at failure-class level, and the weak per-service histories (4–12 incidents each) are documented rather than smoothed over.

Not faking the memory. The easiest version of this project is a nice paragraph with no retrieval behind it. The hard version shows every retain and recall on screen with the backend, hit count and latency, and still looks good when the backend falls back to local search.

Accomplishments that I'm proud of

The learning curve. I replayed the corpus in chronological order, blind — the agent could only see incidents that had already happened.

  • First six alerts: 33% correct
  • Last six alerts: 83% correct

No model change, no prompt tuning. The memory got better because the corpus got more complete. That curve is the actual claim: memory is a system that improves with use, not a retrieval trick.

A refusal that holds. On an incident with no precedent, the agent returns low confidence and no action. It would have been easy — and much more impressive-looking — to hallucinate a fix. The Learning screen shows exactly where the corpus was thin.

A demo that's a real app. 34 seeded incidents, a live alert queue, memory cards you can expand, a trace strip with per-stage latency, a Band room with a critic, and a guided tour that runs the whole story without a narrator.

What I learned

Provenance is the product. A confident paragraph you can't check is worse than a list of incidents you can. Every claim cites a record or it doesn't ship.

Retrieval quality lives in the schema, not the model. Adding discriminator and a signal fingerprint did more than any prompt change I tried.

Build the refusal path first. Knowing when to say "no match" is what makes the confident answers worth trusting.

Make the memory visible. If the memory layer is invisible, nobody can tell it's working — not a user, not a reviewer, not me at 2 a.m.

What's next for Déjà Vu — on-call agent with incident memory

  • Real incident sources. Ingest from PagerDuty and Jira so the corpus builds itself instead of starting from a seed file.
  • Outcome feedback. Record whether a recommended fix actually held, and let that update the weight of the incident that suggested it. The current version records the outcome but doesn't yet learn from it.
  • Cross-service transfer, measured. The corpus hints that a missing index on one service transfers to another. That needs a larger corpus and an honest benchmark before I'd claim it.
  • A pre-flight check. Before an alert fires, compare the current signal fingerprint against the corpus and warn when a known failure class is approaching — the same memory, used earlier.

Built With

Share this project:

Updates

Submission history