Recall

Turns every past incident into instant diagnosis for the next one."

Comment

Inspiration

On-call teams solve the same incident twice, because last quarter's fix is buried in a closed ticket. Every incident retro promises "we'll write it down"; no team actually does. We wanted an agent whose memory does that automatically — and a memory layer that behaves like production infrastructure, not a transcript buffer.

What it does

Recall is an on-call copilot whose memory is CockroachDB. It ingests incident tickets, diagnoses them with an LLM agent that searches a vector memory of past incidents, and closes the loop: when a human resolves the ticket, the postmortem is embedded and written straight back into memory. The next similar ticket already cites it — the system gets better between two tickets, live on camera in the demo video.

Memory has a lifecycle: incidents earn quality_score from thumbs up/down feedback, age out through valid_until, and are chained with superseded_by when newer knowledge replaces them. Ranking combines semantic distance, earned quality and age — so the agent cites the current procedure, never the superseded one. And when there is no precedent, it says so (confidence 0.1) instead of inventing a root cause.

How we built it

  • CockroachDB is the entire data layer: tickets, diagnoses (JSONB) and the vector memory live in one database, one transaction boundary. VECTOR(1024) embeddings behind a distributed vector index (vector_cosine_ops) power the semantic recall.
  • The Cloud Managed MCP Server is a production read path, not an editor accessory: the deployed Lambda queries CockroachDB through it and falls back to psycopg only when the MCP is unavailable. Every evidence step in the UI is labeled via: mcp or via: fallback — you can watch which path each lookup took while the agent is still reasoning.
  • AWS: FastAPI on Lambda (arm64, Web Adapter — the same process runs locally and in production), a Function URL in RESPONSE_STREAM mode so the SSE evidence timeline arrives live, S3 + CloudFront for the frontend, and the whole stack deployed by CloudFormation/SAM from GitHub Actions over OIDC — no long-lived AWS keys anywhere.
  • The agent loop is provider-agnostic: Gemini, Bedrock and Anthropic are one environment variable apart.

Challenges we ran into

  • The Managed MCP Server rejects queries over 16 KB, and a single 1,024-float embedding literal is ~11 KB. Computing the distance once and ordering by its alias keeps semantic recall on the MCP path — EXPLAIN confirms the vector index still plans identically.
  • ccloud has no headless authentication, so the cluster preflight runs on workstations and degrades gracefully in CI — our one piece of feedback on the tooling.
  • Ranking is a knife's edge: a high-quality old incident can outrank the postmortem you wrote sixty seconds ago. The weights earn their keep.

Accomplishments and what we learned

147 unit tests that need no database and no cloud credentials; a security review that closed 48 findings before submission; and a demo where every claim in the narration is verifiable on screen, including which database path served each query.

What's next

Jira and PagerDuty ticket sources (the TicketSource Protocol is already there), changefeeds to react to memory writes, and real user auth in front of the app.

Built With

Share this project:

Updates