RecallOps

Inspiration

Incident response teams often solve the same kinds of production problems again and again, but the useful knowledge gets buried in old tickets, chat threads, runbooks, or someone’s memory. When a similar incident happens later, operators may still start from scratch.

RecallOps was inspired by that gap: what if an incident-response assistant could remember what worked before, preserve what failed, and use that operational memory to recommend better next steps during future incidents?

What it does

RecallOps is an AI incident-response memory assistant for support and engineering teams.

Users can create or select an incident, run an initial AI analysis, generate a safe semantic fingerprint preview, save long-term operational memories, and recall similar memories during future incidents.

For a selected incident, RecallOps generates a Titan embedding, searches active memories stored in CockroachDB using vector similarity, applies a semantic gate, and ranks results using deterministic signals such as similarity, reliability, and same-service match.

The memory-assisted recommendation flow then sends only bounded memory metadata to Amazon Bedrock Nova, so the recommendation is grounded in prior incident outcomes rather than only the current incident text.

RecallOps also includes a Memory Inspector where users can review active, rejected, and superseded memories. Rejected and superseded memories are preserved for history but excluded from future recall, helping the system avoid silently forgetting why certain knowledge became outdated or unreliable.

How we built it

RecallOps is built as a full-stack web application with React, TypeScript, and Vite on the frontend, and Python/FastAPI on the backend.

CockroachDB Cloud is used as the persistent operational memory layer. Incidents and memories are stored transactionally, while memory embeddings are stored as VECTOR(1024) values. RecallOps uses CockroachDB vector search to retrieve semantically similar active memories, then applies deterministic ranking before passing context to the LLM.

Amazon Bedrock powers the AI layer. Titan Text Embeddings V2 generates embeddings for incidents and memories, while Bedrock Nova generates structured incident analysis and memory-assisted recommendations.

The public demo is deployed on AWS ECS Express Mode from a Docker image stored in Amazon ECR. Runtime configuration is provided through environment variables, with the database URL stored as an SSM Parameter Store SecureString. IAM roles are used for Bedrock access and secure runtime permissions.

RecallOps also uses the CockroachDB Cloud Managed MCP Server as a read-only Memory Analyst workflow outside the web app. This lets an MCP client inspect memory health, lifecycle state, reliability counts, and supersession chains without exposing raw vectors or mutating data.

Challenges we ran into

One major challenge was designing memory recall so it was useful but not misleading. A naive approach would simply retrieve vaguely similar old incidents and hand them to an LLM. RecallOps instead uses a semantic similarity threshold first, then deterministic ranking with reliability and same-service signals, so metadata cannot rescue an irrelevant memory that failed the semantic gate.

Another challenge was memory lifecycle management. Real operational memory becomes stale. Instead of deleting rows, RecallOps preserves rejected and superseded memories while excluding them from future recall. This keeps the memory layer auditable.

Deployment also had a few real-world lessons. App Runner was unavailable for this AWS account, so the deployment was moved to AWS ECS Express Mode. We also had to package the public CockroachDB CA certificate into the Docker image so sslmode=verify-full works in the deployed container without relying on a local developer machine.

A final challenge was safety. Raw vectors are never returned through the API or UI, API docs are disabled in production, paid AI endpoints are rate-limited, and the memory-assisted prompt treats incident and memory text as untrusted data.

Accomplishments that we're proud of

I’m proud that RecallOps is not just a chatbot wrapper. It has a real persistent memory layer, semantic recall, deterministic ranking, reliability feedback, and memory lifecycle controls.

The project demonstrates CockroachDB as more than a normal CRUD database: it stores operational state, vector embeddings, lifecycle metadata, reliability signals, and recallable long-term memory in one consistent system.

I’m also proud of the deployment. RecallOps is running publicly on AWS ECS Express Mode, with a Dockerized FastAPI + React app, secure runtime configuration, Bedrock integration, and a repeatable demo dataset.

The MCP Memory Analyst workflow is another highlight because it shows the same CockroachDB memory layer being inspected from outside the app in a read-only, operationally safe way.

What we learned

This project made it clear that agent memory is not just about storing embeddings. Useful memory needs lifecycle state, reliability, relevance thresholds, and guardrails around what gets recalled.

I also learned how important deterministic systems are around LLMs. RecallOps uses Bedrock for reasoning, but retrieval, filtering, scoring, memory lifecycle, and feedback updates are deterministic and auditable.

On the deployment side, I learned how to package and deploy a production-style container through AWS ECR and ECS Express Mode, use SSM Parameter Store for secrets, configure IAM roles, and smoke-test a public cloud deployment.

What's next for RecallOps

Next, RecallOps could add authentication and role-based access so teams can safely use it beyond a public demo.

The memory loop could also become more automated: after a recommendation is used, RecallOps could suggest saving the successful resolution back as a new memory, with human approval.

Other next steps include background jobs for stale-memory detection, richer audit/event history, managed edge rate limiting, stronger production observability, and deeper integrations with ticketing systems, Slack, or incident-management tools.

Longer term, RecallOps could become a full incident memory platform where teams continuously learn from production operations instead of rediscovering the same fixes repeatedly.

Built With

• agents
• ai
• amazon-web-services
• amazonbedrock
• amazontitan
• cockroachdb
• docker
• ecs
• fastapi
• python
• react
• searching
• typescript
• vector