AgentOPS-Memory

Your agent's memory never goes down

Comment

AgentOps Memory

Inspiration

This is something anyone working on-call knows: some system crashes at 2 AM and the first twenty minutes are devoted to rediscovering something which has already been discovered by the same people three months ago — under a different name of the alarm.

This is what AgentOps Memory attempts to solve. It is not "some AI that answers questions about your infra", it is an agent that has its own memory of each and every incident that it has diagnosed before, and it is something that gets faster with every use, rather than being restarted from scratch every time.

The idea behind the CockroachDB x AWS Hackathon challenge — "agents need memory that never goes away" — was just what we needed in order to finally build something like that and not just try it using in-memory vector store.

What it does

AgentOps Memory is an incident-response agent that:

  1. Intake of symptoms, logs, or an alert.
  2. Always first does a search within its own memory — a cosine-similarity search across the embeddings of all past incidents — and then gives a root cause explanation. And if it had encountered the pattern before, it explicitly states it and tells how it was solved the last time.
  3. Investigation process with all actions logged onto an evident timeline.
  4. Once solved, saves the incident and the new embedding into memory, which will help the next on-call engineer (either human or agent).
  5. Is able to assess the health state of its own memory (total number of incidents, number of unresolved, state of vector index), in real-time, through reasoning about its own database via MCP connection.

If there is a hit from similarity search within the relevant timeframe, that's the winning scenario: diagnosis process trending towards $O(1)$ instead of $O(n)$ per incident, with memory buildup.

How we built it

  • Memory layer: CockroachDB, leveraging native VECTOR(1536) column type and CREATE VECTOR INDEX ... USING vector (embedding vector_cosine_ops) for distributed cosine-distance search — no additional vector store required, no hassle of sync between your operational data and embeddings.
  • Self-inspection: a check_memory_health utility running through CockroachDB's Cloud Managed MCP Server, not direct query, allowing the agent to analyze its memory layer, not only retrieve data.
  • Agent reasoning: pluggable across three different backend implementations with a single unified agent schema — Groq (default), Amazon Bedrock, and Anthropic API directly — simply change AGENT_PROVIDER environment variable.
  • AWS: embeddings generation done asynchronously in AWS Lambda, triggered by SQS queue on every incident write, and calls Amazon Bedrock's Titan Text Embeddings model — a slow call to get embedding never stalls your agent response. Raw logs saved to S3 to avoid storing blobs in row-store.
  • Frontend: dark ops console React/Vite frontend with a "memory recall" live panel, to watch the agent's hits during reasoning process.
  • Demo data: seed incidents are actual bugs, found in previous author's projects, not generated.

Challenges we ran into

  • Rigorous tool call validation across multiple provider schemas. Different backends require their tool schema in a different wrapper (input_schema in Anthropic's case, toolSpec.inputSchema.json in Bedrock, parameters as per Groq's OpenAI clone), and Groq's Llama sometimes produced a numeric input parameter (such as limit) as a string, which Groq's schema validator rejected out of hand with a 400, even before our code received the call to validate. We fixed that by loosening the schema to accept both [number, string] and adding an additional system prompt hint as a fallback mechanism.
  • Schema drift between the "migration script" and "actual schema in the cluster." Our migration script used CREATE TABLE IF NOT EXISTS on a fresh cluster, but became a no-op on subsequent migrations once the table existed — so the new column we later added to the migration (resolution) never appeared in our actual schema, leading to a 500 and column "resolution" does not exist. This was fixed by reconciling any missing columns explicitly using ALTER TABLE ... ADD COLUMN IF NOT EXISTS, so this kind of drift could not occur anymore.
  • Maintaining the schema of the single tool definition across multiple formats of function calls.

What we learned

  • “A vector index is the easy 20% of agent memory.” The hard thing is to make memory a forcing function, an integral part of the system prompt — “search first!”— instead of something that is politely optional for the model to implement.
  • Multi-provider tool call seems to be solved problem until you go to the boundaries of it: identical JSON schemas and three different levels of strictness and three different failure modes.
  • “It works locally” and “the schema in the database is identical to the schema in the code” are two different things, and only one of them is detected by IF NOT EXISTS statement.

What's next

  • Push the seed corpus up to be the production feed, allowing memory to compound beyond the demo period.
  • Bring forward a “confidence” metric on recalled events, based on vector distance, to allow the interface to visually differentiate between a close call and a weak one.
  • Introduce write-side deduplication – if two events come together at a common root cause, collapse their embeddings rather than treat them as distinct memories.
Share this project:

Updates

Submission history