TriageTrace

A temporal memory firewall that helps Qwen-powered incident-response agents recall safe procedures and reject stale or poisoned instructions.

Comment

Inspiration

Incident-response agents face a difficult memory problem. A stateless agent forgets every previous incident, but an agent that remembers everything can be even more dangerous: it may follow a stale runbook, trust a poisoned log entry, or choose an obsolete remediation over a newer operator-approved procedure.

We built TriageTrace around one question:

How can an incident-response agent learn from experience without allowing its memory to become an attack surface?

Our answer is a temporal memory firewall that decides what an agent is allowed to remember, recall, supersede, or quarantine.

What it does

TriageTrace gives Qwen-powered incident-response agents durable, governed memory across sessions.

Each memory includes its tenant, scope, subject, predicate, type, provenance, authority, source timestamp, expiration, and lifecycle status. Before a memory can influence the agent, TriageTrace applies temporal and trust rules:

  • Newer authoritative procedures supersede stale procedures.
  • Lower-authority and out-of-order conflicts are quarantined.
  • Instructions embedded in untrusted logs cannot become trusted policies.
  • Quarantined, superseded, expired, future-dated, and cross-tenant memories are excluded from retrieval.
  • Operator-approved actions are screened by a deterministic simulator before becoming simulated_safe memories.

During an incident, the Qwen agent recalls only eligible memories and calls tools to inspect metrics, recent deployments, and the current runbook. The operator can see:

  • the active incident and production signals;
  • which memories were recalled or rejected;
  • the agent's tool and reasoning trace;
  • stateless and memory-informed recommendations side by side;
  • the human approval decision;
  • the predicted simulation outcome.

TriageTrace never automatically executes production remediation. Human approval is always required.

How we built it

The backend is built with Python, FastAPI, SQLAlchemy, PostgreSQL, and pgvector. It uses Qwen Cloud through an OpenAI-compatible API:

  • qwen3.7-plus for tool-calling and incident reasoning;
  • text-embedding-v4 for semantic memory retrieval.

Memory conflicts are resolved using authority and source timestamps rather than database insertion order. PostgreSQL advisory transaction locks serialize conflicting writes so two concurrent requests cannot both create an active memory for the same logical key.

The retrieval pipeline combines vector candidates, reranking or a deterministic fallback, utility scoring, maximal marginal relevance, and a strict token budget. Policies and preferences receive packing priority.

The dashboard is built with React, TypeScript, Vite, and Tailwind CSS. It is served by nginx, packaged with Docker Compose, and includes Terraform infrastructure for deployment to Alibaba Cloud ECS.

Challenges we faced

Treating time correctly

Insertion order is not the same as temporal truth. An older event can arrive after a newer event, so lifecycle decisions must use the source timestamp and authority together. We implemented one shared conflict-decision table and serialized writes per memory key.

Preventing memory poisoning

Similarity alone cannot determine whether a memory is trustworthy. A malicious instruction may be highly relevant to an incident. We therefore separate trust filtering from semantic retrieval and quarantine instructions from untrusted external content before they reach the agent.

Keeping retrieved context useful

Giving the model every eligible memory increases cost and can reduce reasoning quality. We added diversity scoring, utility weighting, and token-budget packing so the agent receives a small, auditable memory pack.

Building an honest safety demo

Simulation is predictive screening, not proof that an action was executed successfully. The interface consistently labels outcomes as simulated_safe and “predicted to improve” rather than claiming execution validation.

Accomplishments

On our 13-scenario live Qwen adversarial evaluation:

  • Correct-action accuracy increased from 23.1% stateless to 84.6% with memory.
  • Poisoned-memory recall was 0 out of 2.
  • Stale-memory recall was 0 out of 3.
  • Temporal-conflict accuracy was 3 out of 3.

These results are specific to our evaluation suite, but they demonstrate the intended behavior: relevant experience accumulates while stale and poisoned memories remain non-recallable.

We also built a complete operator workflow with memory inspection, Qwen tool traces, human approval, predictive simulation, automated tests, Docker deployment, and Alibaba Cloud infrastructure.

What we learned

The most important lesson was that agent memory is not simply a vector database. It is a policy system.

Relevance does not imply trust. Temporal order does not imply insertion order. Operator decisions, provenance, authority, expiration, and simulation outcomes must all participate in the memory lifecycle before semantic retrieval begins.

We also learned that memory safety must be visible. Showing recalled, rejected, superseded, and quarantined records makes the agent's behavior easier for operators to understand and audit.

What's next

Next, we want to:

  • improve multi-hop memory retrieval;
  • stream live pipeline progress while Qwen is reasoning;
  • evaluate against live Kubernetes incident environments;
  • add stronger production authentication and tenant administration;
  • integrate real post-remediation observations while preserving the human approval boundary.

TriageTrace demonstrates that useful agent memory and safe agent behavior do not have to be opposing goals.

Built With

Share this project:

Updates