Inspiration
On-call engineers often meet the same failure twice, but ordinary incident memory is unsafe. A chat transcript mixes hypotheses, failed attempts, approvals, and outcomes. Similarity search can retrieve a highly relevant fix without proving that it worked, matched this service version, or is still current.
Outage Atlas asks a stricter question: what action was justified, approved, applied, and verified for this exact scope?
What it does
Outage Atlas turns completed incidents into auditable causal runbooks. Six bounded agent roles coordinate triage, investigation, risk criticism, remediation, and verification. Every mutating tool remains a simulation and is blocked behind a persisted human approval gate.
The memory lifecycle is Learn -> Verify -> Promote -> Recall / Forget:
- A proposed recovery starts as a non-recallable candidate.
- Promotion requires consumed approval, an applied sandbox mutation, fresh receipt-correlated evidence from an isolated read-only observer, and a Verifier decision citing the exact evidence set.
- Recall rejects failed, unverified, expired, superseded, wrong-service, wrong-version, and symptom-mismatched records before ranking anything.
- Eligible memories are packed into a bounded 1,200-unit estimated context budget, with a reason and provenance retained for every inclusion and rejection.
- A resolved incident can start a new session. The new incident begins from fresh evidence while the persistent ledger recalls the verified recovery and suppresses the superseded record.
The result is memory that can abstain. Similarity alone never grants authority to change a system.
How we built it
The operator console uses React 19, TypeScript, and Vite. A Node.js and Express API owns the incident state machine, six-role orchestration, JSON-backed audit repository, approval tokens, sandbox writer, memory policy, and evaluation endpoints.
After approval, the API writes a simulated state revision to a read-write volume. A separate recovery-observer container mounts that state read-only, has a read-only root filesystem, receives no secrets, exposes no host port, and accepts only GET/HEAD. The API correlates observation evidence with the trusted execution receipt and fails closed on stale, malformed, oversized, duplicate, or mismatched evidence.
Qwen is integrated server-side through the Qwen Cloud OpenAI-compatible endpoint. Five roles use validated JSON responses; the Remediator uses Qwen native function calling against server-owned proposal schemas. A model tool call creates only an unexecuted proposal and never bypasses human approval. The API key stays out of the browser.
The public deployment runs Docker Compose, Caddy, Nginx, the API, and the private observer on an Alibaba Cloud Simple Application Server in Shanghai. Caddy provides trusted TLS, HSTS, and an HTTP-to-HTTPS 308 redirect. The repository also includes an auditable ECS Terraform alternative.
Challenges we ran into
The hardest boundary was separating retrieval from admission. A plausible past action is not reusable evidence. We preserve failed candidates for audit while making them impossible to recall as guidance.
Cross-session replay introduced another constraint: a new incident must not inherit old execution state, yet it must see a newly verified memory and reject the record that memory superseded. We encoded that lifecycle in automated API tests instead of relying on a staged UI story.
Evaluation honesty was equally important. Deterministic fixtures can prove policy invariants, but they cannot prove model quality. The repository therefore keeps fixture validation and credentialed Qwen measurement as separate artifact classes.
Accomplishments that we are proud of
- A four-part outcome gate for approval, application, independent observation, and exact evidence citation.
- A private read-only observer with no secrets or host port and strict receipt correlation.
- Inspectable filters for failure status, scope, version, expiry, supersession, symptom relevance, and context budget.
- A tested
run -> reject -> replan -> approve -> verify -> promote -> new-session replay -> recallpath. - A six-scenario, three-strategy fixture suite plus a separate repeated-run Qwen ablation runner.
- A public Alibaba Cloud deployment with four healthy containers and a verifiable source commit.
What we learned
Useful agent memory is a policy system, not just a retrieval system. Outcome, scope, version, provenance, and time belong to the memory itself. Forgetting should usually suppress unsafe guidance while preserving the audit trail rather than erase history.
What's next
A realistic adoption path starts in read-only shadow mode: import structured incident records, preview every admission decision, and let service owners review the policy without invoking tools. Later stages would add durable multi-tenant storage, signed policy bundles, read-only production observability sources, and measured reviewer overrides before considering any real actuator.
Current evidence boundary
The public host is currently an explicit zero-cost deterministic fixture rehearsal while a late-submission request is pending. It proves the Alibaba Cloud hosting, container isolation, TLS, approval, memory-policy, and sandbox lifecycle paths. It is not presented as a credentialed Qwen result. The final Qwen measurement and public demo video will be added only after the real-account artifact is complete and reviewed.
Links
Built With
- alibaba-cloud
- caddy
- docker-compose
- express.js
- nginx
- node.js
- qwen-3.7-plus
- qwen-cloud
- react
- terraform
- typescript
- vitest