Failure DNA

Inspiration

I've watched incident-response playbooks fail in the same way twice: the symptoms look familiar, so someone (or an agent) reaches for the fix that looked right last time — even when that fix already failed, or only worked under a different database driver, region, or config. Vector search and "memory agents" make this worse if similarity is treated as safety.

I built FailureDNA for Track 1: MemoryAgent because I wanted memory that doesn't just recall — it qualifies experience before an LLM acts. The question isn't "what happened before?" but "should we trust that memory now?"

What it does

FailureDNA is a persistent experience-memory layer for incident-response agents.

Each resolved incident becomes an episode: symptoms, environment fingerprint (driver, topology, config hash), action taken, and real outcome. On the next incident, the system:

Retrieves candidates with hybrid semantic + keyword search (pgvector + symptom overlap)
Labels each memory use, inspect, or avoid using deterministic rules — failed actions are never offered again; stale successes are flagged when the environment drifted
Hands bounded context to Qwen, which picks one allowlisted action as validated JSON
Persists the new outcome back to memory

You can compare three modes side by side: no memory, naive (similarity only), and failuredna. In the live demo, a known-bad restart_service is blocked while a still-relevant inspect_downstream is chosen — with zero repeated failures.

Try it: Live dashboard · API health

How I built it

As a solo developer, I built the full stack end to end:

Backend: FastAPI on Alibaba Cloud Function Compute (custom Docker container, port 9000)
Memory: ApsaraDB RDS PostgreSQL + pgvector (HNSW index), Alembic migrations, hybrid retrieval with Qwen text-embedding-v3
Reasoning: Qwen chat completions (DashScope, OpenAI-compatible) with strict JSON validation, single retry, and bounded timeouts
Safety boundary: Qwen selects actions; validity and outcomes stay deterministic — the model never decides whether a memory is safe or what happened in the simulator
Frontend: GitHub Pages dashboard (docs/) talking to the FC API cross-origin
Quality: Reproducible benchmark harness (5 scenarios, fairness controls, shortcut baselines), 199+ tests, deployment smoke scripts

Architecture: retrieve → fuse scores → validity gate → Qwen decision → validate → execute → persist. See the architecture diagram and deployment guide.

Challenges I ran into

Production on Alibaba FC was harder than the app logic. The default *.fcapp.run domain forces browsers to download HTML instead of rendering /ui — so I hosted the dashboard on GitHub Pages and kept the API on FC.

Cross-origin POST was blocked for days. Browsers send an OPTIONS preflight for X-Demo-Token + JSON. The FC gateway injects Access-Control-* headers, but the app still had to return 200 on OPTIONS — my early fixes gated on an env var that was empty at runtime, so preflight kept returning 405. The fix: unconditional OPTIONS 200, zero duplicate CORS headers from the app.

Docker image deploys failed with platform of image is unknown/unknown. BuildKit was attaching provenance attestations FC rejects. I had to use buildx build --provenance=false --sbom=false and verify manifests with imagetools inspect before each deploy.

Solo scope: memory store abstraction, pgvector pipeline, Qwen client, benchmark fairness, deployment docs, and a judge-friendly live UI — all one person, one repo.

Accomplishments that I'm proud of

Shipped a working live demo end to end: GitHub Pages → FC → RDS pgvector → Qwen, with Seed and Run working in the browser
Designed a clear split: similarity finds candidates; deterministic rules decide trust — not "let the LLM figure out if memory is safe"
Avoid enforcement removes known-bad actions from the candidate set before Qwen sees them — provable, not prompt-based
Built a fair benchmark with evaluator-only labels, isolated stores per mode, and shortcut baselines so wins aren't from one lucky action
Documented a real Alibaba Cloud path (ACR Personal Edition, VPC + RDS TLS, FC custom container) that others can follow
Went from "Failed to fetch" on demo day to a dashboard that shows the validity gate live

What I learned

Memory agents need a validity layer. Embeddings surface relevance; they don't prove a fix still applies after environment drift.
Platform behavior matters as much as application code. FC gateway CORS, attachment headers, and OCI manifest formats all affected the demo — none of that showed up in local curl tests.
Keep LLMs in a narrow lane. Qwen is strong at choosing among bounded options; deterministic gates handle safety and auditability.
Deploy early, probe with the browser. OPTIONS preflight and imagetools inspect would have saved hours if I'd checked them before assuming the API was "fine" from GET /health alone.
Solo hackathons reward boring engineering: migrations, health probes, smoke tests, and docs are what make a demo survive contact with judges.

What's next for FailureDNA

Custom domain on FC (e.g. cloudaimldevops.com) for a cleaner demo URL and optional same-origin /ui
More incident scenarios beyond the simulator — webhooks from PagerDuty/Opsgenie, real runbook actions behind a tool boundary
Memory lifecycle: expiry, invalidation, and supersession when a newer episode contradicts an old success
Multi-tenant memory with scoped episodes per service/team
Observability: export validity-gate decisions and Qwen choices as structured traces for post-incident review
Open-source polish: Helm/Serverless Devs templates, one-command deploy, and a public benchmark leaderboard

Testing for judges

Live dashboard: https://prabhakaran-jm.github.io/failuredna/
API health: https://prabhakaran-jm.github.io/failuredna/api.html

Open the dashboard and paste the demo token into Demo token (if required)
Click Seed demo memory (once)
Run incident — Memory mode: failuredna, Engine: deterministic
Switch Engine to live Qwen and run again

Demo token: Shared privately with the hackathon organizers for judging.
If you are a judge and do not have the token, contact the organizers or the project submitter.