Inspiration

When production breaks at 3 a.m., dashboards give you forty graphs and no story. The on-call engineer has to work out which service is sick, how badly, and what changed — all while half asleep. We wanted an incident view you can read in one glance, that a newcomer understands without training, and that is fun enough that people actually keep it open. Halloween gave us the metaphor: a system in trouble is a haunted house.

What it does

Haunted Ops is an incident war room drawn as a 2.5D haunted house at night. Four services — gate, orders, payments and ledger — are four rooms connected by corridors.

Reading the house

  • Ghosts = errors. More ghosts, angrier ghosts as the error ratio rises.
  • Fog = latency. Fog fills a room when its p95 latency climbs.
  • A dark room = the service is down.
  • A lantern and floating dust = healthy.
  • Room labels look like the side-panel pills: a level dot, the service name and the level word (payments · haunted); a red outline and dot mean an incident is open.
  • A legend ("What am I looking at?") explains every symbol, and a status line sums up the house (1 haunted · 3 calm).

Haunt levels

Every service gets one of five levels: calm, uneasy, haunted, possessed, dark. The level is the worst of three signals:

  • error ratio ≥ 5 % / 20 % / 50 %,
  • p95 latency ≥ 500 ms / 1 s / 3 s,
  • availability below 50 % → dark.

To avoid flapping, a level only rises after 2 polls in a row and only falls after 3. An incident opens at haunted and closes after 30 seconds of calm.

The Exorcist

When an incident opens, the Exorcist — a custom Kiro agent — walks into the most downstream sick room and investigates. It can only read, through tools of our own MCP server:

  1. query_metrics — error ratio and latency over time
  2. search_errors — matching error lines
  3. tail_logs — recent logs of the service
  4. recent_deploys — what was deployed and when
  5. diff_commit — what a suspect commit changed

It ends with submit_diagnosis: the culprit service, deploy and commit, a short summary, a fix hint and the evidence it cited. Every tool call is a step: a mint tick lights up on the ring on the room floor, a mint badge on the wall counts the steps (Hunting · 4 steps → Diagnosed), and the side panel lists each call with its time.

Hunts are fenced in: a credit budget per hunt and per day, a 90 s timeout, a cooldown, at most 40 hunts a day and a cap on tool calls. A kill switch (environment flag plus an API endpoint) stops every hunt at once. Tool output is treated as untrusted: it is sanitized, size-limited, and never followed as instructions.

Fix proposals — humans merge

The Exorcist can propose a fix: a separate agent writes a patch, and the server — not the agent — checks it before anything leaves the box: at most 5 files and 60 changed lines, allowed paths only, syntax and size checks (8 checks in all). It then pushes a new exorcist/<hunt> branch and opens a pull request. Nothing merges by itself: a person reviews and merges, and the room's lantern comes back when the fixed deploy is healthy.

Chaos lab and replays

Haunted-Lab is our second repo: a small four-service system with Prometheus metrics and a supervisor that triggers chaos scenarios on a timer — an error spike in payments, fog (slow responses) in orders and a dark room (outage) in ledger. Real hunts against the lab are recorded, sanitized and published as replays you can play at 1×, 2× or 4× and scrub, e.g. "Orders slows down after a deploy".

Accessible by design

Everything the scene shows is also text. Accessible mode (a header switch or ?mode=accessible, remembered by the browser) swaps the canvas for a full table; every room is a real button you can Tab to, with a focus ring that follows the room's outline; one polite live region announces changes (queued, at most 5, 2 s apart); text contrast is at least 4.5:1, labels never go below 14 px on phones, and reduced motion freezes or removes every animation. A hook checks on every save that the scene and the table still show the same facts.

Sound

Short sound cues for the moments that matter — a ghost appears, a room is possessed or goes dark, calm returns, a hunt starts, each step, a diagnosis, a merged fix. Sound is off by default; one click on "Sound off" turns it on with a soft confirmation. Cues are rate-limited (one every 0.7 s, steps at most every 3 s) so an incident never becomes noise, and the audio code is a tiny chunk loaded only after that click.

Try it

The public demo (?demo=1) plays a scripted 48-second incident loop so judges always see a haunting: payments gets ghosts and becomes possessed, orders fills with fog, ledger goes dark, the Exorcist hunts payments in five steps and names the bad deploy, and the house returns to calm. No login needed.

How we built it

  • Web: React 19 + Vite 8; the scene is rendered with PixiJS 8. Hybrid art: generated sprites for rooms, ghosts, the Exorcist and the moon, plus procedural effects in code (fog, glow, dust, rings, badges) — and a procedural fallback for every sprite, so the house still renders if an image fails.
  • API: NestJS + Prisma + PostgreSQL. The API polls Prometheus every 5 s (error ratio, p95, up), computes the haunt level, stores signals and incidents, and streams changes over Server-Sent Events with Last-Event-ID resume, a 500-event ring buffer and heartbeats.
  • Exorcist: a custom Kiro agent restricted to the tools of our exorcist MCP server (TypeScript, in the repo); the API runs hunts and records every step.
  • Deploy: one small EC2 instance (Debian), systemd + Caddy with a strict Content-Security-Policy and security headers. No SSH port: everything goes through SSM. Every release is a reproducible tarball with a checksum; a pre-flight of 24 checks and a smoke test run after each deploy, and the previous release stays ready for a one-command rollback.
  • Performance budgets enforced by a build check: main entry ≤ 95.5 KiB gzip, scene chunk ≤ 180 KiB, sprites ≤ 400 KiB, sound chunk 0.6 KiB loaded on demand.
  • Art & sound: sprites generated with Seedream 5.5 and sound effects with Mirelo SFX 1.5 (via Runware), then curated, trimmed and normalized by the team.

How we used Kiro

  • Spec-driven, end to end. 9 specs — foundation, ingest-haunt-score, scene-engine, exorcist-agent, fix-proposal-flow, replay, accessible-mode, deploy-demo, visual-refresh — with 177 EARS requirements, designs with correctness properties, and 173 tasks, all done. Even visual polish was a spec (visual-refresh, 71 tasks in 8 parts), so "make it prettier" had acceptance criteria and tests.
  • Property-based tests. Each correctness property became a fast-check test tagged // Feature: <spec>, Property N — 53 property test files, e.g. "every fact in the scene also appears as text in Accessible mode", "room labels never overlap at any screen size", "sound cues obey their rate limits".
  • Agent hooks (4). typecheck-test-after-task; scene-table-parity (on every scene save, check the scene and the table still agree); guard-shell (blocks destructive commands and secret paths); kiro-usage-log (one line per finished task in docs/KIRO_USAGE.md — about 190 entries).
  • Steering (8). product, tech, structure, security, scene-rendering, a11y-perf, mcp-tools and lab-chaos — so every session started from the same rules: read-only Exorcist, humans merge, budgets, contrast, no flashing.
  • Custom agents + MCP. The Exorcist and the fix agent are Kiro agents with the narrowest tool sets we could give them; the MCP server they use lives in the repo and is tested like the rest of the code.

Challenges we ran into

  • Agents in the wild. Our first live hunts against the lab did not land a clean diagnosis; we needed several runs, reading the tool traces each time, before the Exorcist named the right deploy. One demo hunt stopped with foreign_tool because a tool title started with Running: — we learned to treat every agent output strictly.
  • Untrusted text. The log sanitizer had a ReDoS: 562 ms for a 24 KB input. We caught it before release and rewrote it.
  • Spooky but readable. Dark scenes fight contrast. We had to keep labels from overlapping at every width, keep 14 px text on phones, avoid flashing, and make the focus ring follow the real room shape.
  • Sound in browsers. Autoplay rules silence audio until a user gesture, so the audio context is created inside the click and a soft cue confirms it works.
  • Tight budgets. The entry bundle stays under 95.5 KiB gzip while the scene, sprites and sound all load lazily.

Accomplishments that we're proud of

  • An incident view you understand in one glance — with a fully accessible twin that never disagrees with it.
  • An agent that can only read, with hard budgets and a kill switch — and a fix flow where a human always has the last word.
  • About 1,900 automated tests (unit, property and end-to-end), deterministic release builds, and every task traceable to a spec requirement.

What we learned

Specs are not only for features: writing visual polish as requirements and properties stopped endless "small fixes" and kept the scene and the table honest. Narrow tools make agents better, not worse. And a metaphor only works if it stays consistent: every ghost, fog bank and lantern had to mean exactly one thing.

What's next for Haunted Ops

  • Connectors for real observability stacks (OpenTelemetry, more metric back ends).
  • A multi-house view for many teams and many services per room.
  • Live fix proposals on a team's own repository, with the same human-merge rule.
  • Spatial sound and custom room themes.

Built With

Share this project:

Updates

Submission history