Inspiration
Most engineers do their first on-call shift with no practice. They learn incident response during a real outage, with customers affected and someone senior watching. Runbooks and old postmortems don't build the instinct to go from an alert to the evidence to the root cause under pressure. We wanted a place to practice on a system that is actually broken, where getting it wrong costs nothing.
What it does
Witching Hour is an on-call training game. A "haunting" injects a realistic production fault into a small sandboxed online shop (storefront, cart, payments, an async worker, Postgres and a Redis queue). Then:
- Page. You get an alert on a pager, like a real page.
- Investigate. Real Grafana dashboards, logs in Loki, and a restricted
hauntterminal (status, logs, grep, PromQL, queue, config, restart, scale). - Name the root cause. Which component failed, and why.
- Fix. The fix only counts once live Prometheus and Loki checks confirm it.
- Postmortem. A deterministic score (time to detect, root cause, verified fix, penalties for harmful actions), the evidence you found and missed, and a full timeline.
Some terminal commands are traps. Purging the orders queue deletes orders customers already paid for, and the postmortem explains why that cost you points.
| Haunting | Difficulty | What breaks |
|---|---|---|
| The Vanishing | easy | The queue worker stops silently. The storefront stays green while orders pile up. |
| Possessed Config | medium | Payments' DB pool is set to 1 and stored in Postgres, so restarting doesn't help. Then retries pile on. |
| Slow Whisper | hard | Cart leaks memory slowly; later, GC pauses stall every request. |
Daemonwatch, a Kiro Crew App, is the team side. A lead schedules the same drill for the whole team and compares everyone's scored postmortems. Crew agents rehearse drills on a weekday cron and write a blameless team postmortem.
How we built it
- Engine (TypeScript): applies, escalates, verifies and reverts hauntings across four fault types (Toxiproxy latency and timeouts, Docker stop and pause, config values, debug toggles). A per-resource ledger records each resource's original state, so reverts work in any order, even when hauntings overlap, and survive a crash. The engine is also an MCP server, so Kiro agents can run drills.
- Sandbox: Docker Compose with CPU and memory caps on every container, an internal network with no route out, read-only non-root containers with all capabilities dropped, and Prometheus, Loki and a provisioned Grafana dashboard.
- Game server: sessions with escalation timers, a WebSocket terminal that parses allowlisted commands into typed commands (nothing reaches a shell), scoring and postmortems, and an optional narrator that uses Claude on Amazon Bedrock. A guard checks every generated line, so the narrator can never invent a service or a number.
- UI (React, Vite, xterm.js): a 3 a.m. night shift with a Halloween streak. The pager is the centrepiece. A ghost with moods worries when you're paged, panics on escalation and cheers when your fix verifies. There's fog, bats, jack-o'-lantern difficulty meters, an "Exorcised" stamp, and candy corn for a perfect shift. All of it is hidden from screen readers and stops under
prefers-reduced-motion. - Hosting: one Azure VM behind Caddy with Let's Encrypt, running 24/7 with only ports 80 and 443 open. No password: each shift belongs to the browser that started it, through a random HttpOnly cookie.
How we used Kiro
Steering came first, then a spec for the engine with four correctness properties, which became fast-check property tests. Hooks validate hauntings on save, review UI changes for accessibility and keep a build journal. Custom agents write and QA hauntings, and the Crew App's agents run team drills over MCP. The full journal is in docs/kiro-journal.md.
Challenges we ran into
- Property tests found real bugs. The idempotence property found that re-applying an active haunting silently undid its escalation. The two-haunting property forced a redesign from per-fault undo to a ledger. One property even caught a bug in our own test double.
- Fakes can't catch everything. Every test passed against fakes, but the first run against the real sandbox found a wrong route label in a PromQL signal, a memory leak built on Buffers (outside the V8 heap, so the heap metrics never moved), and a red herring that wasn't actually red. Now a bot playtest plays every haunting end to end against the real sandbox.
- One public sandbox and no password. Without an access code, any visitor could land in someone else's shift. We tied each shift to the browser that started it, and a shift whose tab was closed ends after 5 minutes.
What we learned
Write down the properties before the code. Test your fakes as well as your code. A training tool for incidents has to be as safe as production tooling: every fault reversible, nothing reaching a shell, and narration that can't make things up.
What's next
Human playtests to tune pars and penalties, more hauntings (retry storms, slow dependencies, a bad deploy), and multiple sandboxes so a whole team can drill at once.
Built With
- amazon-bedrock
- azure
- caddy
- claude
- docker
- docker-compose
- fast-check
- grafana
- kiro
- kiro-crew
- loki
- mcp
- nginx
- node.js
- postgresql
- prometheus
- react
- redis
- toxiproxy
- typescript
- vite
- vitest
- websockets
- xterm.js
