Incidentgym

A company-calibrated cyber war-game with deterministic, explainable, Aurora-audited after-action scoring

Comment

Inspiration

Most incident-response (IR) training is broken in two ways. First, it's episodic — a consultant-run tabletop once a year that tells you almost nothing about whether your team is ready today. Second, the new wave of AI-driven trainers grade you by having a language model judge how plausible your written answer sounds. For a security product, that's a credibility hole: plausible prose is not a contained breach.

We wanted IR training that's continuous, measurable, and — above all — defensible: a readiness score a CISO can put in front of the board and an auditor can't wave away. That meant the outcome of a decision had to be computed from what would actually happen in the environment, not graded on vibes.

What it does

IncidentGym is "the gym for incident response" — a multi-tenant B2B SaaS where security teams drill on realistic cyber incidents and get a score grounded in their environment, not an opinion. (All scenarios and environments are original and fictional.)

  • Pick (or register) your industry and company size and you get an incident unfolding across a service-dependency graph calibrated to a company like yours (finance, logistics, B2B SaaS).
  • Make turn-by-turn response decisions — isolate a service, fail over, escalate, communicate — advised by a cast of AI personas (CEO, security lead, payments lead, comms officer).
  • Get a deterministic score. A pure graph-cascade engine propagates each decision through the dependency graph and computes concrete outcomes: blast radius, services impacted (criticality-weighted), recovery, and a causal trace of exactly what your choice changed.
  • A debrief that holds up. A blast-radius gauge, a baseline-vs-your-run scorecard, the impacted-services readout, "why each number moved," and counterfactuals showing what other moves would have done.
  • Team readiness for managers. Per-org leaderboards roll individual runs into an organizational readiness view over time.

The LLM drives the narrative and the advisor personas; it does not own the numbers.

How we built it

  • Frontend + API: Next.js (App Router) + tRPC + Prisma, deployed on Vercel (with serverless function duration raised so a full LLM turn completes within budget).
  • Database — Amazon Aurora PostgreSQL (Serverless v2) as the system of record, doing real work rather than just storing rows:
    • Tenant isolation as a database guarantee. Multi-tenancy isn't only enforced in application code — it's a composite foreign-key invariant in the schema, so a cross-tenant write is rejected by PostgreSQL itself (live Postgres error 23503).
    • An auditable decision ledger. Every scored decision — the action, the resulting metrics, the causal effects — is persisted in Aurora. That's what lets a manager track readiness over time and prove it to an auditor.
    • Company-calibrated environments keyed to each tenant's industry and size.
  • Scoring engine: a pure, deterministic graph-cascade resolver, separate from the LLM — same inputs always produce the same score, which is what makes it auditable.
  • Auth / multi-tenancy: Clerk Organizations in production (dev-cookie mode locally); the same composite-FK org scoping applies in both modes.
  • LLM: narrative + persona generation behind a metered client with bounded per-tenant spend.
  • Safety: the entire deterministic-scoring slice shipped behind default-OFF feature flags, so the legacy path stays byte-for-byte unchanged — verified with flags-off vs flag-on end-to-end tests.

Challenges we ran into

  • Deterministic scoring that still feels alive. The hard part was drawing a clean line between the LLM (narrative, personas, flavor) and the engine (the numbers). Keeping the resolver pure — no hidden randomness, stable ordering — is what makes the score reproducible and defensible.
  • Isolation you can prove, not just assert. Pushing tenant isolation down to a composite-FK invariant so the database rejects a cross-tenant write took schema care, but it turns "trust us" into a demonstrable guarantee.
  • Exactly-once turn finalization. Scoring + persistence had to be one idempotent, resumable operation so a refresh or a second tab can't double-advance or double-charge a turn.
  • Serverless + LLM latency. A real LLM turn doesn't fit a default serverless timeout; tuning Vercel function duration and Aurora connection behavior (so a cold scale-to-zero resume doesn't stack on top of LLM latency) took iteration.
  • Shipping risky changes safely. Putting the whole new scoring path behind default-OFF flags let us land a large change under a deadline without ever breaking the working app.

Accomplishments that we're proud of

  • Deterministic, environment-grounded scoring — a score you can defend and audit, not an LLM grading prose.
  • Tenant isolation enforced by the database, demonstrable as a live PostgreSQL foreign-key rejection.
  • An auditable, per-decision ledger in Aurora that powers both the player's debrief and a manager-facing readiness rollup.
  • A debrief that explains itself — blast-radius gauge, baseline-vs-your-run, impacted services, causal chain, and counterfactuals.
  • Shipped a major capability behind feature flags with a green test gate, and verified the flags-off regression — the working product never broke.

What we learned

  • For a security product, how you score matters as much as what you score. "Auditable and reproducible" isn't an implementation detail — it's the whole pitch.
  • The database can be the backbone, not the plumbing. Encoding a security invariant (tenant isolation) as a foreign-key constraint made the guarantee stronger and easier to demonstrate than any amount of application code.
  • LLMs are best as narrators, not judges when credibility is on the line.
  • Feature flags plus a real test gate make it safe to ship ambitious changes against the clock.

What's next for IncidentGym

  • Bring your own topology — drill on your real environment, not a generic one.
  • Deeper team readiness — richer leaderboards and a CISO rollup of where the org is strong and where it's exposed.
  • Guided & expert modes — multiple-choice drills for less-specialist players (legal/comms/exec tabletop), free-text for seasoned responders.
  • Benchmark your agents, not just your people — because the scoring is deterministic and topology-grounded, the same engine can evaluate how well an autonomous IR agent handles an incident. As security shifts toward agentic response, "how good is our agent, on our environment?" becomes a question that needs a reproducible answer.
  • Real-time multiplayer — team-vs-incident drills on the same authoritative engine.

Built With

Share this project:

Updates