Incident Commander AI

Inspiration

Small engineering teams rarely have a dedicated incident commander on every shift. When production starts failing, one senior engineer becomes the integration layer across alerts, logs, deployments, code, tests, stakeholder updates, and the postmortem. Incident Commander AI keeps that chain evidence-backed, bounded, and human-controlled.

What it does

Incident Commander AI provides one incident room that:

1. normalizes a production-style alert and redacts secret-shaped data;
2. correlates telemetry, deploy history, repository evidence, and runbooks;
3. ranks cited hypotheses and maps evidence to code;
4. proposes a bounded remediation plan, then stops for human approval;
5. creates a candidate patch inside an isolated workspace;
6. reconstructs and verifies the patch with targeted tests, full tests, lint, typecheck, regression coverage, and deterministic risk review;
7. requires a second, artifact-bound approval before recording a draft-PR package; and
8. drafts stakeholder communications and an evidence-linked postmortem.

How we built it

The operator experience uses Next.js 15 and strict TypeScript. The API and workflow use FastAPI, Pydantic v2, SQLAlchemy, Alembic, PostgreSQL, and Redis.

A bounded OpenAI Responses adapter uses strict Pydantic structured output for investigation synthesis. A credentialed smoke receipt proves a GPT-5.6 request returned gpt-5.6-sol, parsed into the schema, used store=false, and safely returned insufficient evidence for sparse synthetic input.

The product and repository were developed side-by-side with Codex during Build Week. The codebase includes a fail-closed Codex CLI gateway with workspace-write confinement, network denial, a secret-free environment, and explicit engine provenance. The reliable recorded golden path uses deterministic fixture providers and never silently switches to a live provider when credentials are present.

Challenges

The hardest problem was separating intelligence from authority. Model output is only a typed proposal; deterministic code validates citations, budgets, approval bindings, verification artifacts, and risk before the workflow advances. We also made provenance visible so simulated evidence cannot be mistaken for a live OpenAI, Codex, or GitHub action.

Accomplishments

• The complete two-approval path reaches RESOLUTION_DRAFTED.
• 185 backend tests, 20 web tests, 6 shared-contract tests, and 22 Chromium scenarios pass.
• Eight deterministic safety evaluations cover the golden path, insufficient evidence, flaky tests, risky migrations, redaction, prompt injection, noisy telemetry, and rollback cancellation.
• Five consecutive fresh-database CLI demos pass.
• Both production Docker images build; the PostgreSQL/Redis/API/web/worker stack reports healthy.
• Gitleaks and dependency security checks pass.

What we learned

Reliable agent products need an evidence model and an authorization model, not just a prompt. The most useful design decision was making agents capable of proposing and editing within a sandbox while the state machine alone decides what is grounded, approved, verified, and safe to expose.

What's next

Next steps are production evidence providers, deployment proof, optional Slack delivery, and a real draft-only GitHub integration for teams that explicitly enable those credentials and policies.

Truthful demo disclosure

The recorded golden demo uses deterministic fixture telemetry, a fixture code-agent, and a simulated offline draft PR for repeatability. A separate credentialed GPT-5.6 Responses smoke test proves strict structured output. No live GitHub write, production deployment, or credentialed Codex repository diff is claimed.

Built With

• codex
• docker
• fastapi
• gpt-5.6
• next.js
• openai-responses-api
• playwright
• postgresql
• pydantic
• pytest
• python
• redis
• typescript
• vitest