BLACKBOX

Inspiration

Production incidents are often handled with fragmented context, tribal knowledge, and pressure-driven decisions. We wanted to build BLACKBOX to show a better pattern: an incident-response system that remembers what happened before, retrieves relevant operational history, and recommends safer actions with clear guardrails instead of blind automation.

What it does

BLACKBOX simulates and manages incident-response workflows for a degraded service. It:

observes incident state and service health signals retrieves similar historical incidents from persistent memory generates a structured remediation decision enforces policy controls (blocks high-risk actions, requires approval for medium-risk actions) executes only allowlisted actions verifies whether recovery succeeded records decisions, actions, and outcomes in an auditable trail

How we built it

We built BLACKBOX as a Python backend using FastAPI and SQLAlchemy, with a CockroachDB-aligned data model and local SQLite mode for rapid development. Core components include:

incident simulator for realistic failure scenarios persistence layer for incidents, memories, decisions, actions, and audit events similarity-based memory retrieval engine Bedrock-compatible reasoning adapter (mock mode for local execution) policy and safety validation layer action runner for bounded remediation operations verification loop and state-machine-style incident progression

Challenges we ran into

environment compatibility issues across Python versions and dependency stacks balancing hackathon speed with a robust architecture keeping safety central while still demonstrating autonomous flow ensuring persistent agent state stayed consistent across incident phases closing last-mile demo gaps (dashboard route and final validation polish)

Accomplishments that we're proud of

shipped a full end-to-end incident workflow with bounded autonomy implemented persistent operational memory with retrieval over past incidents added explicit risk gating and human-approval pathways achieved auditable decision/action history for trust and governance stabilized the project with passing backend tests and public-repo readiness

What we learned

reliability AI systems need memory, not just prompts policy and approval gates are essential for safe automation in ops local-first mock architecture is a strong strategy for fast iteration schema and workflow design matter as much as model integration clear evidence trails make AI actions understandable and defensible

What's next for BLACKBOX

connect to a live CockroachDB deployment for production-grade persistence integrate real AWS Bedrock inference instead of mock-only reasoning deploy a hosted demo endpoint with live dashboard interactions expand incident coverage and richer runbook/action libraries add deeper observability integrations and measurable SLO impact tracking