Inspiration

Infrastructure recovery presents a domain in which multiple simultaneous failures present challenges to operators. In the face of communication outages, power failures, and equipment failures, operators must evaluate what to recover, what to defer, and when to declare backup infrastructure active.

We envisioned an interactive tool to help people explore tradeoffs during infrastructure recovery. Given a scenario in which users have control over limited resources and responsibilities, how might we help them recognize implications for the broader system without experimenting on actual infrastructure?

Our prototype, CommandGrid, takes the form of a fictional, noncombat infrastructure recovery simulator.

What it does

CommandGrid supports users in designing a recovery exercise, navigating its response to failures, and analyzing the evidence for the outcomes. Our scenario is an emergency-support facility in which communications are disrupted alongside power and equipment outages.

Users can

  • design an exercise by defining its duration, environmental conditions, reliability, allocation of resources and support teams, and access to backup power,

  • define objectives with measurable targets for uptime, counts of restored systems and equipment, availability of resources and support, and deadlines,

  • navigate the recovery by reviewing the terrain, active incidents, and costs and projected impacts of recovery operations,

  • explore approaches and strategies, including manual and automated recovery and pauses in the simulation,

  • and review reports that document the simulation's outcomes, evidence, telemetry, incidents, events, and accepted commands.

In the simulation, current system health differs from historical values. Although communications might be stable at the end of a mission, a communications objective might still be missed if outages occur during the exercise.

We implemented the frontend in React and TypeScript, with REST requests for mission configuration and commands and WebSockets for updates from the simulation.

The backend is implemented in Python and FastAPI to manage commands and progress and persist results. Using SQLAlchemy, the backend stores results in Postgres and TimescaleDB and SQLite for development and demonstration.

The core of the application is a deterministic simulation. Given an input configuration, random seed, and command schedule, the same outcomes will be produced to support comparative analyses of approaches.

Optional AI support provides language models to interpret incidents and recovery tradeoffs and reports, but the core of the simulation is focused on determining numeric results and objective evidence. Throughout the interface, mock responses and API provider replacements are noted, and users can access the core application without AI assistance.

We used coding agents for parts of the implementation, and image generation for the terrain. We implemented working UI elements, controls, and status separately from the terrain for performance and flexibility.

What we learned

We learned that a useful simulation provides both accessible controls and thorough evidence. Although users might benefit from an immediate sense of accomplishment upon achieving a target, they also need to understand what combination of decisions and conditions led to an outcome.

We developed an improved understanding of historical metrics, deterministic simulation, asynchronous state management, and persistent command handling. We also learned to define the role of AI support, in which language models can help explain the details of a scenario without being responsible for calculations and objective comparisons.

What we're proud of

We implemented a connected set of tools that support users in navigating a mission from configuration through live recovery and post-assessment. Automated testing in the backend and browser supports the reliability of the simulation, persistence, recovery controls, reports, and interface.

We are especially proud of supporting users in making decisions and exploring the implications while also making the evidence accessible in reports.

What's next

CommandGrid currently uses fictional locations and a reduced set of infrastructure for its simulations. We hope to conduct an evaluation with learners and trainers to determine the most valuable directions for the tool. We also hope to implement additional scenarios, richer resource and team scheduling, and multiplayer exercises in the future.

Built With

Share this project:

Updates

Submission history