FaultLine

Master infrastructure debugging by fixing real failures: not symptoms.

Comment

## Inspiration

FaultLine grew from my experience leading technical communities around Kubernetes, GPUs, HPC, and large-scale compute infrastructure.

I often saw people understand a concept in theory but get stuck because they could not recreate the environment where a failure occurred. Reproducing GPU and HPC infrastructure can require expensive hardware, cloud credits, cluster access, and hours of setup—far too much for someone who simply wants to investigate one failure mode.

In our communities, we addressed this by running focused labs built around specific situations. Instead of recreating an entire production system, each lab isolated one realistic problem and let learners practice observing, reasoning, fixing, and verifying.

FaultLine turns that approach into an accessible, reusable learning platform.

## What it does

FaultLine is an adaptive infrastructure debugging gym. It presents realistic incidents inside safe, simulated environments without requiring learners to provision expensive infrastructure.

Learners can:

  • Investigate evidence through a constrained terminal
  • Commit to a hypothesis and confidence level
  • Edit and apply Kubernetes manifests
  • Request progressive hints
  • Verify changes with a deterministic grader
  • Explain the mechanism behind a successful fix
  • Complete spaced-review drills and transfer challenges
  • Earn evidence-based badges and signed credentials

The grader distinguishes between a correct fix, partial progress, a symptom-only workaround, and a forbidden shortcut.

For example, deleting a NetworkPolicy might remove a connection error, but it also removes the intended security boundary. FaultLine detects that distinction and rewards the smallest correct change.

## How we built it

We built the frontend with React, TypeScript, and Vite, supported by a Node.js and Express backend. Zod validates structured contracts, YAML tooling parses infrastructure manifests, and Vitest tests the deterministic grading and progression logic.

We deliberately separated AI generation from grading. AI can design and improve a lab, but it never decides whether a learner passed. All verdicts come from deterministic checks and are signed using Ed25519.

Custom generation uses a layered pipeline:

  1. A router identifies the concept, difficulty, learning objective, and target misconception.
  2. A fault designer writes a working manifest first, then introduces one causal mutation.
  3. An independent adversary searches for symptom-only fixes and forbidden shortcuts.
  4. A pedagogy stage produces the brief, hypotheses, hints, explanation signals, and investigation path.

  5. A deterministic verifier proves that the initial state fails, the reference fix passes, and every shortcut is rejected.

  6. A critic evaluates realism, solvability, and single-fault scope, routing corrections back to the responsible stage.

Validated scenarios are stored with provenance, validation reports, signed reference results, and content-addressed identifiers. They can be replayed through stable links and promoted into the mastery library after repeated clean solves.

## Challenges we ran into

The first major challenge was accessibility. Not every learner has an API key, GPU access, cloud credits, or permission to use a managed AI service. Making custom generation depend on an API key would have recreated the same barrier FaultLine was designed to remove.

We solved this with several paths:

  • A built-in validated library requiring no key
  • Managed generation when configured
  • Temporary bring-your-own-key generation
  • A keyless AI CLI workflow

The CLI workflow produces a staged prompt that learners can run with an AI tool they already use. They can then import the resulting portable .faultline.json artifact. Before the lab opens, FaultLine runs the same deterministic safety and correctness checks used for managed generation.

The second challenge was generating labs that were not merely plausible, but provably usable. Every scenario must remain namespaced and safe, contain exactly one causal mutation, provide enough evidence to solve it, begin in a genuinely broken state, and include a minimal reference fix.

Shortcut detection was particularly difficult. We needed to recognize changes that made the visible symptom disappear while weakening the system’s intended safeguards. Introducing an independent adversary stage significantly strengthened this part of the pipeline.

## Accomplishments that we're proud of

We are especially proud that FaultLine keeps deterministic software—not an AI model—as the final authority.

Other accomplishments include:

  • A complete no-key learning experience
  • Layered generation with independent adversarial review
  • Content-addressed scenario storage and deduplication
  • Stable replay links for generated labs
  • Automatic promotion based on distinct clean solves
  • Detection of symptom-only and boundary-weakening fixes
  • Confidence calibration using the Brier score
  • Review streaks based on due retrieval rather than logins
  • Transfer-proven concept badges and clean capstones
  • Shareable Ed25519-signed credentials
  • A test suite covering grading, storage, decay, drills, validation, and session safety

We are also proud that the system makes sophisticated GPU and infrastructure concepts approachable through focused situations rather than requiring learners to reproduce an entire production environment.

## What we learned

We learned that AI is most effective here as a content and pedagogy system, not as a grader. Models are valuable for designing realistic situations, challenging assumptions, and producing explanations. Deterministic code is better suited to issuing verdicts and enforcing safety boundaries.

We also learned that a successful fix is not automatically evidence of understanding. A learner might guess correctly, use a hint, or remove an important safeguard. Stronger evidence comes from a clean investigation, a causak explanation, a spaced review, and successful transfer to a different surface.

Confidence is part of that learning process. FaultLine uses the Brier score to compare confidence with correctness:

[ \text{Brier}=\frac{1}{N}\sum_{i=1}^{N}(p_i-y_i)^2 ]

This lets the platform recognize calibrated wins and flag overconfident misses.

Most importantly, our community experience taught us that learners rarely need a larger environment—they need the right failure, isolated clearly, with enough evidence to reason about it.

## What's next for FaultLine

Next, we want to expand FaultLine with:

  • More GPU, HPC, Kubernetes, networking, and AI-serving scenarios ( + Option to add custom and publish it )
  • Richer runtime-behavior simulations beyond manifest inspection
  • Safe, allowlisted evidence scripts for generated labs
  • Additional far-surface boss challenges
  • Classroom cohorts with anonymized, mastery-based leaderboards

How Codex Multi-Agent Thinking Improved the Experience

Codex’s multi-agent approach helped us break the problem into specialized roles rather than treating content generation, safety, gamification, and validation as one large task. Different agents focused on scenario design, shortcut detection, pedagogy, deterministic verification, progression, storage, and user experience.

This made the final learning flow more coherent and exposed important edge cases, especially situations where learners could bypass a core concept while still appearing to resolve the incident.

Codex also helped assess and refine the UI through its built-in browser capabilities. By reviewing the experience directly in the browser, we were able to identify usability issues, validate interactions, improve visual consistency, and polish the overall flow.

The multi-agent process also helped shape the gamification loop around meaningful evidence:

Hypothesis → Investigation → Verified Fix → Explanation → Spaced Review → Transfer Challenge → Signed Achievement

Rather than rewarding clicks or speed, FaultLine rewards clean solutions, calibrated confidence, retained knowledge, and the ability to apply a concept in a new situation.

• Session ID: 019f85f9-7f26-7ff0-be0d-7cbf97b1bf54

Built With

Share this project:

Updates