AURA Systems Lab

Real Linux and sched_ext labs that test whether a fix transfers—and issue a verifiable Intervention Receipt.

Comment

Inspiration

Most operations labs stop when a service turns green. That can reward a lucky restart, a one-host patch, or a fix that silently harms healthy systems. Recovery is useful, but it is not evidence that the operator understood the cause.

AURA Systems Lab treats every operational intervention as a falsifiable claim:

Machine recovered. Understanding not proven.

What it does

AURA is an evidence-first learning environment for real Linux incident response and kernel scheduling.

Its learning loop is:

Observe → Predict → Inspect → Contract → Apply → ACK → Verify → TransferGate → Receipt → Reflect

The signed v1 scenario families are:

  • O1 — a drifted multi-node Nginx and Ansible web fleet
  • O2 — a broken Linux service identity and permission chain
  • K1 — a CPU noisy-neighbor problem
  • K2 — an interactive-latency versus batch-throughput SLO problem

The hosted capability panel is authoritative: a lab is clickable only when its real worker is available. Missing infrastructure is shown as unavailable, never replaced with a mock success.

A learner begins with competing hypotheses, records a prediction, chooses bounded probes, and receives immutable EvidenceRefs from a real sandbox or kernel worker. The learner then submits a typed InterventionIR containing the hypothesis, supporting evidence, falsifier, actions, postconditions, invariants, and rollback plan.

Primary recovery is not enough. AURA's deterministic TransferGate evaluates:

  1. primary recovery,
  2. a hidden same-cause twin,
  3. a healthy negative control,
  4. repeat and idempotence, and
  5. reversibility.

A fix that only makes the primary system green remains RECOVERED_UNPROVEN and receives no verified receipt. A passing intervention produces a downloadable Intervention Receipt that binds its evidence, environment fingerprint, actions, before/after facts, TransferGate verdict, rollback history, model-assistance disclosure, and SHA-256 hashes.

The GPT-5.6 Evidence Coach is deliberately non-authoritative. It can read only learner-selected, already-unlocked EvidenceRefs. It returns schema-validated claims, citations, unknowns, safety notes, and a suggested next probe. It has no execution tools, cannot inspect hidden worlds, and cannot grade an attempt.

Judge Mode provides a no-login evaluation path through a real incident attempt. It is designed to contrast a signed bad intervention that may recover the primary world but fails causal transfer with a signed gold intervention that must pass every gate before a receipt can exist.

How we built it

The web experience uses React, TypeScript, Vite, xterm.js, and ECharts. A Python and FastAPI control plane owns the attempt state machine, signed scenario registry, active diagnosis, InterventionIR validation, TransferGate orchestration, receipts, and proof projections.

PostgreSQL stores metadata and append-only events. Versioned S3-compatible MinIO stores evidence, traces, and receipts. The API and execution workers are separate trust boundaries: the API never receives a container-runtime socket, and internal requests use bounded authenticated contracts.

Incident attempts run in digest-pinned rootless Podman environments with per-attempt internal networks, private volumes, cgroup limits, seccomp, and AppArmor. Scenario packages are Ed25519-signed and fail closed if their manifests, checksums, signatures, or image digests do not match.

Kernel scenarios run on a separate Linux x86_64 worker with real eBPF and sched_ext actuation, delegated cgroups, exact request/profile/generation ACK readback, an independent watchdog, bounded recovery, and verified final unload. The public deployment exposes only high-level judge-safe operations; judges need no root access or API key.

GPT-5.6 is called server-side through the OpenAI Responses API with Structured Outputs, a bounded evidence allowlist, redaction, timeout and retry controls, and one schema-repair attempt. Deterministic graders—not the model—retain verdict authority.

A real counterexample, not a scripted green path

The central product idea is the counterexample. A bad fix can restore the visible primary service while failing the hidden same-cause world. In that state AURA explicitly refuses to issue a receipt.

The gold intervention is then evaluated only after a real reset and must pass all TransferGate properties. Live and replay results are always distinguished. Missing evidence remains unknown rather than becoming a fabricated green result.

How Codex and GPT-5.6 contributed

AURA Systems Lab meaningfully extends the documented pre-event AURA-SCX kernel prototype at commit e37fc1bc204d41bd55d50a6d70dc5d9da6f568c6.

That baseline contained an early CLI, sched_ext/BPF scheduler and observer prototypes, cgroup controls, and benchmark scripts. During OpenAI Build Week, the project was transformed into the complete Systems Lab product: the FastAPI control plane, React application, signed scenario system, isolated incident runtime, four-world TransferGate, GPT-5.6 Evidence Coach, Intervention Receipts, Judge Mode, proof pipeline, PostgreSQL/S3 persistence, and two-host deployment. The native kernel path was also substantially reworked into an isolated worker with typed privilege separation, delegated cgroups, watchdog recovery, and evidence-bound evaluation.

I made the core product and safety decisions: recovery must not equal understanding, success must survive hidden and healthy worlds, AI must never grade or execute, unavailable hardware must fail closed, and the final deployment must not use mock success paths.

Codex was the primary implementation collaborator. It accelerated architecture decomposition, Python/FastAPI and React/TypeScript implementation, schemas, contract tests, scenario signing, security review, deployment manifests, and live-host debugging. On the production hosts it helped diagnose rootless Podman network, filesystem, systemd sandbox, cgroup-v2, BPF map ownership, watchdog, and service-lifecycle failures.

GPT-5.6 contributes twice: through the Codex-assisted build workflow and as AURA's evidence-bounded runtime coach.

Challenges we ran into

The hardest part was preserving truthful semantics across several privilege boundaries. A 200 response cannot mean a sandbox is isolated, an ACK cannot mean a scheduling policy worked, and a recovered primary host cannot mean an intervention transfers.

Real deployment exposed problems that portable tests could not: rootless container MAC stacking, cgroup-v2 delegation, immutable image identity, BPF reference ownership, private CNI name resolution, worker cleanup, observer lifecycle, and safe scheduler unload. Each had to be fixed without weakening the product contract.

Accomplishments that we're proud of

  • A real bad fix can be rejected even when it restores the primary service.
  • The model can assist reasoning without becoming an execution or grading authority.
  • Incident and kernel execution are isolated onto appropriate workers.
  • Receipts and proof artifacts are hash-bound to the deployed candidate.
  • Missing evidence remains unknown rather than becoming a fabricated green result.
  • Judges can exercise the hosted path without an account, API key, rebuild, or root access.

What we learned

Operational education becomes much more rigorous when an intervention is treated like a property-tested program. Hidden variants test causal transfer, healthy controls expose collateral damage, repeat worlds test idempotence, and rollback verification tests whether safety claims are real.

We also learned that AI assistance is more useful when it is constrained by evidence references and explicit unknowns instead of being asked to invent a confident answer.

What's next

Next steps include more signed Linux scenario packs, cohort analytics for instructors, additional kernel workload families, and a scenario-authoring SDK that preserves the same deterministic grading and evidence-integrity boundaries.

Try it

No login or credentials are required.

  1. Open Judge Mode.
  2. Confirm the live capability panel.
  3. Start the O1 incident path and follow its signed bad/gold contrast.
  4. Open an issued receipt and verify its hashes.
  5. Visit Proof & Evaluation.
  6. Open the Kernel Lab for live sched_ext, exact ACK, workload, watchdog, and recovery evidence.

Built With

Share this project:

Updates