RunSkeleton — a governed AI experiment skeleton
Kiroween 2026 · Category: Skeleton Crew
About the project
Every AI experimenter knows the failure mode: a promising number scrolls past in a terminal, and a week later nobody can answer the basic questions. Which config produced it? Which data snapshot? Did the evaluation set change halfway through — and was any of it ever written down?
RunSkeleton is a reusable skeleton template for governed AI experiment runs. The inspiration is a simple rule: no record, no result. An experiment that lives only in scrollback does not count. RunSkeleton wraps any experiment in a thin, strict layer of governance so that a run is:
- proposed against a written, falsifiable hypothesis before it exists,
- admitted or rejected by a deterministic gate — a program, not a vibe,
- executed inside a fixed folder layout with its config frozen inside,
- evaluated pairwise against the current champion,
- and recorded, step by step, in an append-only, hash-chained ledger.
The impact we care about is trust in experimental evidence. Recipe-search workflows generate hundreds of candidate runs; if even a few results are misattributed, the whole search climbs the wrong hill. RunSkeleton makes the classic failures — lost configs, silently changed evaluation data, unrecorded verdicts, reused run IDs — structurally hard instead of merely discouraged, and it is small enough to copy into a project in minutes.
How it was built
The template core is template/skeleton.py, a zero-dependency Python 3.10+
standard-library CLI (init / record / verify / list) — a governance
tool must run anywhere, including air-gapped machines. init creates the
workspace layout (runs/, ledger/, eval/dev, eval/sealed, config/),
freezes run_config.json inside the run folder, and appends a
run_proposed entry before any artifact exists. Duplicate run IDs are
refused outright.
The ledger is one JSON object per line in ledger.jsonl. Each entry carries
prev_hash and entry_hash = sha256(prev_hash + canonical_json(entry)), so
verify can recompute the chain and name the first tampered entry. Verdicts
(admitted / rejected / promoted / rolled_back) can only be recorded
by the gate actor — the CLI raises an error if a proposer or runner tries to
write one — and no event can be recorded for a run that was never proposed.
Eight pytest tests cover the round-trips, tamper detection, duplicate-ID
refusal, and the gate-actor rule.
Two demo apps, each a single dependency-free index.html with embedded
sample data in exactly the shape the skeleton produces, prove the outputs
are consumable read-only:
- Ledger viewer — the ledger as a filterable table (event type, verdict, free-text search) with a stats strip. Its sample data tells a real story: an admission rejected for a stale data snapshot, a promotion rolled back on negative paired evidence.
- Paired-evaluation dashboard — champion vs. challenger across 12 paired tasks: per-pair deltas, mean paired Δ, win/loss record, and an SVG delta chart.
How Kiro was used
This project was built spec-first, and the Kiro configuration is part of the
deliverable: everything lives in /.kiro at the repository root, never
gitignored.
- Specs.
.kiro/specs/governed-experiment-skeleton/holdsrequirements.md(EARS-style: WHEN … THE SYSTEM SHALL …, six requirement groups),design.md(ledger and hash-chain design, CLI surface), andtasks.md— the checklist the repo was actually executed against. - Steering.
product.md(the six non-negotiable rules),tech.md(stdlib-only, JSONL, single-file apps), andstructure.md(where files belong) all carryinclusion: always. The effect: generated code followed the rules — canonical JSON, UTC timestamps, never-reused IDs — instead of improvising around them. - Hooks.
run-checks-on-saveruns the pytest suite whenever a template Python file is edited, so a broken skeleton is caught at edit time.block-sealed-eval-writesblocks writes touchingeval/sealed/**outside the promotion flow — a rule enforced by tooling, not by memory. - Spec mode vs. vibe coding. The ledger core — hash chain, gate-actor enforcement — was built strictly from the spec, because its correctness is what the whole skeleton stands on. The demo apps' presentation layer was vibe-coded against the steering rules, which was faster and safe precisely because steering pinned the data shapes and the read-only constraint.
Challenges, accomplishments, and what's next
The hardest decision was scope: a skeleton that does too much stops being a skeleton, so RunSkeleton deliberately is not a training framework or an orchestrator. The second challenge was honesty in the demo data — the samples had to show governance biting (a rejection, a rollback), not just successes. What we are proudest of is the symmetry: the project is a governance skeleton, and it was itself built under governance — specs before code, steering always on, hooks guarding the sealed set.
What we learned: constraints stated once, in steering, beat constraints repeated in every prompt.
Next: wire the apps to load a real ledger.jsonl export instead of embedded
samples, add a promotion-gate command that opens the sealed set exactly
once, and dogfood the skeleton on a live recipe-comparison study.
