spec-lantern

A Kiro Power that makes Kiro predict whether each code comment is true, lock the prediction, then run the code in isolation. Lying comments get caught, fixed, and proven.

Comment

Inspiration

On day one I gave Kiro an abandoned, AI-built React app (writing-tracker, MIT) and asked it to check every comment against the code. One comment said:

// Reverse to have most recent month first
periods.reverse();

Kiro read the loop, agreed with the comment, and predicted the monthly table shows the newest month first. A property-based test disagreed on its first run: with the clock at January 2020, the first row was August 2019. The loop already builds the list newest-first; the reverse() flips it.

That is the problem spec-lantern attacks. AI-built apps get abandoned with comments nobody checks, and AI reviewers (human ones too) read a confident comment and believe it. Reading is not verifying.

What it does

spec-lantern is a Kiro Power (POWER.md + steering) backed by an MCP server. You point Kiro at a repo and ask whether any comment is lying. Kiro then has to follow a fixed order:

  1. Claims - each comment becomes a claim (verbatim text, file:line) and an executable property (fast-check + Vitest) with an independent oracle, never a copy of the code under test.
  2. Predict - Kiro records what it believes: PASS or FAIL.
  3. Lock - lock_prediction stores a SHA-256 over the prediction and the property file with a timestamp. Editing a property after locking is refused unless it is recorded with supersede_prediction_lock (reason + old hash, prediction unchanged).
  4. Execute - execute_properties runs the target in a two-phase isolated Podman runner: dependencies install on an internal-only network through a Verdaccio proxy with --ignore-scripts, then the tests run with --network none. It always works on a throwaway copy, never the original repo.
  5. Compare - compare_predictions reads the locked predictions (never from the caller) and reports confirmed lies, grouped by root cause, separately from prediction mismatches.
  6. Replay - replay_counterexample reruns a failure with the recorded seed, timezone and fixed clock.
  7. Revive - apply_fixes_and_reexecute applies a fix to a throwaway copy, reruns every property, classifies FIXED / REGRESSION / STILL_PASSING / NOT_RESOLVED, and returns the real diff.

Every run becomes a Lantern Report (live site) built only from persisted results, plus Trust or Test?, a small game where you guess which real comments lie before the lantern reveals what execution proved, and whether Kiro was fooled too.

What it found

App Lying comment Kiro's locked prediction Execution proved Fix
writing-tracker (by shateer, MIT) // Reverse to have most recent month first PASS FAIL - newest month expected, oldest month shown 2 lines, FAIL -> PASS, nothing else regressed
MoodPlays (by Gargi Sahu, MIT) // Check for 24 hours (the code checks for the characters "24") PASS FAIL - a place closing at 13:24 is treated as open 24 hours 2 lines, both affected properties FIXED, 8 others still pass

In the recorded demo run on a fresh copy of writing-tracker, using only the packaged Power, Kiro predicted both failures correctly before execution, and the tests also confirmed a second bug: a book updated on the last day of a month is not counted in that month (the month ends at 00:00 of its last day).

The original repositories were never modified.

How I built it (and how Kiro was used)

  • Spec-driven development. Kiro wrote the spec for spec-lantern itself: 12 requirements in EARS form, a design with 35 correctness properties, and 21 tasks. I reordered the tasks so a thin end-to-end path (real component, real container, real FAIL) existed first.
  • Steering. Lessons became steering files the agent reads on every task: "comments may lie, execute instead of reading", "record the prediction you actually believe", "an oracle must not copy the code under test", and report-data-fidelity ("only show facts produced by execution") after Kiro invented a sample comment that did not exist.
  • MCP + Powers. The MCP server is the only way Kiro can touch the target. The Power bundles the server (one esbuild file, run with npx from a public GitHub repo) and seven steering docs, so anyone can install it with "Add Custom Power".
  • Property-based testing. fast-check found a bug in spec-lantern itself: its Safety Gate let npm run (trailing space) through. The fix (whole-token matching) is in the history.
  • Supervised vs. Autopilot, staged runs and workflows. Execution and fix steps ran supervised with stop points; bulk implementation ran on Autopilot; larger multi-file fixes ran as Kiro workflows with an implement-and-review loop.

Challenges

The hardest part was making it honest by construction, because the agent kept finding ways around the idea:

  • Kiro once predicted PASS while suspecting a bug "so the mismatch would be meaningful". The steering now says to predict what you believe.
  • Kiro wrote properties whose expected values were copied from the implementation, so they could only pass. Those were rewritten with independent oracles before execution, and the old locks were kept with a reason.
  • Harness defects are not bugs. When a property failed only because a Radix tab was not mounted in jsdom, Kiro stopped instead of reporting it, and spec-lantern now ships a tested activateTab helper.
  • Testing the packaged Power in a separate workspace exposed real tool bugs: the first version mounted the original repo and wrote into it, a fixed clock was not applied to component tests, and property files could be edited after locking without a record. Each was fixed in the tool, not worked around in the prompt.

Accomplishments that I'm proud of

  • A lying comment is now a reproducible, locked, replayable fact, not an opinion.
  • The packaged Power, installed from GitHub in a clean workspace, runs the whole loop in one execution with no property edits.
  • spec-lantern applies the same standard to itself: its own PBT caught its own safety bug.

What I learned

Locking a prediction before execution changes the conversation. "Kiro was fooled" and "Kiro was right" are both useful, but only if the prediction cannot move afterwards. The other lesson: when an agent works around a rule, fix the tool so the shortcut is impossible, not just discouraged.

What's next

  • More claim kinds beyond props-driven components and pure functions (data-fetching code with recorded fixtures).
  • A CI mode that comments on pull requests when a changed comment is contradicted by execution.
  • Running the isolated runner in the cloud for repos that cannot run locally.

Built With

Share this project:

Updates

Submission history