KiroClops

Coding agents write animation code that is judged by eye. KiroClops measures it: OpenCV 5 tracks every frame against your reference video and catches regressions on held-out tests.

Comment

Inspiration

Motion is almost always specified by example: jump like that character, open like this panel, re-layout like this diagram. Coding agents can now write that code, but they cannot see what it renders. The result is judged by eye, constants are tuned by trial and error, and a fix that looks right in one case can quietly break another.

KiroClops started with a small question in September: could the feel of a platformer jump be measured from gameplay footage of Tiny Crate and reproduced by an AI coding agent in a different game? It grew into a bigger one. If OpenCV can track objects in every frame of a recording, can it give a coding agent eyes that measure, and prove whether the agent's result really matches the reference, even on test cases the agent never saw?

What it does

KiroClops lets a developer show the motion they want and have a coding agent build it, with OpenCV 5 deciding whether it really matches.

  1. Show. The developer gives a short reference clip (a video, or a recording of a reference implementation) and a plain-language request for their real application: a game, a web UI animation or an SVG diagram editor.
  2. Build. Kiro CLI edits the motion code in a copy of the app. Through eight MCP tools it asks KiroClops to re-record every candidate in headless Chromium or Godot and to measure it.
  3. Measure. OpenCV 5 tracks the selected objects frame by frame and extracts motion events: departure, rest, reversal, overshoot, connector attachment. Each time window is compared with the reference against limits fixed before the run, and Kiro gets back per-window errors and timing offsets it can act on.
  4. Verify. After at most three candidates, Kiro seals one. Only then are the held-out recordings revealed, so the verdict covers cases the agent never tuned on. A failed run stays failed.

Every run leaves a replayable evidence chain: sources, frames, measurements and an audit of every tool call the agent made. The same measurement also runs as an authenticated AWS service on a Graviton4 worker booted from the COOL (Cloud Optimized OpenCV) Marketplace image, which judges can call with one command.

How we built it

I built KiroClops alone, in phases, each starting from a written plan approved before any code ran. AI coding assistants wrote much of the code under those plans, and separate AI reviewer agents did the blind pixel and source reviews.

  • Measurement core (Python 3.10, OpenCV 5.0.0, NumPy; about 2,700 lines with no other dependencies). Objects are tracked with normalised cross-correlation template matching, Lucas–Kanade optical flow with a RANSAC similarity, and Canny/contour geometry. Template matching uses mirror and quarter-turn variants and a colour gate. A symmetric sub-pixel refinement cut positional error from 0.37 px to 0.05 px on synthetic shifts. SVG connectors are found with edges, connected components and line fitting, with a rule that accounts for the focus ring the browser draws around a selected node.
  • OpenCV 5 specifically. The new features module provides ALIKED keypoints and LightGlue matching. KiroClops uses them as a scale- and rotation-tolerant locator and as an independent cross-check of the template tracker. LightGlue in OpenCV 5.0.0 aborts the process when a later call is larger than the first one; I documented this and isolated it with a warm-up at the maximum size and a helper process. The measurement refuses to run on OpenCV 4.x.
  • Capture. Web targets are recorded in headless Chromium through the Chrome DevTools Protocol, with per-frame clocks. Games are recorded through a Godot adapter at a fixed physics rate. A recording that misses its timing limits is recorded again, never patched.
  • Agent loop. An MCP server gives Kiro CLI eight tools. A pre-tool-use guard limits what the agent can touch, and a post-run audit replays Kiro's own logs.
  • AWS. A CloudFormation stack puts Cognito, API Gateway, Lambda, S3, DynamoDB and SQS in front of a c8g.xlarge Graviton4 worker booted from the COOL Marketplace image (OpenCV 5.1.0-dev). Every result carries a receipt that names the engine, and the worker reproduces the local measurements on all 17 test recordings.
  • Evidence. Thresholds are pre-registered, source trees are frozen with SHA-256 manifests, and about 3,750 unit tests pass on a clean clone.

Challenges we ran into

  • A measurement the agent cannot game. An agent that only sees calibration cases can tune its code to them. KiroClops therefore reveals the held-out recordings only after the agent seals a candidate, and that is exactly where the hardest failure showed up: in the V6 web-panel run, Kiro's sealed candidate had a calibration error of 0.0067, yet it failed one held-out drag-and-release case it had never seen. The run is recorded as failed, as the protocol says.
  • Recordings whose timing can be trusted. KiroClops compares motion frame by frame, so a single scheduling hiccup during capture would look like a motion error. Each capture command has a fixed 120 ms deadline, and real stalls of 140–270 ms do happen. Instead of relaxing the deadline or patching frames, a recording that misses it is rejected and recorded again, the host's scheduling is sampled before each capture, and the capture process runs at a higher priority. In the V6 panel run, all 224 final recordings passed their timing checks.
  • A crash in OpenCV 5.0.0. LightGlue sizes its buffers on its first inference and aborts the whole process on a later, larger call. I mapped the failure over 1,470 keypoint-count pairs, then isolated it with a warm-up at the maximum size and a helper process. After that, 1,617 pairs ran in two random orders with identical results and no restarts.
  • Honest sub-pixel accuracy. Integer template matching quantises positions to half a pixel (0.37 px RMSE on synthetic shifts). A three-point parabola refinement helped but pulled positions toward whole pixels; a symmetric bicubic upsampling of the template and the search window brought the error down to 0.05 px.
  • COOL inside a locked-down sandbox. The AWS worker runs every job in a systemd-run sandbox with a private network. COOL's cv2 import contacts the EC2 instance metadata service, so it failed inside the sandbox until exactly that one address was allowed. The report documents this, together with its security trade-off.

Accomplishments that we're proud of

  • Two domains accepted under rules fixed in advance. Game character motion passed 48 of 48 regression cases and 30 of 30 held-out validation cases. SVG diagram editing passed every measured window, and 32 of 32 connector endpoints landed within 2 px of blind marks.
  • Measurement that changes what the agent does. In the logged V6 session, KiroClops showed Kiro the worst windows of its first candidate and a timing offset of about 10 ms. Kiro then wrote its own scripts to track the panel in the exported frames and fit its input latency, rewrote its input clock and re-tuned the spring. Calibration error fell from 0.0244 to 0.0067, 3.6 times lower, with zero lag.
  • A failure we are glad we caught. The same sealed candidate passed 55 of 56 held-out comparisons. The one it failed was a drag-and-release case the calibration set never contained. KiroClops recorded the run as failed, and so does this submission.
  • OpenCV 5 used where it matters. ALIKED and LightGlue locate 98.9% of SVG object-frames within a median 0.35–0.91 px of the template tracker. A LightGlue crash in OpenCV 5.0.0 is documented and contained, and sub-pixel refinement cut tracking error from 0.37 px to 0.05 px.
  • COOL with OpenCV 5 on Graviton4, verified on every job. Each receipt names the engine (OpenCV 5.1.0-dev and the COOL libraries), and the worker matches the local engine on all 17 test recordings. The honest speed result: no measurable speed-up on this workload, at about 0.02 cents per analysis.
  • Everything is replayable. Source trees are frozen, about 3,750 tests pass on a clean clone, and every number in the report traces to an evidence file, with a public subset in the repository.

What we learned

  • An agent optimises what it can see. Kiro brought the calibration error down 3.6 times, and the held-out set still found a behaviour it had never been shown. Sealing a candidate before revealing the held-out cases is what made that visible; without it, the run would have looked like a success.
  • Give the agent evidence it can compute on. Kiro started from the per-window errors, limits and lag estimate, then wrote its own scripts to track the panel in the exported frames. It never opened the image-strip tool we built for it. For a coding agent, raw frames and compact numbers were more useful than pictures made for a human.
  • A measurement needs its own error bars. Before trusting any verdict, I measured the tracker against independent marks and synthetic shifts. That is how a sub-pixel method that looked fine on average turned out to pull positions toward whole pixels.
  • Classical OpenCV for the verdict, learned features for the cross-check. Template matching and optical flow give exact, replayable numbers. ALIKED and LightGlue add robustness to scale and rotation, but on tiny, low-texture UI glyphs they find too few matches, so they check results instead of deciding them.
  • Profile before you accelerate. COOL returned the same numbers as the reference engine, but this pipeline is dominated by Python control flow, PNG decoding and small templates, so the optimised kernels did not shorten it. What COOL gave us was a verified OpenCV 5 build on Graviton4, not speed.
  • Write the rules before the results. Thresholds, recipes and held-out sets were fixed before each run, and a failed run keeps its identifier. That made the failures as believable as the successes.

What's next for KiroClops

  • Close the gap the held-out set found. Add drag-and-release cases to the panel calibration recipes, register them before the run, and let Kiro try again under a new run identifier. The failed V6 run stays on record.
  • Visual feedback, tested. The image-strip tool now lines up exactly with the scored windows. The next run will measure whether showing the agent aligned reference, candidate and difference strips changes its edits.
  • Motion tests in CI. Package the measurement as a step that runs on every pull request, so an animation regression fails a build the same way a unit test does.
  • More of the screen. Mobile app emulators, canvas and WebGL scenes and CSS transitions, each with its own pre-registered suite.
  • Faster where COOL can help. Profile the pipeline kernel by kernel, batch the work OpenCV can do in larger calls, and measure again on the COOL worker against the standard wheel.
  • Real users. A study with game and front-end developers, to test whether measured feedback saves them time compared with tuning by eye.
  • Hardening. Re-certify the tool-call audit for current Kiro CLI releases, give the worker sandbox only the credentials it needs, and replace the hand-written IAM scope with a least-privilege deployer policy.

Built With

Share this project:

Updates

Submission history