Knack: robots that learn by practising

A robot arm learns a new task from one sentence by practising in simulation, sees only through OpenCV 5 on AWS, and shows its work: 0/8 to 50/50 for under one cent.

Comment

TL;DR

Type one sentence. A coding model writes a Python program for a robot arm, eight simulated arms practise it on random layouts, every look goes through an OpenCV 5 RGB-D perception pipeline, failures are analysed with their camera records, the program is rewritten, and the result is measured on 50 unseen layouts with a 95% confidence interval. Zero demonstrations.

On the live AWS Graviton deployment, a lift skill went 0/8, then 8/8, then 50/50 held-out, for $0.008 of model calls.

Live demo: http://13.212.24.5/ (browse every run; passcode for new runs is in the testing instructions) Health check: http://13.212.24.5/api/health returns opencv 5.0.0, aarch64

Problem and users

Teaching a robot arm a new tabletop task means days of hand-written code or 50 to 200 teleoperated demonstrations. Robotics students, small labs, classrooms and makers with a low-cost arm have neither. They also have no way to know how reliable a skill is before trusting it. Knack gives them a working skill plus a measured success rate and a cost receipt.

How OpenCV 5 is used (the robot never reads simulator state)

  1. Overhead and front RGB-D cameras with a sensor model: depth noise growing with distance squared, 1 mm quantisation, RGB noise, and a per-episode extrinsic calibration error the pipeline does not know about.
  2. cvtColor to HSV, inRange colour masks, morphologyEx opening.
  3. connectedComponentsWithStats with size gating (so a blue tray wall is never mistaken for a blue cube).
  4. Back-projection of every mask pixel to 3D, removal of fixture points, top-face selection by height.
  5. minAreaRect in the table plane for centre and yaw. Fixtures are mapped once per episode.
  6. An object neither camera can see (usually under the gripper) is reported missing and get_pose() raises, so the program must decide what to do.

Agentic Vision: OpenCV output changes the next action

  • Inside an episode: the program observes, acts, re-observes and branches on what the cameras report (object missing, cube not at the commanded height, grasp lost).
  • Across practice rounds: each failed episode's trace includes its camera record, for example seed 3002: FAIL | red_cube_z=0.0612, required 0.065 | camera: 4/4 object sightings, median error 5.81 mm. A trace model turns these into failure modes with line numbers; the coder rewrites the program; the next practice round runs the new program. Real example (Super, lift): version 1 discarded the camera's height estimate and grasped at a fixed 15 mm; its look-again helper moved the gripper sideways and lost the pose. The final version grasps at the camera-measured centre, re-observes after every move to check it arrived, corrects low lifts, and re-looks in place. Practice went 0/8, 0/8, 4/8, 8/8, then 42/50 held-out.
  • Human control and safety: the learned skill runs inside a deterministic safety layer (workspace box, joint and speed limits, table-force limit) that AI-written code cannot catch or override; AI-written code runs in an AST-allowlisted sandbox, one process per episode, with a wall-clock kill; every run has a hard budget cap; the public console needs a passcode to start a run and caps it at $0.10.

Evaluation (all numbers are in the repo, results branch)

Perception accuracy vs ground truth, 30 random layouts per scene, 180 total, with the sensor model on: | scene | found on first look | median error | worst | found while the arm works | |---|---|---|---|---| | lift | 100% | 4.29 mm | 9.17 mm | 100% | | place | 100% | 4.25 mm | 8.86 mm | 100% | | push | 100% | 4.30 mm | 9.41 mm | 97% | | stack | 100% | 4.29 mm | 9.17 mm | 100% | | tray | 100% | 4.29 mm | 8.94 mm | 100% | | sort | 100% | 4.51 mm | 9.89 mm | 99% |

Skills learned with OpenCV perception, 50 unseen layouts each: | coder | lift | push | stack | sort | total model cost | |---|---|---|---|---|---| | Nemotron 3.5 Super | 42/50 | 48/50 | 50/50 | 49/50 | $0.038 | | Nemotron 3 Ultra | 50/50 | 28/50 | 50/50 | 49/50 | $0.135 | | Nemotron 3 Nano | 0/50 | 0/50 | 49/50 | 0/50 | $0.036 |

Failure cases we report: Ultra reaches only 56% on push. Nano never learns lift, push or sort; its sort program kept looking while the gripper covered the cubes (467 of 1,682 looks missed). A vision-language judge we tested as a reward was fooled by lifts a few millimetres short, so ground truth is kept wherever one exists.

Photoreal stress test (AI-generated images, not real captures): with the simulator's HSV ranges unchanged, the pipeline found 23 of 24 red and blue objects across six photorealistic images of a light-brown wooden table (wood hue close to red, plastic glare, hard shadow, distractors including an orange pencil and pink eraser, touching objects, a cube half inside a gripper), with no false detections. The miss is a navy cube deep in shadow, reported as missing rather than guessed. Image-space only (no depth, no calibration). See docs/photo_check/.

AWS deployment

One EC2 t4g.small (Graviton, arm64) in ap-southeast-1 on the Free plan runs the console and the practice episodes (OpenCV 5 and MuJoCo on Arm). Secrets live in SSM Parameter Store (SecureString), run summaries sync to a private S3 bucket every 10 minutes, administration is Session Manager only (no SSH port), IMDSv2 is enforced, the instance role is least-privilege, and AWS Budgets alerts fire at $1, $5 and $10. Reproduce with deploy/aws/user-data.sh and the pinned requirements-lock.txt (taken from the running instance).

Challenges

Same-coloured fixtures needed size gating and geometric rejection. A perfect render made perception look implausibly good (under 0.3 mm), so we added the sensor model. Practice scores on 8 layouts are noisy, so the best version is re-checked on 16 fresh layouts before the held-out evaluation, with revert-to-best on regression.

What we learned

Most first-try failures are not perception failures; they are programs that do not use what the camera tells them. The rewrites that work trust the measurement and check more often.

Limitations and what's next

This is simulation: the cameras are modelled, the photoreal test images are AI-generated, and no real camera or arm has been tested yet. Colour segmentation assumes a known palette. Next: the same OpenCV steps on a real low-cost arm (an overhead-camera path with a homography is in knack/real/), and OpenCV 5's DNN module for objects outside the palette.

Built by Suhas Suren (MSc Robotics, NTU Singapore), solo.

Built With

Share this project:

Updates

Submission history