Robot-As-A-Verifier

Inspiration

Some of the fastest recent progress in AI-assisted research has come from automatic verifiers. In math, proof assistants like Lean check every step of a proof mechanically, so a model can propose thousands of attempts and keep the ones that check out. In combinatorics and complexity research, systems now propose new constructions and bounds in short programs, then score each one instantly. The pattern is the same each time: the generator proposes, the verifier can’t be argued with, and the loop runs fast enough to iterate. It’s a good verifier that turns a model’s guesses into progress.

Physical hardware has nothing like that. A prototype gets handed to a few testers, someone writes “button 2 feels sticky”, and a week later the designer guesses at a fix, CADs it, reprints it, and no one can tell whether the new revision is actually better without testing it all again from hand. So hardware can’t iterate like a proof search can, and an AI that designs physical things has no ground truth to learn from, only its own opinion of its own work.

We wanted to build the missing verifier: a robot that tests the real object with force and vision, writes evidence that you can trace, and scores every revision on the same tests so a fix only counts if the score goes up.

Medical hardware is where this matters most. Infusion pumps, insulin pens, glucose meters, hospital beds, lab instruments like centrifuges – all are used by tired clinicians, older patients, and people with arthritis or limited grip. A button that takes too much force, a latch that needs a pinch, or a step in the manual that gets wrong isn’t just an annoyance there. It’s a safety risk. Medical devices already have to go through usability testing before they reach patients, and today that’s slow, manual, and hard to repeat. A robot verifier can run the same procedure on every revision, flag a control that’s too stiff for weaker hands, and check a fix the day it’s printed.

That’s why our test subjects are lab appliances, a centrifuge and a microwave. That’s why rules like “usable by no thumbs” are tests that the robot runs, not guidelines. One rule held the whole project together: whatever thinks up a plan or a fix has no say in how it’s scored.

What it does

Robot-As-A-Verifier (RAAV) is a verifier for hardware R&D. You give it a product, its instruction manual, and if you have one, its CAD model. A low-cost robot arm and a $19 wrist camera then

Find the controls on their own. No CAD, no fiducial tags, nothing. The arm scans the panel with the wrist camera, finds buttons, switches, knobs, lids, wells, measures where they are, and which way they face.
Build a test plan from the manual. An LLM reads the manual, turns it into steps like “power on, set speed, close lid, press START”, and a person approves the plan before the arm moves.
Run the tests. The arm presses, flips, turns, hook-pulls, and places on every step. At the end, the force trace, the appliance’s own state report over Wi-Fi, and any photos that the arm took during the run all go into a report. If the arm misses a part, it can re-calibrate itself and retry.
Write bug reports that you can trace. Each finding says what happened, what the manual says should happen, the force trace with frames as evidence, and a link back to the exact step. When the robot itself has missed, that’s reported apart from a real device fault. When a failure doesn’t repeat, it’s reported as intermittent.
Fix the CAD. A CAD analyst agent reads the findings, works out the root cause from the actual geometry, edits the model in Onshape on its own branch, or in CadQuery offline, and the patch must pass automatic geometry checks: only the faulty control, a reprintable part, changes; nothing collides; the other parts still move freely; the robot can still reach and hook each control. A person approves the patch.
Retest. The patched part goes back through the same tests. The fix only counts if the reward (control coverage × pass rate, recomputable from the logs) goes up.

You can watch and steer all that from a web Console: a live 3D view of the arm, the camera feeds, the test flowchart, approvals, pause/skip/prioritize, and a one-click replay of any past run.

Our test subjects are two mock lab appliances that we built for the event: an MC-6 centrifuge and a MW-1 microwave, each with real buttons, a switch, a knob, a lid or door, a display, and an Arduino Uno R4 WiFi reporting its state.

How we built it

We split the project into two systems that talk through a deliberately narrow channel.

System 1 is the robot. It has perception, inverse kinematics, motion skills, safety limits, and verdict rules. It never calls an LLM.

Perception. OWLv2 finds candidate parts, SAM 2.1 outlines them, a plane sweep over nearby wrist-camera photos gives heights, shape rules classify each part, and Claude Haiku names the ones with faint labels from the manual and their place on the panel.
Skills. press, flip, turn, hook_pull, pick, place, all done with the SO-101’s stock gripper, and each guarded by a 15 N hard force cap and a torque gate. Force comes from the servos’ own load estimate, no added sensor.
Simulation. A MuJoCo model of the SO-101, rebuilt from our bench measurements of field of view, lens fit, servo dead zone, and reading jitter, with 12 hand-authored appliances and seeded random ones.

System 2 is the harness. This is the only place where an LLM runs: planning from the manual, steering, reports, memory, reward, the CAD analyst, and the Console. It reaches System 1 through seven MCP tools. It can start named routines from a closed list. It has no tool that moves a joint, and System 1 clamps every parameter. A model turn takes seconds while the arm’s control loop runs at 50 Hz, so a prompt should never be able to drive a motor.

Every model call in the harness goes through Snowflake Cortex, and the model is a config value (details below). Opus 5.5 runs the CAD analyst. The CAD analyst works through its own MCP tools (get_part, measure_gap, write_part, check, submit_fix) against Onshape branches or a sandboxed CadQuery workspace.

The appliances network to a Vultr server through a hotspot. Each appliance’s Arduino posts its sensor state over HTTPS to the server, and the server also carries System 2’s tool calls to the robot (details below). If a board goes quiet for 2 seconds, the next step stops instead of running blind.

The Console is a Vite + React app with a FastAPI/stdlib bridge. The browser draws the arm from streamed body poses rather than server-rendered frames, so it stays smooth, and every run replays in the same 3D view from its events.jsonl.

How we use Snowflake Cortex

Why it’s integral. System 2 is the only part of RAAV that thinks, and every one of its model calls goes through Snowflake Cortex: turning the manual into a test plan, deciding what to do when a step fails, handling a person’s steering messages, and writing the bug reports. We could only make claims about the models themselves because we had one endpoint and one credential for all of them.

Four frontier models, one fair comparison. Cortex serves Claude Haiku 5.5, Claude Sonnet 5.5, GPT-5.4, and Grok 4.7 behind the same account. Every model gets the same prompts, the same tools, the same test devices, and a rule-based baseline runs alongside. Switching models is one config value, never a code change, so nothing gets tuned to one model.
A cost ledger for every call. Each request is appended to a ledger with its input, output, cached tokens, and latency, priced in Snowflake AI credits. rav spend totals it per model, and rav eval --budget-usd won’t start a run that could cross the cap. Our first full model round, with its connection tests, cost 0.5 credits.
Prompt caching. The system prompt and the tool definitions are cached on Cortex’s Messages API, so the long, repeated parts of each turn are cheap.
What it showed. In round 1 (seven scenarios, from planning with a dependent step to telling a robot’s miss from a real fault), all four models scored 1.00, and the rules baseline was 0.49. That showed that tool calling and image reading worked for every model through our harness. It also showed that we needed harder rounds: random planted faults, a 5% chance of an injected error on each action, to separate them.

Model	Tokens	Tool errors	Median turn
Claude Haiku 5.5	388,590	1	3.4 s
Claude Sonnet 5.5	243,671	3	2.5 s
GPT-5.4	157,810	4	2.8 s
Grok 4.7	221,060	3	2.9 s

Challenges.

Before Cortex, we couldn’t compare anything. Our first model test used free API keys from different providers, hit a quota limit, an overload error, and an 85-second turn. Moving everything to Cortex gave us one route with no free-tier cap.
Two wire formats. Cortex serves Claude through the Messages API and the other models through Chat Completions. We wrote one small provider-neutral client over both, so the harness never knows which it’s talking to.
Claude’s thinking blocks are signed. The API checks that signature when a thinking block comes back in the next turn, so the client stores each response exactly as it arrived and resends it unchanged rather than rebuilding it.
Small protocol surprises. Cortex refuses requests without an Accept header. The newest models refuse a fixed thinking budget and need an effort level instead. Each provider names its reasoning field differently, and many send none.
Failing safely when the API does. The client waits out 429 and 503 responses (honouring Retry-After), then falls back to another model. If the budget or the API runs out mid-run, the robot holds the arm safely or falls back to the rule plan, and the run log says so.

How we use Vultr

Why it’s integral. Our rule is that sensors decide, and the most important sensor is the appliance itself: did START actually start the centrifuge, did the lid really close. That signal has to get from two Arduinos to the robot reliably and securely, at a venue where everything sits behind a phone hotspot. A hotspot’s carrier network accepts no incoming connections, and the laptop’s address changes. A Vultr server gives us the one fixed, public, TLS-secured address that everything can reach.

Appliance readouts. Each Arduino Uno R4 WiFi posts a JSON snapshot of its sensors over HTTPS to Vultr whenever something changes, and at least every 0.5 seconds. Caddy on Vultr terminates TLS and passes each snapshot on to System 1.
A reverse tunnel to the robot. Since nothing can connect in to the hotspot, System 1 opens one outbound WebSocket to Vultr. A relay carries both the appliance snapshots and System 2’s MCP tool calls down that tunnel. It forwards raw TCP streams without reading the MCP messages.
The harness runs in the cloud. System 2 runs on the Vultr server and reaches the robot’s seven tools through the relay’s loopback port. The arm stays on the table, and the planner never needs to be on the same network.
A hosted Console. Vultr also serves a read-only build of the Console, so anyone can replay our recorded runs in 3D from a browser.

This security is built into what we designed. Three separate credentials: one for the tunnel, one for MCP and tools to start the robot’s routines, and one key per appliance that can only publish that appliance’s readings. Caddy forwards only POSTs to the public readout route, and a public request to /mcp or a GET for a reading gets a 404. A board’s key gets a 401 on any other board’s route. A board gets a success response (204) only after System 1 has accepted the snapshot, and if the tunnel is down, the board sees an error instead of a false acknowledgement.

Challenges we ran into

Nearly every big design change came from a measurement that disagreed with our plan.

TLS on an Arduino

The Uno R4 WiFi checks certificates against the root CA bundle in its Wi-Fi chip’s firmware. For proper DNS hostnames, that meant a matching certificate. Caddy manages it with Let’s Encrypt, with short-lived IP certificates as a fallback.

Wi-Fi with blocked buttons

The board’s Wi-Fi joins and HTTPS calls are synchronous, so a slow retry that froze button and sensor polling. We bounded timeouts and spaced out retries after an error.

Presses too short to see

A quick button press could start and end between two snapshots, so the robot never saw it. The firmware now queues up to eight debounced changes while a POST is in flight, and System 1 keeps the last 32 changes per device. A queued change is dropped after 2 seconds, so an old press can’t be replayed after an outage.

Stale data must never pass for fresh

Every reading carries its age. A reading older than 2 seconds is marked stale, and the real arm refuses to judge a step on it.

Reconnects in the middle of a call

A dropped tunnel can interrupt an MCP call partway through, so after a reconnect, the harness checks the robot’s status before repeating a command, rather than pressing twice.

Debugging blind

With the boards on a hotspot, the VPS and the laptop in a chain, we gave every hop a clear signal. The board logs an HTTP status (204 accepted, 401 bad key, 502 Vultr can’t reach System 1), and rav readout --watch on the Vultr shows each device as fresh, stale or silent.

Accomplishments that we’re proud of

The full loop works end to end in simulation

On the MC-6 centrifuge, the robot found that a tube dropped into well A stopped 8.15 mm above its seat, so START refused with TUBE. The CAD analyst (Opus 5.5, 14 tool calls, 102 seconds, $0.43) traced it to a bore 0.4 mm narrower than the tube body, widened it and added a 45° lead-in. On retest, the tube seated within 0.97 mm, and the centrifuge ran. The analyst never saw the designer’s own fix, and landed within 0.2 mm of it.

On an interval timer with three planted faults, a Claude Haiku run found two of them in its tests, the CAD loop patched them, and the retest’s reward rose from 0.42 to 0.64.

Perception that generalizes

On 16 devices held back from all tuning, the sim passed 110 of 115 attempted tests (96%), with no false failures.

The real arm runs on our real appliance

The SO-101 scans the MC-6 panel with its wrist camera, finds and names the controls, flips POWER, pinch-turns SPEED and presses START, while the Arduino reports what actually changed.

A safety boundary that we trust

The LLM can plan, steer and explain, but it can’t move a joint, can’t override a verdict and can’t approve its own CAD patch.

Honest numbers

Every claim above comes from a run folder, and our docs say plainly which results are from the sim and which are from the real arm.

What we learned

Check the sensor model against how the sensor works

Our most expensive mistake was trusting a simulated depth camera that a real one could never match.

Cheap hardware can work if you’re careful about noise

A $19 camera and a plane sweep got outlines within 0.41 mm and heights within 1 mm (93–95%) of the nominal.

Keep the model out of the loop that moves things

Splitting thinking (System 2) from moving and judging (System 1) made the system safer, and also made failures easier to diagnose. Every failure gets filed as Model, Pipeline, Robot or Device.

Easy benchmarks don’t separate models

In our first round, all four frontier models scored 1.00, and the rules baseline was 0.49. Telling them apart needs random faults, injected errors and held-out devices.

Have someone other than the builder review the work

Every fix was rechecked on seeds its author never used. That caught friction modelled at half strength, a safety cap tripping on harmless moves, and one fix that had to be reverted.

What’s next for Robot-As-A-Verifier

Close the loop on real plastic

Print the analyst’s patched part, put it back in the appliance, and show the reward rise on the physical arm, not only in the sim.

More skills and more products

Finish door swings and placing into wells on the stock gripper, and test products we didn’t design, including a remote and receiver.

Read the product, not the wires

Today, verdicts read the appliance’s own Wi-Fi readout. Reading displays, LEDs and motion from the camera would let RAAV test products nobody instrumented.

A shared vocabulary for devices

Describe every product as parts, relations, procedures and test properties, so the planner composes tests from general pieces instead of learning one appliance at a time.

A reward for AI-designed hardware

The long-term goal is to make RAAV the verifier that scores generated designs, so a design model can improve against physical tests instead of its own judgment.

Accessibility as a test property

Rules like “usable with no thumbs” or “opens under 5 N” become tests the robot runs on every revision, so products work for weaker hands from the first print.