Inspiration

Every building has a list of jobs nobody has time for: take the bins out, wipe the tables, change the hallway bulb. Each one is a nothingburger. Together they eat hours of a landlord's week, and for an older or disabled tenant, carrying a bin down the hall can be the hard part of the day. Routine upkeep is the first thing to slip when a building is short of staff, and the tenants are the ones who pay for that.

The robots that could help are designed first, and the building has to cope: one humanoid, one shape, one price, whatever the job. Then we measured real buildings. Across 24 Matterport scans, doorways ran from 0.54 to 0.86 m and ceilings from 2.13 to 3.57 m. A robot that fits one unit does not fit the next.

One of us has deployed robots in the field. The robot was rarely the hard part; the data was. Nobody had measured the building or the job. So we flipped the order: start from the building, let the person who already does the job teach it once, and size the robot to fit.

What it does

Describe a job in plain words, demonstrate it once in a Meta Quest headset and force-sensing gloves, and agents design the cheapest robot that can do it in that building, plan its route and arm motions, and show it working inside the scan.

Nobody builds robots this way. Every robot maker ships one body because one body is repeatable. Ours is repeatable too (same modular parts), but specialized per job, so you pay only for the reach, payload and base the job needs.

How we built it

1. Agent Intake

  • You type the job: "Carry the recycling bin outside." A LangGraph agent reads it. SubQ Snap makes the fast yes/no and which-skill calls, and Claude Sonnet takes anything Snap can't settle and can name a new skill.
  • It turns one sentence into ordered tasks, requests for demonstrations and open questions.
  • If no one has taught it that skill yet, the run pauses and asks for a demonstration. Once taught, the skill is reused in other buildings.
  • If the building can't settle something, it asks. "Carry a container outside" got back: which one, the 4.5 kg recycling bin by the entrance or the 0.55 kg sample bottle on the atrium table? The answer is added to the job and the run resumes.

2. The headset and the gloves

  • Glove: five force-sensitive resistors, one per fingertip, hand-soldered to an ESP32, streaming over Bluetooth at about 100 Hz.
  • Meta Quest: tracks the head, wrists and every finger joint 30 times a second, in the frame of the loaded building scan. The glove readings are lined up with it by time.
  • Why both: a scan tells a robot where a bin is. It can't tell it how hard to hold one. Grip too softly and it slips; too hard and you crush it.
  • What comes out: demo.json (the hand's path relative to the object: approach, work, retreat, and the kind of motion: wipe, carry, twist, hinge) and grip.json (a grip force band, from the lightest hold to the person's peak).
  • While you record: the studio shows the five fingertip lines live, says what it read from each take, and lets you redo one you fumbled.
  • No demonstration needed: vacuuming a room or mowing a lawn is only driving, so those are planned straight from the scan.

3. The agents that design the robot and plan its path

  • Code enumerates 7,140 combinations of base, lift, arm and gripper, then builds them as MuJoCo models cheapest first and keeps the first that passes every check: doorway, turn, door header, steps, reach, tip-over, payload, gripper opening, force and wrist roll.
  • Plans the motion. It finds a parking spot for each task, plans the base route with A*, re-anchors the demonstrated hand path onto the real fixture in the building, and solves it for the chosen arm with our own inverse kinematics (damped least squares on MuJoCo's Jacobian).
  • Fixes its own mistakes. If the arm can't reach from where the robot has to park, it first tries a spare parking spot; if that fails, that arm and lift are ruled out and the design runs again, up to five times.
  • Claude reads the job, asks questions and writes the robot's task program (57 steps for the JRR Hall job), which code then checks. Every pass or fail is plain code, and the model can't overrule one.
  • Every step is traced in LangSmith, and a live page in the studio shows each tool call and every robot that was rejected and why. The JRR Hall job made 353 tool calls in about three minutes.

3b. Matterport and Vexcel

  • Matterport Model API: we probed 24 real models for rooms, ceiling heights and doorway widths. That is where the 0.54 to 0.86 m doorway spread comes from.
  • Matterport, read straight in: a house in La Mirada, Los Angeles, is built from its Model API floor plan and MatterPak mesh: 14 rooms, 22 doorways and the table tops measured off the mesh, with no tracing by hand.
  • Matterport Showcase SDK: the studio opens inside the tour of the building you're working on, and the robot plays its plan in the tour.
  • Vexcel, outside: aerial photos (about 6 cm a pixel) of lots in Houston and Los Angeles, traced into worlds with lawns, driveways and kerbs. Mowing a lawn or taking trash to the kerb is planned outdoors on the real photo.

4. Seeing it

  • Studio: one prompt bar for every space, indoor or outdoor.
  • Robot Design page: watch the chosen robot assemble part by part, then see its route on the floor plan.
  • In 3D: the robot drives its route and does the job, in the browser or with a VR mode for the headset, and each run writes a walkthrough video.

Challenges we ran into

  • A hand is not a gripper. Different joints, different shape, and the person stood wherever they liked. What made the transfer work:

  • We kept only what's about the object. How the hand approached, moved and let go carries over; where the person happened to stand does not. The path is re-anchored onto the fixture in the target building.

  • Grip as a range, from the lightest hold that keeps the object to the hardest squeeze the person used.

  • Real building data is uneven. 10 of the 24 scans had no doorway widths, and none had detected objects, so bins, bulbs and tables had to be placed by hand and labelled that way.

Accomplishments that we're proud of

JRR Hall is traced from its Matterport tour; the rest are hand-written test scenarios, designed from the building and the job.

  • A real search. JRR Hall: 7,140 combinations, 860 pass arithmetic, 84 composed and checked in MuJoCo, cheapest kept. 15 route legs, 13 arm motions, no failures.
  • Hardware we built. Ten fingertip force sensors, hand-soldered, over Bluetooth, time-aligned with Quest hand tracking.
  • It says what it doesn't know. Every number carries its source.

What we learned

  • Model for judgement, code for arithmetic. Reliability came from that split.
  • Long tasks need a ledger. A file with ids and sources beat any prompt.
  • Real building data is uneven: 10 of 24 scans had no doorway widths, none had detected objects.
  • A demonstration is mostly about the hand positioning, not where the person stood.
  • Cheap force sensors give a safe range.

What's next for Nothingburger

  • Contact simulation in MuJoCo; today's playback is positions only.
  • Any Matterport model straight to a plannable world, with fixtures found in the panoramas.
  • A manufacturer's package: CAD, parts list, assembly order.
  • Cost comparison against a person doing the job.

Built With

  • anthropic
  • grok
  • langgraph
  • mujoco
  • quest3s
  • subq
Share this project:

Updates

Submission history