Effector Lab

Describe a multi-step robot job in plain words. Nemotron plans it, physics simulation tests every tool, and you get a recommended setup with replays and evidence.

Comment

Inspiration

Every robot job is a chain of small steps: open a lid, pick a part, press a button, tighten a clamp. One gripper rarely suits every step, so teams guess at the tooling, buy the parts and find out on the robot whether the guess was right.

We wanted to move that guess into physics, before anyone buys a part.

What it does

Effector Lab picks the tools for a robot job.

  1. Describe the job in plain words. NVIDIA Nemotron, served by Nebius Token Factory, turns it into a plan: the objects, the steps and what counts as done for each step. Code checks every rule, and Nemotron repairs what fails.
  2. Physics tests every tool. MuJoCo tries every gripper, hand, suction cup, vacuum frame and process tool on every step. The best candidates then run the whole job 8 times, with the parts moved and the friction changed.
  3. Get one clear answer. The simplest setup that finishes the job: one tool, two tools on one head, or a tool change. You also see why the others lost, a replay of the physics and a build pack with the parts to print, the tested scene and a bill of materials.

Four real jobs, tested end to end:

  • Returns tote: open the lid, pack two items, close it. One two-finger gripper, 8 of 8 runs.
  • Solar panel: lift an 8 kg panel onto its rails and tighten its clamp. A tool change: vacuum frame, small suction cup, nut runner. 7 of 8 runs.
  • Floor tile: lay a tile on adhesive and tamp it level. A vacuum frame and a tamper on one head, 8 of 8 runs.
  • Hotel delivery: press the call button, open the door, place the parcel. A tool change: hook, gripper, suction cup. 8 of 8 runs, while the best single tool finishes none.

A use-case atlas maps 95 real jobs to the setups an experienced integrator would expect. Everything is simulated; verify on hardware.

How we built it

  • Planning: NVIDIA Nemotron 3 Super on Nebius Token Factory, with the plan's schema sent as one forced tool call and a repair call for anything the checks reject. Every answer is cached, so every plan replays for free.
  • Physics: MuJoCo with a tilting six-servo wrist, 12 object models (hinged lids and doors, drawers, threads, panels on rails, tiles on adhesive, spring buttons) and 7 basic moves. Every tool gets the same wrist, conditions and random seeds.
  • Rules an integrator would use: one tool by default, a second tool only when it finishes clearly more runs, and process tools (a nut runner, a tamper) for process steps.
  • Compute: heavy batches run on Nebius Serverless Jobs from one pinned image; our pre-registered experiment ran there.
  • Trust: every replay is checked against the physics before it reaches the site.
  • Website: Next.js and React Three Fiber, replaying the recorded physics in the browser. Open source under Apache-2.0.

Challenges we ran into

  • Plans that read well but cannot run. A door placed beyond the arm's reach, a lift too short for the load. Each one became a check with a plain message, and Nemotron fixes it in a repair call.
  • Keeping the physics honest. Parts had to be set down rather than dropped, tolerances had to be stated for every measurement, and a deep overlap between any two bodies marks a run as unreliable.
  • Grips that only work in simulation. A gripper once opened a door by pinching the top of its handle from above. Door handles are now gripped from the front, the way a person would, and the door is placed where that grip can reach.

Accomplishments that we're proud of

  • From one sentence to a tested tool setup, with replays and a build pack.
  • Four real jobs from warehouses, construction and hotels, each with a clear winner and the reasons the others lost.
  • A pre-registered experiment: Nemotron with physics feedback designed good gripper fingers faster than random search (p = 0.0011).
  • An open library of 29 end effectors, each licence-checked and credited.
  • Honest failures, shown on the site with their reasons.

What we learned

  • The test bench shapes the answer, so check what the simulator cannot do before trusting a ranking.
  • Most jobs need one tool: 69 of the 95 in our atlas. A second tool has to earn its place.
  • Planning is the hard part. Nemotron plans well when code checks every rule and names each problem.
  • Say what is approximate. Every approximate result says "verify on hardware".

What's next for Effector Lab

  • Plan and run it on the site: visitors bring their own NVIDIA or Token Factory key, and the physics runs on Nebius.
  • Hardware: build the winning setups with a partner and test them on a real arm.
  • More jobs and more tools in the library.

Built With

  • coacd
  • docker
  • fastapi
  • mujoco
  • mujoco-warp
  • nebius-ai-cloud
  • nebius-container-registry
  • nebius-mysterybox
  • nebius-object-storage
  • nebius-serverless-jobs
  • nebius-token-factory
  • nemotron-3-super
  • nemotron-3.5-lightning
  • next.js
  • numpy
  • nvidia-nemotron
  • nvidia-warp
  • playwright
  • python
  • react
  • react-three-fiber
  • tailwindcss
  • three.js
  • trimesh
  • typescript
Share this project:

Updates

Submission history