AI Robot Quality Inspector: Self-Healing Work Cell

A simulated robot cell that inspects every nut from above and below with a fine-tuned NVIDIA Cosmos model, explains what's wrong in plain words, and stops bad parts before assembly.

Comment

Inspiration

A typical robot cell has one sensor that matters: "is a part in my gripper?" A cracked, rusted or wrong-size nut is still a part, so it ships, and damage on the underside is invisible even to a camera that looks down. Adding real checks means writing a vision program per part and per failure, which small, high-mix shops can't afford. I wanted one general inspector that sees what the gripper can't, says why, and runs beside the robot on a single small GPU.

What it does

A UR10 arm in NVIDIA Isaac Sim takes each M16 nut from a feed tray, photographs it on an inspection nest from the top, then lifts it over an upward-looking camera for an underside photo. My fine-tuned NVIDIA Cosmos-Reason1-7B answers for each photo: good, wrong part or damaged, with a one-line reason ("A thin dark crack runs across the nut's bottom face"). Bad nuts go to the bin, good ones to the rack, and NVIDIA's pretrained Isaac Lab NutThread policy threads good nuts onto an M16 bolt. An operator screen shows the live tally against today's factory, both photos, the model's reasons, and an "Inject a fault" control that swaps the next nut for a chosen defect mid-run.

On 40 trays the model never saw (240 nuts, 120 bad), today's gripper-only cell ships all 120 bad nuts; mine ships 24 (61 with the top camera alone), and the arm placed 240/240 nuts where it meant to. NVIDIA's threading policy threaded 47 of 48 nuts.

This is a simulation project with no physical hardware component; the video shows the key modules operating in Isaac Sim and Isaac Lab.

How I built it

  • NVIDIA Isaac Sim 5.1: a work cell with NVIDIA's own M16 nut and bolt meshes; every scene and tray is generated from a seed with its answer key, so ~6,000 training photos needed no hand-labelling. Chips, cracks and rust are placed on the top or the underside; M12, M20 and an M16 bolt are the wrong parts.
  • NVIDIA Cosmos-Reason1-7B + LoRA (51.5M trainable parameters), fine-tuned on a Nebius AI Cloud H100: first on 1,800 single scenes (21.7% → 94.2% on 120 held-out scenes, ahead of every per-token vision model on Token Factory), then retrained on the robot cell's own top and underside photos. It runs 4-bit on one 15 GB T4 next to Isaac Sim, served over local HTTP.
  • NVIDIA Nemotron on Nebius Token Factory: the recovery planner (tool calling over five whitelisted actions with code guardrails), and the comparison across Token Factory's vision models.
  • NVIDIA Isaac Lab 2.3.2: the pretrained Factory NutThread policy, evaluated with the task's own physics success check.
  • A Streamlit operator screen, a filming mode that overlays the photos, verdicts and tally on a wide view of the cell, and benchmarks that score every decision against the answer key.

Challenges I ran into

  • The model fine-tuned on staged scenes rejected good nuts in the robot cell: new arm, shadows, an underside view it had never seen. Retraining on 2,400 photos from the robot cell itself fixed it. Of two variants, freezing the vision encoder (a common recommendation) made it worse, so I kept the full LoRA.
  • The arm: RMPflow steered the wrist flange instead of the suction tip (161.7 mm off), long moves stalled in a local minimum, and the underside camera saw nothing until I changed USD's default 1 m near-clip.
  • Isaac Sim crashed after 5 live trays because reused scene paths left stale mesh buffers in the renderer; every part now gets its own path.
  • Small chips: a plain ResNet-18 trained on the same photos is about as accurate as my model. Before paying for another fine-tune, I tested more chip photos and a zoomed view on the CNN for free. At an equal false-alarm rate neither helped, so I didn't spend the money.

Accomplishments that I'm proud of

  • A closed loop that actually moves: 240 of 240 nuts physically handled on unseen trays, with the decisions made live by a 7B NVIDIA model on one T4.
  • The underside camera halves the bad nuts shipped and catches every underside crack and every wrong size.
  • Honest reporting: the CNN baseline, the planner that ties a lookup table, the chips I still miss, and a real-photo test where the sim-trained model fails. All of it is in the README.

What I learned

  • Simulation with free labels makes measurement cheap, which made it easy to find out what doesn't work.
  • A VLM isn't automatically a better classifier than a CNN for one fixed question. Its value here is the reason it gives, and one model covering every check in the cell.
  • Test the cheap version of an idea before paying for the expensive one.

What's next for AI Robot Quality Inspector: Self-Healing Work Cell

Real M16 photos in the training mix and a real camera; a raking light for chips; the inspector deciding which nuts reach the threading policy within one continuous scene; and Nemotron choosing recoveries the arm executes.

Feedback on the tools

  • NVIDIA Cosmos-Reason1-7B: out of the box it scored 21.7% on my task and couldn't tell an M16 nut from an M12; after LoRA fine-tuning it reached 94.2%, and it still worked in 4-bit on a T4 (93.3%). An official SFT prompt/answer format and a 4-bit/8-bit recipe for older GPUs (T4, no bf16) would help. 4-bit broke one of my two adapters (vision frozen) but not the other; a note on adapter quantization sensitivity would have saved a day.
  • NVIDIA Isaac Sim 5.1: rep.orchestrator.step pauses the timeline by default (my arm froze until I found pause_timeline=False); the USD camera's default 1 m near-clip silently hid close objects; reusing a prim path for a mesh with a different point count crashed the renderer instead of raising an error; and a crashed standalone script leaves the app open and idle.
  • NVIDIA Isaac Lab 2.3.2: the pretrained Factory NutThread checkpoint worked first time (47/48). An official evaluation script that reports success counts would help.
  • Nebius AI Cloud: H100 VMs ready in minutes, ~4 min setup including the model download. A stopped VM still bills for disk, which isn't obvious; a warning and a ready "CUDA + PyTorch" image would help.
  • Nebius Token Factory: one OpenAI-compatible API made model comparisons a config change. Fine-tuning and LoRA hosting are text-only, so a vision model like Cosmos had to go to AI Cloud; and the hackathon credit couldn't be used there.

Links

Built With

  • bitsandbytes
  • cosmos-reason1
  • lora
  • nebius-ai-cloud
  • nebius-token-factory
  • nemotron
  • nvidia-isaac-lab
  • nvidia-isaac-sim
  • peft
  • python
  • pytorch
  • streamlit
  • transformers
  • usd
Share this project:

Updates

Submission history