TwinPulse

Digital twins are hard to build and hard to keep current. TwinPulse lets a real robot keep its Isaac Sim twin up to date: Nemotron on Nebius Token Factory plans the task, LiDAR adds what changed.

Comment

▶ Demo video (2:59, a real TurtleBot3 and its Isaac Sim twin): https://youtu.be/v8mOFSjLvd4

Try the loop in your browser: https://htmlpreview.github.io/?https://github.com/lee2017-bit/TwinPulse/blob/main/docs/demo/twinpulse-walkthrough.html (a simulation of the loop, not the product UI; the real robot run is in the video)

  • What: one chat sentence plans a route on an Isaac Sim twin, a real TurtleBot3 drives it, and its LiDAR adds what the twin is missing.
  • For: the few people who keep a factory twin current.
  • Real robot: filmed run (Oct 27): 3 stops (6.57 m) and back to the start; repeat runs (Oct 28): 4 mission attempts + 1 guard-stop retry, 3 completed, 16 of 20 stops reached.
  • Planner: 5 of 5 schema-valid plans in the repeat runs (median 5.1 s; 4.5 s in the filmed run); 100 scored benchmark calls (60 planner, 40 photo).
  • Twin vs tape: box found 4/4 (13 cm mean error), return drift 4.5–29 cm.
  • Real: the living room and the robot. Concept: the factory scene from 2:31, labelled as one. Failed runs are listed below.

Inspiration

I was a Factory Automation Engineer. In Siemens Plant Simulation I built digital-twin environments and developed, simulated and evaluated AGV (automated guided vehicle) operating logic. Two problems kept coming back.

Building a digital twin is hard and labor-intensive. The initial build is a project of its own: many engineers are assigned to model every machine, rack and aisle, and to code the operating logic.

Keeping it current is just as hard, with far fewer people. After the build, a small team maintains the twin while the site keeps changing: pallets left in aisles, objects moved, layouts evolving. There is no easy way to find what changed. Every change means editing 3D objects and SimTalk logic by hand and validating again, so the twin slowly drifts away from reality.

Commercial licenses add procurement and approval overhead on top. I wanted a more accessible stack, built on NVIDIA Isaac Sim, where the twin can be built faster and the robot that already drives the floor helps keep it true.

What it does

TwinPulse is a product and workflow layer on top of NVIDIA Isaac Sim. It closes one loop: build → operate → observe → update → re-simulate. In the twins I worked on, data flowed only from the twin to the robot. Here the robot's own LiDAR also feeds the twin as it drives.

Why not just refresh the robot's map? A SLAM map is an occupancy grid, cells that are free or blocked, built for one robot's navigation. The engineering twin is a different artifact: the USD scene in Isaac Sim on which routes are planned today and on which, later, plant dispatch logic is meant to be simulated (the founding goal; not in this demo). TwinPulse does not redraw a grid; it adds named, reviewable, undoable objects with sensor provenance into that scene and re-plans on it. And a photo step covers what the LiDAR cannot see (the soft bag: 0/4 by LiDAR, 5/5 by photo), which a map refresh does not do and which the room scan does not contain either.

  1. Build the twin. Import an iPhone RoomPlan USD scan into Isaac Sim. NVIDIA Nemotron VL looks at a room photo and proposes objects the scan missed; accepted ones are placed next to the furniture they belong to.
  2. Place the robot on the twin where it really stands (tape-measured), with its heading.
  3. Say what you want: "Survey the room and come back." NVIDIA Nemotron, served by Nebius Token Factory, turns it into a mission plan. TwinPulse draws the numbered route on the USD scene.
  4. Drive it for real. A physical TurtleBot3 runs the route while its live pose and LiDAR appear in Isaac Sim.
  5. Update the twin. LiDAR returns the twin cannot explain become "not in model" objects and are added to the USD scene, automatically (Auto-apply, with Undo) or after review.
  6. Re-plan on the updated twin. I pick one point past the new box, and TwinPulse plans the next route on the updated twin: around the box and back to the start.

Measured results

Every number below comes from the app's own run records and from tape measurements in the room. The failed runs are included.

  • Filmed run (Oct 27, real living room): Nemotron planned the mission in 4.5 s; the robot drove 3 stops (6.57 m) and returned to the start. LiDAR found 5 real objects the room scan did not have (including a cardboard box we dropped in the room) and added them to the twin automatically. The second route (6.21 m, through one point I picked) went around the box on the updated twin and added 2 more. No link drop in 37 minutes.
  • Repeat runs (Oct 28, build 61): the same command from the same taped start, with twin edits on Review (every LiDAR candidate skipped; one photo step before run 2 added two objects away from the route). 4 mission attempts plus 1 guard-stop retry: 3 completed; 1 reached all 4 stops and the start, then had its final heading turn refused by the fail-closed recording guard (the 74-minute session hit its 768 MiB recording budget); 1 was stopped by a robot-side guard at the third move, with no contact. 16 of 20 stops reached.
  • Planner: Nemotron 3 Super returned a schema-valid plan in 5 of 5 calls (4.5–8.9 s, median 5.1 s). The step order varied (2 of 5 plans began with observe), and the deterministic route on the twin was identical every time: 4 stops, 6.22 m.
  • Planner sizes (Oct 28, 60 live calls): the app's planning instruction, schema and settings, 3 prompts x 5 trials per size, every answer checked by the app's own validator. Nano 15/15 valid (median 5.1 s), Super 15/15 (4.8 s), Ultra 15/15 (3.7 s, but 2.3x Super's output tokens), Lightning 13/15 (11.6 s, about 2,800 hidden reasoning tokens per call). For this narrow task, model size barely matters; the schema and the capability list do.
  • Photo step vs ground truth (Oct 28, 40 live calls): 4 vision models x 2 photos of the room x 5 trials, scored against a hand-made list of the floor objects (all 40 answers schema-valid). Nemotron 3.5 Super VL, our observer, found the cardboard box and the soft bag 5/5 and the stool 0/5 in the placed-object photo, 3 of the 4 listed floor objects 5/5 in the other, with zero false positives and the same objects found in every trial (median 12.4 s). DeepSeek V4.1 Flash found more (the stool 2/5; recall 0.80 vs 0.67) with one repeated duplicate; Qwen 27B matched Nemotron's recall; Gemma 27B returned valid JSON with wrong boxes. LiDAR and photo cover each other's blind spots: the LiDAR missed the soft bag 0/4 and found the stool 4/4; the photo step found the bag 5/5 and missed the stool 0/5.
  • Twin update vs tape: the cardboard box was found in 4 of 4 runs (mean position error 13 cm), the stool in 4 of 4 (mean 32 cm). A soft bag 0.9 m off to the side was missed in all 4: survey stops are chosen for floor coverage, not to look at objects.
  • Return to the start by tape: 0–7 cm along the heading and 4.5–29 cm to the side, while the app's odometry estimate said it was within 2 cm every time. That gap is our main limitation, and it is measured, not guessed.

Time and people: reported vs measured

We did not measure a factory. The left column is what practitioners report for industrial twins, cited; the right column is what we measured in one living room on Oct 28 (build 61). TwinPulse does not make the initial build faster; it changes what happens after something on the floor changes.

Reported for industrial twins (cited, not measured by us) TwinPulse, measured in our living room
Initial twin build Practitioners on Gartner Peer Community estimate 4 months to over a year (thread); a vendor guide says "several weeks to several months", and for a whole facility "many months or even years" (Toobler). Unchanged. We start from an existing room scan (USD); building it is outside TwinPulse.
Keeping the twin current after a change A Scania/Volvo study: the virtual layout "often" does not match the physical factory; updates are "most often performed manually", a "cumbersome iterative process", "not done frequently" and "often scheduled when production is stopped" (Lind et al., SPS2022, doi:10.3233/ATDE220166). Warehouse twins "drifted from reality" without real-world validation (Modern Materials Handling, Apr 2026). One chat command, one robot run of about 7 minutes (under a minute from the command to the wheels turning, 5 min 42 s of driving in run 1), two confirmations (Run route; Apply or Skip on the LiDAR card). The box and the stool were on that card in 4 of 4 runs. The soft bag, invisible to the LiDAR, needed one photo: 46 s from upload to five candidates on screen, applied one by one from their previews (two of the five were applied that day).
People and tools Specialists editing 3D objects and logic in the twin tool. One operator in the chat; no modeling tool opened.
Model cost per update cycle - About one US cent at Token Factory list prices (one plan on Nemotron 3 Super, about 400 output tokens; one photo on Nemotron 3.5 Super VL, 3,000-8,000 output tokens).

Scope: one room, 4 mission attempts plus 1 guard-stop retry, objects on the robot's route. The minutes are ours; the months are other people's, and we have not run TwinPulse in a factory.

Who it is for and why it matters

Factories and warehouses already run AGVs and AMRs across the floor every shift, while a handful of people maintain the twin and learn about a moved rack or a pallet left in an aisle only when a simulation disagrees with the site. TwinPulse turns every robot run into a twin check: what the LiDAR saw that the twin could not explain comes back as placed objects to accept or skip, and the next route is planned on the corrected twin. A living room with a TurtleBot3 is the real site I can run under control. It proves the loop on a real robot; it does not prove industrial dispatch optimization, which is the next goal.

How we built it

  • NVIDIA Nemotron 3 Super on Nebius Token Factory (nvidia/nemotron-3-super-120b-a12b) plans robot missions: it receives an English instruction and a fixed list of robot capabilities and returns a JSON plan of task names (for example survey → return_to_start → stop) in about 5 s. NVIDIA Nemotron 3.5 Super VL, also on Token Factory, looks at room photos and proposes objects the twin is missing. Every call goes live to api.tokenfactory.nebius.com, and the route card names the model and shows the measured latency.
  • The model plans; it never drives. TwinPulse validates every plan against the capability list, computes the route on the RoomPlan geometry with keepouts (deterministic: the same twin gives the same route), and executes it as short finite moves, re-aiming after each leg. Robot-side guards (proximity stop, battery cutoff, output timeout, wheel-readback check, Stop) work independently of the app and the model.
  • NVIDIA Isaac Sim 6.0.1 + OpenUSD hold the twin. A TwinPulse extension owns its own USD subtree and shows the route, live robot pose and LiDAR; the app talks to it over a narrow, authenticated local command set.
  • TurtleBot3 Burger on ROS 2 with its LDS LiDAR, connected over SSH from a Windows PC.
  • Electron/React app + Python backend, with trace IDs and recorded outcomes for every request, operation and robot move. If the session record cannot be written, the robot refuses to move (fail-closed).
  • Built with AI agents. Codex and Claude Code drove the installed app through its visible controls and read the recorded evidence in a develop → operate → observe → diagnose → revise → retest loop. The "TEST MODE" badge in our recordings means this agent automation was switched on; the robot, the LiDAR data and the model calls are real. Physical sessions were always supervised and separately authorized.
  • Seoul. I attended Builders & Brews: Seoul Hack Edition (September 11, 2026, WOOMUL, Hapjeong, Seoul), the Physical AI-themed night of the Nebius tour, and entered as a Seoul attendee. TwinPulse is a one-person project built in Seoul.

The autonomous agent pipeline, and the 10× rule

Models propose; the host validates; the user approves; the robot's own guards execute. Seven stages: intent (host rules; stop words always deterministic) → observe (Nemotron 3.5 Super VL, strict-schema JSON, re-validated by the host) → plan (Nemotron 3 Super turns the instruction and the robot's six declared capabilities into a short JSON plan) → validate (capability list, step/distance/time limits; an invalid plan is rejected, not repaired or retried) → route (deterministic, on the twin geometry, shown on a card that waits for Run route) → execute (finite moves behind robot-side guards) → sense and update (unexplained LiDAR returns become rough boxes, with Undo; every move carries a trace ID).

The architecture is designed for an agent 10× more capable than today's models: a design principle, not a shipped feature. Its parts are (A) scaffolding around 2026-model weaknesses, (B) fixed host rules where the model is not yet trusted, and (C) safety, approval and evidence contracts. A stronger model lets us delete (A) and turn (B) into "the model proposes, the host verifies", one item at a time, promoted only after both paths are compared on the same held-out scenarios. (C) never changes: no validation, approval or physical safety control is removed because a model is more accurate on average. Item-by-item list: docs/analysis/TWINPULSE_AGENT_PIPELINE_AND_10X_DIRECTION_2026-10-29.md.

Design

  • One chat, one twin, every decision a card. The twin takes most of the window. The chat beside it holds the mission, the plan card (model, latency, task list), the route card with numbered stops, and a card listing the objects the robot found, each added as its own undoable twin edit.
  • Review or Auto-apply, always with Undo. Twin edits from photos and LiDAR can be accepted one by one (Review) or applied as they arrive (Auto-apply); and Undo last edit reverts the latest change. The repeat runs used Review and skipped every LiDAR candidate, so the runs themselves did not change the twin (the one photo step before run 2 added two objects away from the route).
  • No silent fallbacks. If the planner call fails, the app shows the error and offers the deterministic route as an explicit card to click, not as a quiet substitute.
  • A physical run is a supervised session. The operator confirms supervision, arms the robot and runs a bootstrap stop before the first route; Stop is always one click; the robot refuses motion once the session record is full.
  • Evidence by default. Every request, operation and robot move leaves a trace ID and a recorded outcome, and the physical runs were screen-recorded. That, together with the tape measurements in the room, is where every number on this page comes from.

Challenges we ran into

  • Odometry is not the truth, and the app cannot see the drift. The robot curves left a few degrees on every 3 m leg; early runs ended at a wall or a plant, so we added re-aiming after each leg and wider clearance. The Oct 28 tape measurements show what is left: the robot came back 4.5–29 cm to the left of its mark while its own estimate said within 2 cm. The start pose is declared and there is no twin-based relocalization yet, so TwinPulse trusts odometry it cannot check.
  • Guards that cost a run. A robot-side guard halted a move whose wheel readback did not match the command (no contact), and a final heading turn was refused because the 74-minute session had filled its 768 MiB recording budget and the robot does not move without a complete record. Both worked as designed, and both cost a run. The lesson: keep physical sessions under an hour, or reconnect between runs.
  • Real obstacles are bigger than their footprint. Plant leaves at LiDAR height stopped the robot a few centimetres before a stop. We added back-off and replanning around new obstacles.
  • Timing between the PC and the robot. A step sent the instant the previous one finished was silently ignored. We added a settle time and an acknowledgement timeout.
  • Model output discipline. Long free-form outputs sometimes ran into the token limit, ended in whitespace, or spent the budget on hidden reasoning with an empty answer. Short, schema-bound plans fixed this.

Accomplishments that we're proud of

  • A real robot kept a real twin current, on camera: the filmed run added 5 objects the scan had missed, and the next route, through one point I picked, was planned around one of them.
  • Repeat runs: 4 mission attempts plus 1 guard-stop retry gave the same planned route 5 times out of 5, 3 completed missions and 1 partial. Both misses were guards that worked as designed (no contact), and the return drift is measured with a tape, not estimated.
  • An earlier run completed the first fully automatic physical survey and return: 4 stops, 6.42 m.
  • Failed physical runs are documented with their cause and the fix that followed: Oct 25 runs, Oct 28 repeat runs, planner size benchmark, photo observer benchmark.

What we learned

  • Tight schemas and capability lists mattered more than model size for robot planning. We measured it afterwards: three of four Nemotron sizes scored 15 of 15 on the same request.
  • The LiDAR and the photo step miss different things (the soft bag only the photo saw, the stool only the LiDAR), so the twin needs both, and measuring each against a ground truth told us which.
  • Keep the model on the planning side of a hard boundary; let trusted code and the robot own motion and safety.
  • A number the app reports is not a measurement until something outside the app confirms it. Our odometry said "within 2 cm"; the tape said up to 29 cm.
  • A recorded failure with a clear cause is worth more than an unexplained pass.

What's next for TwinPulse

  • Future goal (the 24 s concept scene from 2:31 in the video, not a recording): one operating twin for a whole floor. Every sensor already on the floor (robot LiDAR, RGB cameras on robots and humanoids, fixed CCTV) reports changes into the twin, the operating logic follows, and AI control recomputes the routes of AGVs, overhead transport and humanoids. That extends what an ACS/MCS covers today from one fleet to the whole factory. Not built; the living-room run above is what exists.
  • Twin-based relocalization in place of a declared start pose, so the app can see the odometry drift the Oct 28 runs measured.
  • Survey stops that also look at objects, so a bag off to the side is not missed 4 times out of 4.
  • Natural-language changes to AGV operating and dispatch logic, re-simulated before deployment (the founding goal; not in this demo).
  • Calibrate → optimize → deploy: route-optimization machine learning in Isaac Lab on Nebius AI Cloud, then push the mission to the real AGV and stream telemetry back.
  • Tighter twin alignment with open-source reconstruction such as COLMAP, run on Nebius AI Cloud.
  • On-premises model endpoints for factories that cannot send data outside their network.
  • Customer discovery beyond factories: warehouses, hospitals and service-robot sites.

Built With

Share this project:

Updates

Submission history