Inspiration
At the hackathon’s Seoul kickoff, a presentation about training robot policies got me thinking: what if you want to use a Policy trained only on Robot A, to drive another Robot B? Then came a second question: if I improved the policy on its original robot, would that improvement carry over through the same adapter?
I wanted to explore this hands-on. My first reaching experiment proved the simulator worked, but I wanted to see a robot pick something up. I switched to a banana-and-bowl task with two simulated arms: a Franka (Robot A) and a Kinova (Robot B).
Reusing policies could make robot applications more accessible and reduce the cost of training separately for each arm. I haven’t measured those savings; they’re the motivation for exploring the idea.
What it does
NVIDIA’s GR00T N1.7 DROID policy, trained for a Franka, controls a simulated Kinova through a fixed adapter. The adapter translates robot state and movement commands, changes the exterior image so the policy sees a familiar Franka, and adds a small hardcoded lift.
With the adapter frozen, the original policy completed 7 of 12 shared test layouts on each arm, with no training on Kinova demonstrations. That supports task transfer in this small simulation experiment.
I then fine-tuned the policy on 27 successful Franka demonstrations and tested both arms on the same twelve layouts again. Franka fell from 7/12 to 2/12, and Kinova from 7/12 to 0/12. The scorer and rules were unchanged.
I had enough budget for one only more simulation, so I threw a hailmary, I call, ABBA (Alpha Bravo, Bananas Away). NVIDIA Nemotron 3.5 Super VL watched the robot’s actual exterior and wrist cameras through Nebius Token Factory. With permission to override the policy for Robot actions; It could request a hold, confirm on fresh views, and trigger the gripper to open. The policy still controlled the arm’s movement through the fixed adapter, until Nemotron wanted to take over.
That one selected Kinova run passed(!) the unchanged scorer. It demonstrates the integration working once; it does not establish reliable improvement or isolate Nemotron’s contribution from the controller’s hold and stay-open behaviour.
How I built it
I used Isaac Sim 5.1, Isaac Lab 2.3.2 and RoboLab, with GR00T serving and fine-tuning on a Nebius AI Cloud RTX PRO 6000 GPU VM. Nemotron ran through Nebius Token Factory.
The adapter developed in stages. Movement translation alone wasn’t enough. While investigating, I found Mirage’s cross-painting technique and implemented a simulator version: render a Franka into the Kinova’s exterior camera input. In three paired development comparisons, painted runs carried the banana to the bowl; unpainted runs did not lift it. Watching the painted failures, I noticed the banana hitting the rim and proposed the 15 mm lift.
I kept the adapter and primary scorer unchanged across the before/after comparison and used a locked checkpoint-selection rule. The comparison covers 48 episodes: two robots, two policy versions and twelve shared layouts. All failures and safety stops remain in the record. CPU replay reproduced every locked score. Training-plan changes and later exploratory analyses are documented separately.
For ABBA, the simulation paused while Nemotron examined the images. It returned WAIT 27 times, then READY; the controller held the arm and requested a fresh confirmation before opening. The 29 calls cost about $0.0223, estimated from returned token counts. The older live checker reported no success; the unchanged offline scorer recorded PASS. Both results are disclosed.
This is a solo project. I did the first few weeks by hand, then due to time constraints and desire to complete all my goals- was required to leverage AI agents for implementation, experiment operation, analysis, drafting and review. My personal learnings are available in my Notebook uploaded onto the website I made for this project.
Challenges I ran into
Robot appearance mattered as well as movement. The grippers also closed at different speeds, so the adapter needed timing adjustments.
The biggest setback was fine-tuning. An audit found that, in 25 of 27 demonstrations, the successful opening fell outside the first eight targets of any valid training sample—the portion executed before replanning. The release was present later in the training targets; this was a sampling gap, not simply missing footage. It gives a concrete repair to test, but does not prove the cause of the regression. I ran out of time and compute for another training-and-evaluation cycle.
Accomplishments I’m proud of
- Getting a Franka-trained policy to complete the task on the Kinova through a fixed adapter.
- Finding cross-painting through research when the original approach failed, then seeing it change the behaviour.
- Integrating Nemotron into an actual simulated control flow for one successful assisted release.
- Keeping the footage, scores, source locks and limitations available for inspection.
What I learned
I learned about policies, kinematics, simulation, gripper dynamics, training-data sampling, evaluation design and the cloud infrastructure needed to run all of it. Finding cross-painting was exciting; seeing the adapter work was a relief; seeing ABBA release the banana was a great final moment.
Cloud setup also taught me about SSH, security groups, GPU selection, billing delays and stopping resources properly. Those lessons are included in my platform feedback.
This experience made me want to pursue a career in robotics and machine learning. I’d love the opportunity to keep working on this problem.
What’s next
If I was to continue this project, I would start it from scratch with all my learnings - in order to create a more efficient, and reliable adapter - through a more scientific and patient approach.
A longer-term idea is a shared learned adapter trained across several robot types, then tested on unseen robots. That was outside this project’s budget and scope. Everything demonstrated here is in simulation; cross-painting and lift eligibility use simulator state.
Thank you to NVIDIA, Nebius, Tavily, and Bloom Korea for the opportunity to participate.
The cross-painting technique is credited to Chen et al., Mirage, RSS 2024; the project page and report list the research, models, tools and asset licences.
