Inspiration

When large AI models arrived, I asked myself: why not put them into robots? There is already a lot of Physical AI, but almost none of it reaches competitive robotics such as VEX, FTC and FRC. What competitive robotics does have is controllers that already work: PID, LQR, guiding vector fields and LTV. So the idea was simple. Don't train robot AI from scratch. Give it the tools every robotics programmer already knows, let it learn faster with them, and keep the proven controller in charge.

What it does

One controller, two worlds. A C++20 control cascade drives our VEX V5 X-drive robot: an LTV-LQR outer loop recalculated every 10 ms and four wheel speed PID loops. The same C++ code is compiled to WebAssembly and runs in a browser simulator of the V5RC Override field, so the simulator tests the exact code that drives the robot.

Learning on top, not instead. A guiding vector field gives the robot a safe path to follow. A neural network (residual TD3) adds only small, limited corrections to that path. It never sends voltages to the motors. If the network fails, the robot keeps driving on the proven controller.

NVIDIA Nemotron on Nebius Token Factory is the coordinator. Through tool calls it turns a plain-language request into a checked motion task, and it can play a whole match by choosing one legal action at a time. The game's rules engine calculates the points and time of every option first. If Nemotron gives an invalid answer, the match stops instead of guessing.

Tactical AI with real robot timing. On our robot, a cycle from the loading zone takes about 2 seconds and picking up an object from the field takes about 4 seconds. Using these times, one driver period in our match model gives these score margins:

Loading-zone tactic (our team's strategy): 246
Nemotron 3.5 Lightning on Nebius: 231
Best field-only tactic: 218
Best single move each time: 203
Our trained tactical network: 112
Random moves: 62.5

RoboProof keeps us honest. It stores fingerprints of the code, keeps test situations out of training, saves every run including failed ones, and does not allow us to call anything an improvement until an independent check confirms it.

How we built it

Control: the robot follows a smooth curve (spline or Bezier, using our own C++ rewrite of the Java Pedro Pathing library). LTV-LQR keeps it on the curve, and four PID loops hold each wheel's speed. The same code is compiled to WebAssembly for the simulator, and tests check that both versions come from the same source.

Learning: our main network has 10 layers of 512 neurons and 2,384,387 parameters, and it is the only part needed to drive. Its two training critics have 4,769,794 parameters (14.3 million with their target copies). The tactical network is small, with 28,034 parameters, and learns from real score changes in the game engine.

Compute: we trained on an AMD Instinct MI300X (564 episodes, 100,051 decisions) and on a free NVIDIA Tesla T4 in Google Colab (317 episodes, 56,457 decisions, plus three tactical training runs). Nemotron runs on Nebius Token Factory.

Tools: C++20, PROS, LemLib and Eigen on the robot; JavaScript, WebAssembly and Three.js for the simulator; Node.js for RoboProof and the Nemotron connection; Python and PyTorch for training. One command, npm start, launches everything on Windows, macOS or Linux, and the project has 619 automated tests.

Challenges we ran into

Speed: Nemotron on a laptop CPU needed 153.7 seconds for one planning request. On Nebius Token Factory one tactical decision takes about 0.6 seconds, which made it possible to coordinate a whole match.

Our model disagreed with the real robot: with guessed timing, field play beat the loading-zone tactic (228 vs 182). After we used times measured on the real robot, the order flipped (246 vs 218). Simulators must be calibrated, not trusted.

Free compute is not guaranteed: Google Colab ran out of free GPU time during the project. We never bought compute and recorded every run that could not happen.

Stopping safely in practice: the larger Nemotron Super model once returned an invalid answer in the middle of a match, and the system stopped exactly as designed. A bug in the trainer also left one run marked as unfinished; we fixed it and kept the original record.

Accomplishments that we're proud of

The same C++ controller runs on the robot and in the browser, and tests prove it.
Nemotron plays a full match through tool calls without invalid choices and beats both simple tactics.
The AI never controls motor voltages directly, and every claim in the project comes from a saved run.
We report negative results honestly: our trained networks do not yet beat our team's loading-zone tactic.

What we learned

Giving an AI clear tools and a rules engine that checks it works better than asking it to invent numbers.
Measure the real robot first: two timings, 2 seconds and 4 seconds, changed which strategy wins.
An AI that stops is safer than an AI that guesses on a real robot.
More training is not a replacement for better information: longer training improved practice games but not new matches.

What's next

NVIDIA Cosmos (Cosmos-Reason2): the connection that will describe recorded robot motion is prepared; running it on a GPU with at least 24 GB of memory is next.
LocateAnything: the camera data format and floor calibration are prepared so a camera can turn "the cup by the goal" into a target for the robot; turning on camera recognition is next.
Two robots acting at the same time in the tactical model, more match information for the network, and starting the tactical network from our team's loading-zone tactic.
Larger tests on Nebius Serverless Jobs, and bringing the approach to FTC and FRC.