Hey Jarvis

Inspiration

This past summer, I got to build and work with robotic arms, collecting data to train Vision Language Action Models (VLAs), which are built to control robotic arms. However, that made me wonder, what if an LLM could control a robotic arm? With VLAs, they only output actions, meaning no text, no discussion, no communication. But if an LLM could control an arm, then not only could it speak to people, but it could physically interact with them and the world.

How I built it

I began building the physical robotic arm last week from an open source arm. I modified the gripper to be a parallel design, and also added a camera mount, with everything being 3D printed. For software, only the basic servo calibration was done before the hackathon to ensure it was working properly. Everything else, including the motion control, Nemotron integration, safety limits, camera calibration, path planning, etc, was done during the hackathon. I used AI tools to help develop software, while I debugged, guided the direction of the project, created the pipeline, and executed physical tests.

Challenges I ran into

Initially, I spent a few hours simply doing research and planning out the entire pipeline: what data is needed as inputs, how to format the data, coordinate system conventions, what format it should output, what camera data to use, the list goes on. Once done, I felt confident and started implementing each phase. But eventually, when I attempted to integrate Nemotron to actually control the arm, it failed. It was 2 am and I was devasted, spending the entire night trying to debug and make it work. With only 6 hours left, I pivoted. Abandoning my initial pipeline and trying a different approach. And while it still isn't perfect, I'm extremely glad of this decision and the progress that I've made, finally creating a system that allows Nemotron to reason through and try to execute tasks.

What I learned

From this Hackathon, I learned how important the format of your data is for AI. Although in both of my tests, the data I was inputting was essentially the same, their format, such as the coordinate system, makes a huge impact. While my initial test made it "simpler" for the AI by using the camera axes, this actually made it more difficult for it to control the robot.