Argetlam

Inspiration

We were initially inspired by the fact that there are many groups within UBC who would struggle to interact with the robots that UBC Rover builds. This problem inspired us to design multiple interfaces connected through intelligent models like Nemotron.

Many robotic systems still rely heavily on conventional interfaces such as joysticks, keyboards, or specialized controllers. We wanted to imagine a future where interactions between humans and robots are not limited to these methods, but can instead happen through simple body movements, voice commands powered by LLMs, and perhaps, even further into the future, EEG-based mind control using brain signals.

What it does

Argetlam is a multi-interface human-robot interaction system connected to and tested on a real, physical six-degree-of-freedom (six-axis) robotic arm.

We built it around three interaction pathways:

• Body movement: A webcam tracks the user's gestures using MediaPipe. Moving a hand guides the robotic arm, while additional gestures control reach and the gripper.
• Voice commands: Users can speak naturally to the robot. Local Whisper speech recognition and NVIDIA Nemotron 3 Nano, accessed through Nebius Token Factory, translate instructions into structured robotic actions.
• EEG and physiological signals: A Muse 2 headband captures EEG brain-wave signals for grasp-intent research, while jaw-clench EMG signals provide an additional way to control the gripper.

All three pathways connect through a shared control architecture, allowing different forms of human intent to interact with the same physical robot. We also designed this architecture with future expansion in mind, so more interaction pathways can be integrated as we continue developing Argetlam.

How we built it

Argetlam combines computer vision, large language models, biosignal processing (EEG and EMG), robotics frameworks such as ROS 2 and NVIDIA Isaac Sim/Isaac Lab, and physical hardware integration through microcontrollers and motor-control firmware.

1) For body movement, we use MediaPipe Pose and Hands to track the operator through a webcam. These movements are mapped to robotic arm coordinates using calibration and inverse kinematics, allowing users to guide the arm with gestures.

2) For voice interaction, we use Whisper to transcribe speech locally and NVIDIA Nemotron 3 Nano through Nebius Token Factory to interpret natural-language instructions. We designed a custom prompt and structured JSON output schema to translate these instructions into robotic actions. Simple commands and emergency stops are handled locally to avoid unnecessary inference delays.

3) For EEG and physiological interaction, we use a Muse 2 headband to acquire brain-wave signals and detect jaw-clench EMG activity. We developed signal-processing and classification pipelines to explore grasp-intent detection, alongside an EMG-based method for controlling the gripper.

One of our most important design decisions was to make Argetlam simulation-first. Instead of sending user commands directly to the physical robot, all commands pass through a shared control system operating in NVIDIA Isaac Sim. The controller puts inputs into joint targets using inverse kinematics, applies workspace and motion constraints, and executes the movement in simulation. These joint targets are then mirrored onto the real robotic arm through ROS 2 Jazzy and its Teensy-based stepper-motor control system.

This architecture was particularly challenging to implement. The physical arm does not behave exactly like its simulated counterpart, with differences in motor speeds, communication delays, and joint responses. We had to measure these differences, calibrate our models, and adjust the control system so that the real arm could reliably follow the simulated motion.

However, this approach also became one of Argetlam's greatest advantages. By developing and testing movements in simulation first, we could iterate rapidly, experiment with new interaction methods, and enforce consistent safety constraints before sending motion targets to physical hardware.

Most importantly, all three interaction pathways share the same underlying control architecture, making it easier to introduce additional interfaces without rebuilding the robotics system.

Challenges we ran into

Challenges we ran into

Syncing the simulation with the real hardware was a big challenge, simply because real motors had a lot of communication delays, velocity differences, and mechanical limits that wouldn't appear in an idealized simulation and thus couldn't really be prepared for. We also had a lot of issues with distance measurements from the depth camera that were then passed into Nemotron, as well as syncing the physical environment that the real arm could interact with and the environment that the simulated arm could interact with.

We also had issues with the LLMs themselves. Sometimes, if we let some of them think for too long, they would just get stuck in a loop, and no matter what we did, we couldn't really get them out of it. After a lot of iterations, we figured out that simply turning extended reasoning off, while blocking off some of their thinking, could be compensated for with smart context engineering. In return, we got a quicker model that was more context-aware and more capable of translating natural language into robotic instructions.

In terms of body teleop, we very much struggled with translating human movements into the robot's joint coordinates. There wasn't really an open-source framework that fit what we needed. The closest was NVIDIA's hand teleop, but since our arm was completely custom, and we really wanted a natural feeling of our own movements directly mapping onto the robot, we had to build much of it ourselves. That required a lot of simulation work, as well as real-world validation and iteration.

We also had a lot of issues with brain signals because the Muse is a four-channel BCI, and thus has some degree of error. A lot of the work was tuning the consumer headset to ensure that eye movements and muscle artifacts wouldn't be an issue. We also struggled with our EMG and EEG recognition models, but improved them through calibration and a bunch of other adjustments.

Finally, integrating all of these into one system required us to first plan and design what our integration layer would look like, how it would accommodate all three areas we were building in, and how it could support future interaction interfaces. After a bit of planning on paper, we were able to flesh that out and bring everything together.

Accomplishments that we're proud of

We think our biggest accomplishment was progressing beyond an NVIDIA Isaac Sim-only prototype to controlling a real robotic arm through our simulation-first software model. We were able to take an old arm from UBC Rover's past competitions that we weren't using anymore, bring it back, rebuild it properly, and physically integrate it into our system. We then connected our custom software to the arm's existing firmware so that we could execute voice-driven poses and movement commands on the real arm, with the movements first appearing on the simulated arm.

We also had to manage our networking, since NVIDIA Isaac Sim required a large, powerful computing box that was actually located across campus in Pranav's room. Through some clever networking tricks, we were able to cut down communication delays from roughly three or four seconds in the two-way channel to around 1–10 milliseconds.

Another accomplishment was optimizing Nemotron. Initially, we struggled with getting the model to understand even basic movement commands, since it wasn't always sure what we wanted it to do. Building a local parsing pathway dramatically improved our speed and reliability, while letting Nemotron focus on more complex natural-language instructions. We also achieved 19/20 correct intent classifications with a median inference latency of 0.44 seconds after optimizing its reasoning configuration.

We're also proud that we were able to get the simulation to closely match the real robot. This meant that most of our testing and iteration could be done inexpensively in simulation rather than directly on the physical machine, until we reached the point where real-world validation and integration were needed.

And of course, bringing together body movement, voice commands, and EEG/EMG signals into one shared control architecture was an accomplishment in itself.

We're particularly proud that we were able to take older technology from UBC Rover's past competitions and give it a new purpose, using it to explore a new direction for accessible human-robot interaction.

What we learned

What we ended up learning is that understanding an instruction within a simulation is fundamentally different from actually implementing it in the real world. Even though we eventually had a robust simulation-to-real interface, that was largely thanks to NVIDIA Isaac Sim. We had struggled with RoboSuite much earlier, particularly with distance measurements and getting the depth camera to sync with reality. The depth camera's version of reality wasn't always the same as the simulator's version of reality, and neither was necessarily the same as what was actually happening in the real world.

We also learned that while an LLM can understand what a human wants, robotics requires much more precise timing, coordinate transformations, and mechanical constraints. Because of this, the traditional advice of increasing a model's reasoning capabilities isn't always the best approach. We found that disabling extended reasoning not only improved inference speed but also allowed us to be more deliberate about safety, using a strong but smart prompt-engineering harness alongside our existing safety constraints. This allowed us to speed things up without sacrificing our ability to interpret natural-language commands.

Our EEG experiments also taught us that future brain-signal interaction with our robotic arm would be much more complicated than we initially imagined. We'd need significant hardware upgrades to accomplish anything much larger than gripper-level control, which is why we kept that part of the project experimental. While we trained models on publicly available international EEG datasets to explore controlling the arm through human intention, we discovered that bringing those capabilities into the real world would require much more capable hardware and further research.

But more than anything else, while we originally understood the issue of accessibility in robotics through our research for this hackathon, we came to realize just how important that issue really is. Working on Argetlam reaffirmed our commitment to improving access to robotics for everyone, not just at UBC, but throughout Canada and the world.

What's next for Argetlam

Argetlam is still an early research prototype, and there's very much a lot of work ahead.

One of our biggest goals is to expand real-world testing across all three interaction pathways, particularly with people who are directly affected by the accessibility issues we're trying to address. We want to test the robotic arm's autonomous capabilities in these contexts, understand how people actually interact with it, and fix the bugs and limitations that we discover through real-world testing.

We also want to explore better EEG hardware to make our brain-control interface much more concrete, moving beyond our current experimental work toward more reliable control of the robotic arm through brain signals.

But in the end, we want to imagine a future where human-robot interactions are more accessible, not just to the majority of people, but also to those who might not always be considered when we design robotic interfaces.

And we think Argetlam is our first step toward that future.

Built With

• isaacsim