Inspiration
We wanted to build the ultimate robotic avatar to keep humans out of harm's way. The inspiration for GUARD.IAM (Ground Unit for Autonomous Reconnaissance, Distal Intervention, and Arm Manipulation) came from the need for a safely navigating exploratory rover that could brave hazardous, confined spaces. We envisioned a seamless fusion of cutting-edge AI and Mixed Reality, creating a system where an operator could either let the rover autonomously explore its surroundings or instantly take manual MR control to manipulate the environment using dual robotic arms.
What it does
We brought GUARD.IAM to life by merging advanced AI software with a robust physical and simulated hardware stack, all powered locally by the immense compute capabilities of RYZEN AI.Navigation & Localization: We utilized Nav2 for autonomous path planning. To achieve reliable indoor localization, we fused data from an Intel RealSense T265 using an Extended Kalman Filter (EKF). The EKF prediction step, which estimates the rover's state based on its previous state and control inputs, was critical for smooth movement:$$\hat{x}{k|k-1} = F_k \hat{x}{k-1|k-1} + B_k u_k$$AI & Manipulation: We integrated Vision-Language-Action (VLA) models and utilized Imitation Learning to teach the robot how to intelligently interact with its environment. In our simulated environment, the AI learned a policy $\pi_\theta(a_t | o_t)$ to successfully map visual and sensor observations ($o_t$) to complex robotic arm actions ($a_t$).The Operator Interface: We tied the entire system together using a Meta Quest 3, giving the operator an immersive, mixed-reality interface for sensor-fusion visualization and manual intervention.
Challenges we ran into
While our software and simulated environments were a massive success, the physical world fought us every step of the way. If a hardware component could fail, it did.
We battled faulty motors and unreliable motor drivers that constantly disrupted our physical testing. Connectivity was another major hurdle, with terrible internet slowing down our workflow and unexpected communication bugs plaguing our single-board computers (we spent hours wrestling with our "rubic pie" / Raspberry Pi!). Worst of all was the hardware graveyard we accumulated while pushing our electronics to the absolute limit: RIP to 2 ESP32s, an Arduino MEGA, and an ESP32-S3 that went out with a banzai.
Accomplishments that we're proud of
What we learned
We learned that the bridge between cutting-edge AI and physical hardware is incredibly fragile. Training VLAs and implementing imitation learning is an immense technical challenge, but we successfully built a functional simulated environment where GUARD.IAM could autonomously navigate and manipulate objects.
However, we realized that making those high-level AI models play nicely with physical sensors, misbehaving motors, and easily-fried microcontrollers requires an entirely different level of engineering resilience. We walked away with a profound understanding of sensor fusion, the true power of RYZEN AI for edge computing, and the crucial importance of robust hardware architecture when bringing autonomous robotics into the real world.
What's next for GUARDI.AM
Our next step is bridging the gap entirely between our highly successful simulated environment and the physical rover. We plan to refine our imitation learning models, upgrade our physical hardware to prevent future "magic smoke" incidents, and push the limits of what RYZEN AI can do for edge-based, autonomous robotics!
Log in or sign up for Devpost to join the conversation.