Sparky : AI Desk Companion by AMD Ryzen AI

Inspiration

What it does

Inspiration

What it does

Inspiration Students juggle homework, research, and a cluttered desk while AI tools stay stuck behind a screen. Sparky bridges that gap — a robot that lives on your desk, understands your environment, and actively helps you learn and stay organized, powered entirely on-device by the AMD Ryzen AI PRO 9 HX.

What It Does Sparky is a physical AI desk companion with three core capabilities: a homework assistant that answers spoken questions using an on-device LLM, a visual recognition mode that identifies and describes any object placed in front of it (Google Lens-style), and an autonomous desk-cleaning arm that detects and tidies clutter. Everything runs through a natural two-way voice interface powered by the AMD XDNA 2 NPU.

How We Built It All inference runs locally on the AMD Ryzen AI PRO 9 HX laptop — no cloud dependency. The XDNA 2 NPU (50 TOPS) handles vision, language, and speech concurrently. We built a Python orchestration layer on ROS 2 for motor control, with ONNX Runtime + DirectML accelerating our object detection model and quantized LLM.

Challenges We Ran Into An SSH session dropped mid-training, forcing a checkpoint recovery and relaunch. Our camera hardware failed mid-build, requiring a full swap and vision pipeline recalibration. Synchronizing the robotic arm's control loop with real-time vision inference without stalls took significant debugging time.

Accomplishments We're Proud Of We're proud of shipping a fully functional robotic system — live voice conversation, vision-guided object recognition, and autonomous desk cleaning — all running on a single AMD laptop with no cloud calls. Recovering from a mid-build hardware failure without scrapping the project was a win in itself.

What We Learned : We gained hands-on experience in training machine learning models, including working with vision-language models (VLMs) and integrating them with OpenCV for real-time perception tasks. We explored the concept of physical AI by deploying models on embedded systems and interfacing them with hardware. Additionally, we worked with cloud-based training pipelines, learning how to efficiently train, optimize, and deploy models using scalable compute resources. This project strengthened our understanding of end-to-end ML workflows—from data processing and model training to deployment and real-world integration.

What's Next for Sparky We want to add personalized tutoring by fine-tuning Sparky's responses per student using local model updates on the NPU. Next steps include upgrading to the AMD Ryzen AI MAX PRO for heavier workloads, piloting with K-12 schools, and eventually packaging Sparky as a turnkey AMD-powered developer kit for education robotics.