Kenesis

Rover drive base
3 DOF Arm
Web UI

Inspiration

We wanted to build a robot that feels like an extension of yourself—something you can control as naturally as moving your hands. AR glasses provided the perfect interface to make telepresence more intuitive, and we imagined everyday scenarios where such a robot could help: picking up trash, delivering food, or assisting those with mobility challenges.

What it does

Kenesis is a multipurpose telepresence robot controlled entirely through AR glasses. By pointing, you drive the robot in any direction. With a gesture, you switch to robot mode, see through its camera, and operate a 3-DOF robotic arm with a claw for grabbing items. Its modular design allows attachments—trash bins, delivery trays, or cones for traffic control—making it adaptable for many real-world uses.

How we built it

We used a Jetson Nano as the brain, handling hand-tracking AI, motor control, and ROS2 integration all in one platform. AR glasses send hand position and gestures to the Nano, which translates them into robot movement and arm control. To make the system more modular and support communication across different devices, we introduced an MQTT layer with HiveMQ between our Flask server and ROS2 nodes. Flask collects gesture data from the AR glasses and publishes it to HiveMQ. On the robot’s side, ROS2 nodes subscribe to these HiveMQ topics, translating the incoming messages into motor commands and arm control signals. This architecture ensures that gesture data can flow reliably and with low latency between devices, while HiveMQ’s scalability makes it easy to expand to multi-robot scenarios in the future. The robot’s modular chassis supports add-ons, and the camera streams live video back to the glasses for immersive teleoperation.

Challenges we ran into

Getting low-latency hand tracking to feel natural through AR.
Integrating motor control, camera streaming, and AI inference on a single Jetson Nano.
Designing a robot arm that balances simplicity (3 DOF) with useful functionality.
Ensuring modularity so attachments can swap in and out without complex wiring.

Accomplishments that we're proud of

Getting low-latency hand tracking to feel natural through AR.
A fully working pipeline from AR gestures → Flask → HiveMQ → ROS2 → robot movement and arm control.
Integrating motor control, camera streaming, and AI inference on a single Jetson Nano.
Designing a robot arm that balances simplicity (3 DOF) with useful functionality.
Ensuring modularity so attachments can swap in and out without complex wiring.

What we learned

We learned how challenging it is to merge AR interaction with robotics in real time. We gained experience with ROS2, motor driver integration, and designing modular platforms. Most importantly, we learned the value of user-friendly controls—gestures feel far more natural than joysticks.

What's next for Kenesis

We want to:

Improve hand-gesture recognition with more robust AI models.
Expand the library of modular attachments (delivery box, vacuum module, accessibility tools).
Optimize the camera streaming for even lower latency.
Explore multi-robot collaboration for larger tasks.