handoff

Inspiration

We gave AI a hand. Literally.

Artificial intelligence can see, listen, reason, and communicate. But what if it could physically interact with the world?

That question inspired Handoff, an AI-powered wearable exoskeleton designed to bridge the gap between artificial intelligence and human movement.

Our biggest motivation was accessibility. Millions of people experience paralysis or limited hand mobility, making everyday activities significantly more difficult. We envisioned a future where AI could provide physical assistance, helping people regain independence.

But we wanted to go beyond accessibility. Imagine learning piano, practicing a new skill, or completing an unfamiliar task with an AI that doesn't just explain what to do, but can help guide your movements.

We wanted to take AI beyond the screen and into the physical world.

What It Does

Handoff combines a wearable robotic hand exoskeleton with AI-powered smart glasses to create an interactive system connecting human movement with artificial intelligence.

The system is designed around three main capabilities:

Physical Assistance: A tendon-driven exoskeleton designed to assist individual finger and thumb movements, with the long-term goal of supporting people with limited mobility.
AI-Powered Learning: A system intended to provide real-time explanations and guided movements, helping users develop physical skills through hands-on learning.
Multimodal Interaction: Smart glasses equipped with a camera, microphone, and speaker, allowing the system to capture visual information, receive voice input, and communicate through speech.

Our goal is to allow users to ask questions, receive intelligent guidance, and eventually translate approved AI actions into physical movement.

How We Built It

We approached Handoff as two interconnected systems: an intelligent perception interface and a physical actuation system.

For the wearable interface, we used an ESP32-S3 Sense with an integrated camera and microphone, alongside an external audio amplifier and speaker. We developed firmware supporting simultaneous live camera streaming, microphone capture, and text-to-speech playback.

For physical movement, we designed a wearable exoskeleton using six ST3215 serial bus servos, custom 3D-printed components, tendon-driven mechanisms, and PTFE tubing. Each actuator is assigned to a finger or thumb movement, allowing us to work toward independent hand control.

We also incorporated a Waveshare servo interface and designed the architecture around a central AI application responsible for interpreting user requests and coordinating the hardware.

Our software and hardware were developed together, with an emphasis on modularity so that individual components could be tested, improved, and integrated independently.

Challenges We Faced

One of our biggest challenges was translating rotational servo movement into natural finger motion. Unlike a conventional robotic hand, our exoskeleton must accommodate the complex mechanics of actual human joints. We experimented with tendon tension, spool geometry, routing, and mechanical alignment to make movement more predictable.

Another major challenge was working with hardware we had never used before. This was our team's first time soldering, and we had to learn how to properly solder header pins onto our ESP32-S3 microcontroller. We learned about soldering techniques, electrical connections, and how to avoid damaging sensitive components. What initially seemed like a simple task became an important hands-on learning experience that helped us better understand embedded hardware development.

We also faced difficulties getting our camera, microphone, and speaker to operate simultaneously on a single ESP32-S3. This required troubleshooting I²S communication, audio output, memory allocation, camera orientation, and live video streaming.

Beyond the electronics, we had to consider wearable comfort, cable management, power distribution, and safe servo operation. Integrating so many different technologies into one wearable system challenged us to constantly troubleshoot, adapt, and improve our design.

Each obstacle pushed us to learn something new, turning our first experiences with unfamiliar hardware into a working foundation for Handoff.

Accomplishments We're Proud Of

We're proud of bringing Handoff from an ambitious idea to a working prototype.

We successfully configured and controlled six ST3215 servos, integrating them with custom 3D-printed components and tendon mechanisms to enable finger and thumb movement.

We also achieved simultaneous camera streaming, microphone input, and text-to-speech output on a single ESP32-S3.

Beyond the technical achievements, we designed and assembled our wearable exoskeleton, learned to solder for the first time, and gained valuable hands-on experience in robotics and embedded systems.

Most importantly, we established the foundation for connecting AI perception and communication with a wearable physical interface.

What We Learned

Handoff challenged us across robotics, embedded systems, computer vision, mechanical design, and software integration.

We learned that creating physical AI involves much more than generating intelligent responses. It requires translating software decisions into precise, reliable physical actions while accounting for mechanical limitations and human safety.

We also gained experience combining multiple hardware systems into a single architecture and learned the importance of iterative prototyping.

What's Next for Handoff

Our vision extends far beyond this hackathon.

We want to develop reliable, individualized finger control, expand AI-driven movement assistance, and support increasingly natural interactions between humans and intelligent systems.

Future development includes more advanced movement calibration, more precise real-time visual understanding, personalized learning assistance, and collaboration with accessibility specialists to evaluate potential assistive applications.

Ultimately, we envision Handoff as a step toward a future where AI doesn't just tell people what they can do.

It gives them a hand.