Inspiration

The inspiration behind HeadsUp comes from childhood visits to disability support centres, where I witnessed firsthand the challenges faced by individuals with severe physical disabilities. Those experiences stayed with me and instilled a desire to use technology to make everyday life more accessible. We realized that something as simple as moving a cursor, clicking a button, or navigating a computer can be a significant barrier for people with tetraplegia.

This inspired us to ask: What if we could turn devices people already own into tools for greater independence?

That question became the foundation of HeadsUp.

What it does

HeadsUp transforms Apple's existing ecosystem into a hands-free computer controller, designed to make digital interactions more accessible for individuals with tetraplegia and limited upper-body mobility. By using motion data from AirPods, HeadsUp translates natural head movements into cursor movements on a Mac.

Our goal is simple: Make technology more accessible without requiring expensive, specialized assistive hardware.

How we built it

The central innovation behind HeadsUp is repurposing AirPods' built-in motion sensors as a computer input device. Using Apple's dev tools and Core Motion capabilities, we extracted real-time pitch, yaw, and roll data from compatible AirPods. These values represent changes in head orientation, which we translated into directional cursor movements.

We developed the motion-processing functionality in Swift and connected it to our TypeScript-based application, creating a bridge between the AirPods' sensor data and the computer's cursor-control system.

Challenges we ran into

Our biggest challenge wasn't simply making the technology work. It was understanding how to make it genuinely accessible.We quickly realized that building a functional prototype and creating an assistive technology that people can comfortably rely on are two very different challenges.

We had to consider several important factors, including:

1) Translating subtle head movements into accurate, stable cursor movements. 2) Distinguishing intentional gestures from natural, involuntary movements. 3) Coordinating AirPods motion data with camera-based blink detection. 4) Understanding how differences in mobility, comfort, and fatigue could affect the user experience.

These challenges pushed us to think beyond the technical implementation and consider the needs of the people we hope to serve.

Accomplishments that we're proud of

Our proudest accomplishment is turning an ambitious accessibility concept into a working hands-free computer-control prototype within the limited time of a hackathon.

What we learned

Throughout this project, we gained valuable experience working with Apple's motion-sensing technologies and translating raw sensor data into meaningful user interactions. We learned how pitch, yaw, and roll can be interpreted to control a cursor and how Swift and TypeScript can work together to create a responsive interaction system. But our most important lesson was that accessibility is not just about making something possible; it's about making it practical, comfortable, and reliable for the people who need it.

What's next

Our vision for HeadsUp extends beyond AirPods-powered cursor control.Our next step is to explore additional capabilities within the Apple ecosystem, particularly the Mac's built-in camera and eye-tracking technologies, to make hands-free navigation more intuitive, precise, and accessible. By integrating eye tracking, we aim to allow users to control the cursor through their gaze, while AirPods motion sensors continue to provide head-based navigation. Combining these input methods could create a more flexible experience tailored to different mobility needs. We also plan to collaborate with individuals with tetraplegia and other mobility limitations to gather real-world feedback, improve usability, and refine our technology.Ultimately, our goal is to transform HeadsUp from a working prototype into a practical, affordable, and adaptable assistive technology that empowers people to interact with digital devices independently.

AI disclosure

We used Claude and Claude Code to help with parts of the planning and coding.

Built With

Share this project:

Updates

Submission history