AirCare

Cover Image
4 Game Modes
Trend shows only after atleast 3 uses
Gemini Generates a Summary of all the Patients
Trend based on the results
Main Menu
Free Play
Reaction Test

Inspiration

I wanted to build something that combines object tracking, music, and physical well-being. After a lot of thinking, I came up with the idea of letting users play drums in the air using hand tracking. But then I started wondering: how could this actually be useful?

After hours of brainstorming, I realized that this could be more than just a fun game. It could also help clinicians and caregivers notice subtle things that might go unnoticed during regular interactions, such as small hand tremors or changes in movement patterns. And that's how AirCare was born.

What It Does

AirCare turns repetitive hand rehabilitation exercises into an engaging musical experience. Users can play virtual instruments while tracking their movement progress over time.

Key features include:

Free play and guided exercises Two-hand tracking with adjustable sensitivity Movement metrics to track progress A clinician dashboard with personal baselines and progress trends Gemini AI-generated summaries based on aggregate statistics Local data storage and offline support

How I Built It

I built AirCare using Python. It uses OpenCV and MediaPipe for webcam-based hand tracking, NumPy for movement analysis, Pygame for audio, and Tkinter for the desktop launcher. Session data is stored locally in JSON files, while a web dashboard displays progress.

For the AI summaries, I integrated Gemini to generate insights using aggregate movement statistics, without sending any video footage or raw hand landmarks.

Challenges I Faced

One of my biggest challenges was preventing accidental hits caused by tracking jitter. Initially, I used a drumstick-stroke detection system, but it wasn't reliable enough. I switched to using an open-fist gesture to trigger hits, which made the interaction more reliable.

I also had to deal with different lighting conditions, two-hand tracking, meaningful movement metrics, and data privacy. A lot of tweaking went into the sensitivity and other configuration settings to get the hand tracking working as well as possible.

Getting the Gemini AI summaries to work was another challenge. My API keys weren't working at first, and even when they did, Gemini sometimes generated only part of the summary. I eventually switched to a lighter Gemini model, which helped resolve the issue.

Finally, building four different game modes was pretty stressful. I reused most of the existing code and made smaller adjustments for each mode. There were still a few silly bugs along the way, so I used GitHub Copilot to help fix them because I was running short on time.

Accomplishments

Created a complete rehabilitation game without requiring specialized hardware Implemented a free-play mode and a guided mode that highlights which drum the user needs to hit Added two more game modes: Timed, inspired by Whac-A-Mole, and Reaction Test Built personalized progress tracking and an explainable clinician dashboard Added local storage, offline support, and optional AI-generated summaries Created original drum sounds and tested the core functionality Overall, I'm happy that AirCare makes rehabilitation feel more like playing music than performing the same exercises over and over again.

What I Learned

I learned that simple gestures can make an experience much easier to use and more reliable. I also realized that webcam-based measurements can vary depending on lighting and camera positioning, which is why personal baselines are more useful than claiming clinical precision.

On the technical side, I learned how to work with AI API keys, OpenCV, and MediaPipe, and got to revisit and improve my Tkinter skills.

What's Next

There's still a lot I'd love to improve if I had more time:

Improve hand tracking and calibration Bring back drumstick-stroke detection instead of relying on palm gestures Add more game modes and give therapists more customization options Strengthen data security Turn AirCare into a mobile app that uses the front-facing camera to track hand movements Add more drums and sounds The project has so much room to grow. These are just some of the things I'd try to implement if I had another 36 hours to work on it.

My ultimate goal is to make rehabilitation more enjoyable while giving clinicians meaningful insights into movement patterns in a way that respects users' privacy.