Inspiration
The inspiration came from two different scenarios that proposed the same issue.
- Cooking Although we think of ourselves as professional cooks, we still have to follow recipes. One of the most annoying issues when cooking is having to scroll down the recipe. Cooking is messy, so your hands are either occupied or are dirty, so if you could scroll via just waving your hand, cooking would be so much easier, and your phone would be way less sticker.
- Powerpoint slides. Powerpoint slides are another issue, you don't want to have to walk up to your laptop to click the next arrow button, or hold a mouse and prevent yourself from doing hand gestures. With GRASP, you can solve all those issues all at once. So, we wanted to create it so we can give the best pitches possible. ## What it does GRASP is a gesture activated system built into a glove. It detects hand tilts and movement via an accelerometer and your web cam using MediaPipe. Users can trigger keyboard shortcuts, navigate slides, and even teach the glove brand new gestures all wirelessly through WIFI.
How we built it
The hardware is an Arduino UNO R4 WiFi with a grove accelerometer a grove button mounted on the glove. The arduino sends gestures and raw data over UDP to a Python Flask. The backend then runs MediaPipe hand tracking and combines it with the data from the accelerometer
Challenges we ran into
Getting BLE to work on the R4 wifi with windows was a disaster and we eventually scrapped the whole idea entirely and switched to UDP over Wifi which worked out. Combing the accelerometer and MediaPipe data into one complete gesture took a lot of tuning. We also had to figure out cooldowns, calibration, and offsets to make the gestures feel a lot more solid.
Accomplishments that we're proud of
The custom gesture recording works. You preform a movement 5 times and the glove learns it. The time it takes from a hand movement to a keyboard action runs in under 200 ms.
What we learned
This was are first time debugging a project that was hardware and software based and so we learnt how to better tackle the problems. We learnt that BLE on windows is not the move and that UDP is better alternative.
What's next for GRASP (Gesture-Responsive Adaptive Signal Platform)
Better gesture recognition, using maybe a ML model trained on more samples. Support for more than one step gestures and sequences. Maybe a mobile app for managing your phone from a distance. Make the glove an actual glove thats wearable and smaller.
Log in or sign up for Devpost to join the conversation.