Inspiration
We wanted to explore and learn CV (computer vision) and we're considering using it for some games or data analysis. But during the development of our CV code, we took an educational/accessibility route, understanding the positive implications it could have on teaching young children to write characters & letters.
What it does
GestDraw is a motion-based game where you draw symbols in the air using just your hands, and your screen reflects it. Think JustDance for drawing, but instead of mimicing dance steps you're sketching shapes and patterns mid-air!
How we built it
We starting building the back-end portion of the code in VSCode with an assortment of libraries, primarly utilizing OpenCV and MediaPipe for computer vision and hand tracking, and Windsurf's Cascade to aid in our code development. By using the camera and hand tracking software, we were able to use the index finger and thumb point as a reference point for where our "pen" would be. With this, we started working on the game portion of the project where most of it was scripted with Windsurf's Cascade along with the constant debugging.
- Flask for backend
- React for frontend
Challenges we ran into
We encountered problems with the fluidity and stabilization when drawing lines, caused by camera limitations, MediaPipe processing delays, and just general noise and the setting surrounding the user. While some settings, like general noise and setting (in the dark) were unable to be compensated, we were able to stabilze our lines through smoothing filters and minimum confidence thresholds to counter any rigid lines or any accidental tiny dots. Also we encountered issues to the scope of where our camera can track our fingers as well as the accuracy of the tracking itself, which we were able to minimize enough via changing size and adjusting confidence intervals
Accomplishments that we're proud of
We're proud of learning how to use OpenCV as that was what we wanted to do collectively as a group, and not just use it, but implement it into something fun.
What we learned
We learned alot about how to not just use, but to talk to Windsurf's Cascade AI, primarly through an in-person workshop held this friday, in assisting and scripting our code. We also learned alot about the techincal limitations regarding CV (computer vision) and just OpenCV & MediaPipe themselves as begineers to the libraries and their functionalities. Aside from the techincals, we learned alot about how to work as a team, in regards to coding from the ground-up, splitting up work into front-end, back-end, functionality, game-dev, etc.
Log in or sign up for Devpost to join the conversation.