We took inspiration from Doodle Halloween, however this time, we wanted to make it more interactive. Instead of drawing spells, we tried to make it so you could use your hands to create spells, so you could be the 10,000 year old wizard you've always wanted to be.
What it does
Detect hand gestures and displays an output spell
How we built it
Python was chosen for this project as it's a good language for computer vision
The computer vision component uses Google's MediaPipe Hand Landmarker task for hand gesture detection
OpenCV is used to convert video captures into numpy array
We trained our own Support Vector Machine classifier model from scratch to classify hand gestures.
Tkinter was used for the frontend component
The workflow:
OpenCV (Video frames) -> MediaPipe (Detect hand positions) -> ML model (Support Vector Machine to classify hand gestures) -> TKinter (to display)
Challenges we ran into
One of the biggest challenges we ran into was figuring out the threads for a project as we run multiple loop functions
Another challenge we ran into was how to display the feedback to the user (i.e. the spell to be played)
Accomplishments that we're proud of
That we actually were able to make a minimal viable product
The model we trained
What we learned
Learnt more about computer vision libraries such as OpenCV and MediaPipe
Learnt more about hand gesture detection and different models used to train hand gesture detection
Learnt more about machine learning and training our own ML classification model
Learnt more about Python multi-threading
What's next for Manus Magus
Add sound effects
More spells
Chaining of hand gesture that can make more powerful spells and grant more points/score
Design a start and end spell hand gesture that signifies when chaining of hand gestures starts and end
A local multiplayer method where the person who is the best at matching the hand gestures wins
Log in or sign up for Devpost to join the conversation.