Spellcaster Academy

• 

  SpellCaster Academy

Inspiration

American Sign Language is essential to bridging the communication gap between deaf and non-deaf individuals. Current methods of learning ASL are slow, easily forgettable, and non-engaging.

What it does

Spellcaster Academy aims to fix these problems by gamifying the process, bringing communities closer together through an interactive and addictive RPG experience!

How we built it

We built the game itself using PyGame, and architected the Computer Vision ASL Recognition model on our own dataset using OpenCV, MediaPipe, NumPy, and scikit-learn using a RandomForest regressor.

Challenges we ran into

Both the Computer Vision model and main game loop need to be running at high frames per second for a seamless user experience, so we had to use threading and other core operating system concepts for performance.

Accomplishments that we're proud of

None of us had ever built ground-up AI models or games in a hackathon context, so there was a ton of learning required to get a polished final product, but we're incredibly proud of how fun and useful Spellcaster Academy turned out.

What we learned

As relative newcomers to game development and machine learning engineering, we all learned a ton about the fields and how to produce a quality product in such a short period of time.

What's next for Spellcaster Academy

The architecture of both the model and the game itself allows for incredibly easy configuration, just requiring ~30 seconds of webcam captures to add a new ASL letter or word, ~15 seconds for training, and a few lines changed in waves.json to add more enemies and waves. We would love to use this ease of expansion to cover even more ASL words and phrases.

Built With

• pygame
• python
• scikit-learn