Gesture tracking and recognition to control ghost movements.
Character selection carousel.

Inspiration

When thinking of something a Halloween character would use on a day to day basis, the main idea that came to mind was with various monsters have certain limitations in their abilities, such as a ghost phases through things and thus can struggle to scare people. The idea was to build something monsters could use to practice/train their ability, which is where we came with the idea of Fright Academy.

What it does

Fright-Academy is a web page that users (monsters) can use to train and practice various tasks and actions. The user will first be prompted to choose which monster type they are and then they will be tasked to complete various tasks associated with their monster type.

How we built it

The web page, comprised of different 2D and 3D page elements was built using Three.JS, HTML and CSS, while the specific gesture controls and camera feature for the ghost was built using Mediapipe.

Challenges we ran into

While creating this project, we wanted to keep in mind that the entire theme is that a Halloween character should use it in their day-to-day. We had difficulty in coming up with ideas that would be useful for monsters, which led us to debating whether a feature would be helpful and scrapping ideas we figured would not assist them. Additionally, thinking about how a monster would interact with the program gave an additional level of difficulty - how would a ghost interact with a program? What sort of challenges does a vampire or werewolf have in their day-to-day?

Accomplishments that we're proud of

We're proud of bringing Fright Academy to life by combining multiple technologies into one interactive experience. We built 3D haunted rooms featuring a ghost, moving visitors, and animated objects using Three.js. We added hands-free controls through real-time hand gesture recognition, head movements, and voice commands using MediaPipe and the Web Speech API. We also created unique abilities and a training system for four monster types: Ghosts, Vampires, Witches, and Werewolves. Finally, we integrated Google Gemini to interpret natural voice requests, making Fright Academy a unique experience designed around how monsters would interact with their world.

What we learned

The biggest takeaway from building Fright Academy was learning to integrate alternative ways to interact. Fright Academy uses voice recognition as well as hand and head gestures that correspond to distinct actions - however each of these methods have different delays and failure cases. This project taught us to think about how the user would interact with the program to make a coherent experience, for example, we thought about how a ghost isn't able to physically touch anything, so it has alternative ways to perform actions. We can apply the same way of thinking for a person who may be disabled and cannot control the program using their hands.

What's next for Fright Academy

Moving forward, Fright Academy would introduce adaptive difficulty and personalized progression to allow monsters of all abilities and skillsets to grow and evolve from the training course. A base benchmark test could identify the user's current standing and personalize a training program with specialized challenges to target weak points.