Handemonium

Inspiration

We were mostly inspired by interactive and stimulating classroom games such as Kahoot and Mentimeter that enhance learning in school. Whilst we wanted to emulate the interactive aspects of these games as much as possible, we also wanted to address the problems caused by the use of mobile phones in the classroom. Many teachers shy away from games like these due to the disruption that mobile phones cause in class, which we felt was a shame. Therefore we decided to use advanced computer vision techniques to address this problem with our Hackathon solution.

What it does

Our web application is an interactive class-wide quiz for students, to be run by the teacher. Our main solution was implementing the technology to replace the answering system with hand gestures instead of mobile phones. The teacher will run this website from their computer or laptop on the front of the class, for all students to see, and a number of multiple choice questions will be displayed. We also assume that schools will have datasets of their students already on their systems. The students will only have to raise their hand and signal either a 1, 2, 3, or 4 with one of their hands to select an answer, and our web app will recognise each student and their answer and automatically process the results. A leaderboard will be shown at the end of the quiz and after each question to keep students engaged.

How we built it

We developed Handemonium by integrating several technologies. The frontend, built with Next.js, provides a user-friendly interface for both teachers and students, while the backend, powered by Python and FastAPI, handles the pose and hand recognition algorithms. We chose Python for its robust libraries in machine learning and image processing, essential for accurate hand and pose detection.

The theory of object detection and classification was recently covered in our Computer Vision lectures, which helped us understand the details of how our ML APIs worked. We used the python 'face-recognition' library to detect the locations of faces within a photo. This library also allowed us to encode the faces of participants to identify them each round. We then used OpenPose to detect body, facial and joint key points - this allowed us to build a skeleton of a given person. This was cross-referenced with hand locations taken from the gesture detection library provided by MediaPipe, allowing us to associate users with their hand positions. Using this MediaPipe library, we trained a ML model on many hand images, counting from 1 to 4, which was used to classify the selection of the user.

The application is containerized using Docker, ensuring it runs seamlessly on any VM infrastructure, which facilitates easy deployment and scalability.

Challenges we ran into

One of the main challenges was achieving high accuracy and low latency in hand and pose recognition, crucial for real-time interaction in a classroom setting. We also faced difficulties in containerizing the application for deployment, ensuring it works uniformly across different environments. Balancing system performance with resource constraints on the VM was another hurdle, as was creating an intuitive UI that caters to both tech-savvy educators and young students.

Accomplishments that we're proud of

We are proud of creating a fully functional, innovative educational tool that bridges the gap between physical activity and digital learning. Successfully integrating hand and pose recognition technology into a classroom game represents a significant technical achievement. Additionally, making this technology accessible without the need for personal devices in the classroom is a step forward in inclusive education. Our platform has the potential to transform how educational content is delivered and engaged with by students.

What we learned

Throughout this project, we learned about the complexities of deploying machine learning models in real-world applications, especially in an educational setting. We gained valuable experience in developing and deploying a full-stack application, from frontend design to backend logic, containerization, and VM management. We also learned the importance of user testing and feedback in refining our product to better meet the needs of our target audience.

What's next for Handemonium

Looking ahead, we plan to expand Handemonium's capabilities to include more complex gestures and a wider range of questions and activities. We aim to enhance the accuracy and responsiveness of our pose recognition algorithms to support larger groups and more dynamic classroom environments. Additionally, we will explore integrating with other educational platforms and tools to provide a more comprehensive learning experience. Our ultimate goal is to make Handemonium a staple in classrooms worldwide, making learning more fun, interactive, and accessible for all students.