MarioLuigi

• 

  MarioLuigi

Inspiration

The idea for MarioLuigi originated from a personal experience with our teammate's 80-year-old grandmother. As individuals age, it is common for motor skills and cognitive abilities to decline, making everyday tasks more challenging. It's not only for the aging population, but children with disabilities and individuals who are victims of accidents require rehabilitation to train their motor skills. In the case of our teammate’s grandmother, simple activities such as managing her daily schedule became increasingly difficult due to diminished dexterity and memory. There are also numerous individuals avoiding rehabilitation due to several reasons including shame, fear, lack of support, lack of commitment, and grief which leads to isolation. Therefore, our team wants to design an application or tool to support a diverse range of users, focusing on individuals who face challenges in motor skills, physical mobility, or cognitive conditions such as autism spectrum disorder (ASD) by introducing engaging motor skill practices and exercises.

What it does

The system leverages Mediapipe and OpenCV to process real-time gestures and motion inputs. These inputs are then integrated with PyAutoGUI to automate tasks on the user interface, simplifying complex actions. Jupyter Notebook is used to generate the tool (or application), where a clear user interface is provided alongside the performance evaluation.

The main component is integrating the tool with the Tetris game, where the tool (or the webcam) captures the specific hand gestures and corresponds to the key to play the game. This enables an engaging and interactive rehabilitation method. Tetris was chosen for its challenges, where both an individual's motor and cognitive skills are trained.

How we built it

The algorithm captures frames from a webcam and processes them using the Mediapipe library to detect hand gestures. It recognizes specific hand gestures (up, down, left, and right) by analyzing the positions of the hand landmarks, such as the thumb, index, and other fingertips. Based on the detected gesture, it simulates a corresponding key press (e.g., "up", "down", "left", or "right") using the pyautogui library, with a short delay between key presses to prevent multiple presses. This application provides a simple user interface, where performance percentage is provided to the user based on the number of gestures detected to help evaluate caregivers and individuals. An existing web game is integrated with our tool, where our tool acts as the main device/application (just like our phone having Google inside!). The program continues running until the user presses the 'q' key to exit.

Challenges we ran into

One of the most challenging aspects of this hand gesture control system is maintaining the accuracy of gesture detection, particularly in dynamic environments. Factors like lighting conditions, the user's hand position relative to the camera, and subtle variations in hand shape or movement can significantly impact the precision of gesture recognition. Mediapipe’s hand landmark model, although highly accurate in ideal settings, may struggle with false positives or missed gestures when these factors are not optimal. To mitigate these issues, careful tuning of the gesture detection thresholds, as well as adjusting the minimum detection and tracking confidence, is essential. Additionally, ensuring that the hand remains within the camera's view and remains steady while making gestures helps to reduce inaccuracies in detecting subtle motions or fine distinctions between gestures.

Accomplishments that we're proud of

One of our biggest accomplishments was diving into the world of computer vision. Despite the steep learning curve, we were able to grasp key concepts and apply them to our project, particularly in areas like hand gesture recognition. Externally, we learned that computer vision can be significantly used for better purposes for the community instead of just being a tool for object recognition. Internally, It was challenging, but we embraced the opportunity to learn and grow. The collaboration and teamwork we fostered were essential to our success; everyone contributed their strengths, whether in problem-solving, coding, or brainstorming new ideas. Overall, we’re proud of how we turned these challenges into meaningful learning experiences, which ultimately contributed to the success of our project.

What we learned

At the start of our project, we encountered several challenges, such as difficulty in choosing the right topics since we realized that the project’s scope needed more clarity and focus, leading us to refine our approach and adjust as we gained new insights on computer vision. One of the key breakthroughs was discovering the potential of computer vision. Instead of just focusing on object recognition, we explored how this technology could be leveraged in more innovative ways, such as capturing hand gestures for better use in the healthcare system where it can be used as a motor skill practice. However, this proved to be particularly challenging due to the complexity of accurately recognizing and interpreting dynamic hand movements in various environments. Despite these difficulties, we found ways to overcome them, and this exploration ultimately enhanced the project, allowing us to build a more robust and engaging solution.

What's next for MarioLuigi

MarioLuigi has the potential to expand beyond its current scope by integrating into more gaming platforms and becoming a central figure in home automation systems. In gaming, it could branch into various genres, from action-adventure to RPGs and strategy games, offering players support, guidance, and interactive features. It could also expand across consoles, mobile, and PC platforms, with multiplayer or co-op functionalities. On the home automation front, MarioLuigi could evolve into a voice-controlled smart assistant, helping manage everything from lights and temperature to security and entertainment systems, while adding a fun, engaging personality. It could integrate seamlessly with IoT devices, turning it into a personalized hub for both gaming and everyday home tasks. The long-term vision could see MarioLuigi becoming not just a character in games but a versatile assistant in homes, offering emotional support, personalized experiences, and deeper interaction with the entire smart ecosystem.

Built With

• jupyternotebook
• mediapipe
• opencv
• python