💡 Inspiration

We really enjoyed Spy x Family and hence the name.

Creating a system of controls for a game, bypassing the use of expensive accelerometer based tracking systems, making it more accessible, by attempting to recreate augmented reality on a budget.

Exploring the option of using pre-existing models to accomplish tasks beyond its intended use, circumventing the need to invest time and resources into training multiple separate models for each specific control scheme.

Manipulating objects and exploring a 3D space with the use of gesture controls rather than traditional mouse and keyboard inputs. With the rate of improvement of existing pre-trained models, more complex and accurate gestures can be coded.

🔍 What it does

This game attempts to improve the fine motor skills of both children and the elderly. Our primary target audience are “iPad kids” who have gotten so used to swiping and other gestures are missing out on fine motor skills. Fine motor skills are required for handwriting, which is why it is a problem when these kids go to school. Through our interactive game, they will use their hand gestures to navigate through the story while honing their fine motor skills. Similarly, the elderly would benefit from this as refining fine motor skills would mean that they can be independent.

🔧 How we built it

Tech we used:

  • vanilla js
  • Three.js
  • Mediapipe Hands
    • for handtracking
  • Blender to create 3D models

🏃‍♂️ Challenges we ran into

  • Emily, Johannine: new to Three.js
  • Not easy to envision 3D space with three.js, even with experience. We were also quite new to three.js and needed to learn more about configurations and functions available to create more complex interactions and scenes.
  • New to using the Media Pipe hand tracking model. While it boasts about having more precise points tracking, it does not perform gesture recognition like handtrack.js
    • To be able to achieve different types of gesture recognition without having to retrain multiple models, we had to be creative in figuring out how to code the thresholds and triggers to detect gesture, position and movement.

🏅 Accomplishments that we're proud of

  • we were able to complete and connect our individual components into a complete game
  • we made use of hand tracking for game controls
  • we were able to prioritize and implement design features.
  • That we survived 24 hours of hacking in person for the first time.

🧠 What we learned

  • hand tracking with tfjs pre trained models for the web.
    • We learned how to make use of hand tracking models and extract data from the outputs to use as controls for our own software.
  • three.js
    • Learned how to animate movement and leverage the three dimensions to create new kinds of games for the web.
  • blender
    • Learned how to create many models in a short amount of time, creating props, scenery and characters for this hackathon.

⏭️ What's next for Spy X Penguin

  • Fine tuning our UI, improving on the 2D user interface and providing more interactive tutorials on how to run the games.
  • Make the game more interactive
    • Refining our game mechanics to make interactions smoother and more intuitive
    • Finding better workarounds for the limitations of a computer vision based model, for example fast movements often would not be detected accurately due to motion blur obscuring the images
    • Expanding the range of gestures we can detect and improving the algorithms used to segment different poses
    • increase the complexity of our game mechanics
    • add more music, sound effects etc
    • add more games

Built With

Share this project: