Butterfly Park

• 

  Move the hand infront of the camera to control the virtual hand

• 

  Butterfly Park scene

• 

  Pipeline of the project

• 

  Palm Detector with keypoints

Inspiration

The Butterfly Park game is inspired from the Augmented Reality. The perception pipeline is inspired from mediapipe framework, in which multiple ML models are pipelined together to get the best of ML application.

What it does

In this project a virtual hand is controlled by moving the real hand in front of the camera. By clicking on the butterfly button a flower is created on the hand which attracts the butterfly towards the hand. We can now move the virtual hand as a real hand movement.

How I built it

I built the game in Unity3D. The palm detector and 7 keypoints are detected and mapped to the virtual hand in the game scene. The perception framework is built on the mediapipe pretrained model. The model is converted to pytorch framework by defining the architecture and copying the weights from the pretrained mediapipe model. Later this model is conveted to ONNX format to run in the Unity3D game engines Barracuda Inference pipeline.
The pytorch model and the postprocessing - NMS suppresion are all converted to ONNX format for easier deployment in the Unity3D.

Mediapipe model -> Pytorch model -> ONNX model -> Barracuda Inference -> Unity Gameplay rendering.

Challenges I ran into

The ONNX format models (palm detector and postprocessing) ran fine on the python end but the Barracuda inference in the Unity3D engine did not support some of the operators in the ONNX model. I tried to fix the issue but did not workout.

Accomplishments that I'm proud of

Through this project, I learned how a ML model is deployed end-to-end in an mobile environment. I was able to convert a model from tflite to pytorch all by studying the underlying architecture of the model. My goal is to have an Augmented reality butterfly park. This current project is a good baseline for further devlopment.

What I learned

To convert tflite model to pytorch just by looking at the model architecture. To convert pytorch model to ONNX. To integrate pytorch on Unity3D game engine for mobile game deployment.

What's next for Butterfly Park

My goal is to have an Augmented reality butterfly park. This current project is a good baseline for further devlopment. Currently I am using only a palm detector model, I will continue to use this to do land mark localization of the fingers in 3D and build an Augmented Reality Butterfly Park.

Built With

• onnx
• pytorch
• unity