bananerie | #64

• 

  TensorFlow model training

• 

  Connecting TensorFlow object detection with Unity

• 

  Hand-labeled dataset

bananerie was designed as a fun augmented reality game to help users incentivize healthy produce in their daily routines. The ultimate goal of bananerie is to overlay fruits and vegetables in the real world with fun shapes and images in AR and award the user with points for collecting more amounts & varieties of healthy produce. We began by capturing over 300 images of five types of fruits and vegetables that a user might encounter in daily life (apples, oranges, cucumbers, kabocha, and bananas, hence the name). Next, we hand-labeled the dataset in Pascal VOC format using the Python LabelImg program. Using a custom TensorFlow object detection model based on SSD MobileNet V2 Coco, we trained during a period of about 8 hours on a GPU in Google Colab Pro. We achieved a loss of about 2.55 after 13100 iterations (down from approximately 26 during the first training step, and not bad considering the small dataset size). We spent the remainder of the hackathon attempting to connect the model to Unity with the TensorFlowSharp plugin, which would have enabled us to build a platform-independent augmented reality game. Unfortunately, insurmountable version and other tech issues precluded the complete development of the project, but we developed a fully trained object detection model and learned a lot! We hope to resolve some of the Unity version issues in the future--maybe HoyaHacks 2023!

Built With

• arcore
• arfoundation
• arkit
• colab
• python
• tensorflow
• unity