-
An example image Jibo and recognize
Inspiration
Jibo is cute, and has cameras. What better game to play than pictionary? This game is an exercise both in the interactive capabilities as well as it's cameras.
What it does
First, ask Jibo to take a guess your drawing. It will take a picture and send it off to be classified by our custom-trained tensor flow neural networks. The first step uses openCV to detect the playing area within the board and crop off the rest of the image, and make it black and white. The image then gets passed off the our tensorflow model to be classified. We build trained neural network using hundreds of images pulled automatically from Bing Image Search. We used the "line drawing" category to limit it to pictures that look more like something drawn a whiteboard. We then supplemented the training data with a few examples of our own hand-drawn pictures. This network gives us a likelyhood for each possible label, which is sent back to Jibo. Then Jibo will go through these items one by one, listening for a "yes" or "no" from the user. There is an adorable celebration and frustration animation that plays when it gets the answer right or wrong. This continues until it exhausts its entire vocabulary, at which point it insults your artwork in protest!
Challenges we ran into
The hardest part was learning how to use the jibo SDK. We are arguably the first people outside of private Jibo events to use the robot, so there was a steep learning curve, and we had lots of feedback for the Jibo team. We also don't have good experience with Node and javascript, so that slowed us down a lot.
Accomplishments that we're proud of
This was our first time using Tensorflow, and Jibo. We were able to use transfer learning on the existing Tensorflow Inception image classification model.
Log in or sign up for Devpost to join the conversation.