We are inspired by Google's Quick Draw AI experiment (https://quickdraw.withgoogle.com) to design the first neural network handheld game console. This console runs TF2 trained neural network on the chip, and there is no internet connectivity required. We have tried to have the game rules as close as possible to Google's Quick Draw game.
What it does
Upon starting, the console shows a random keyword out of 100 categories and the user has 5 seconds to think and 20 seconds to draw the keyword on the screen. After 20 seconds system converts the user's drawing to an image and feeds it to the neural network. If the drawing is recognized as one of the top five output probabilities, then the system considers it as a correct drawing for the keyword. STM32F429 ARM microcontroller is used as the main processor. It has 2 Mbytes of Flash memory and 256 Kbytes of RAM. Our neural network only uses 1.42MB of flash and needs 103KBytes of RAM for inference.
How I built it
We trained Convolutional Neural Network with 100 categories from Google`s doodle drawings dataset using TensoFlow 2.0. Our network accuracy for TOP5 category was 95%.
Challenges we ran into
Creating The GUI In the board for the game. Training a neural network with good accuracy and low RAM and Flash requirement(Small amount of layers).
Accomplishments that I'm proud of
Successfully running TF2 trained network on STM32F4 using STM32CUBEMX.AI despite the small amount of documentation and examples available. TF2 is not officially supported by STM32CUBEMX.AI.
What I learned
How to use tensoflow2.0 with Keras. Train different networks from DNN, CNN to LSTM and combination of them using Tensoflow2.0. Integrate TF2 trained networks to STM32 microcontrollers using STM32CUBEMX.AI.
What's next for Quick, Draw! Handheld Game Console
Increase the number of categories which the system can detect. Add multiplayer feature to the game console.