We were interested to see if it would be possible to automatically evaluate whether LEGO figures have been correctly assembled by doing image recognition with deep neural networks. We imagined an app in which kids would be presented with figures to assemble, either by increasing complexity or with set time to memorize the figure.
What it does
Upon opening the app, a figure is presented. The participant attempts to replicate the figure with LEGO and snaps a picture. The app evaluates the picture with a neural network that we deployed online.
How we built it
We build the neural network trained on a homemade toy dataset with Keras on a Tensorflow backend. We built a classifier on top of a finetuned VGG19 architecture with pre-initialized ImageNet weights. This model is deployed as a Flask server on PythonAnywhere. The app is built for Android with Java and Android studio.
Challenges we ran into
Initial design of the neural network architecture was challenging. We were also hindered by having to create our own training data, though finetuning the ImageNet helped a bit in this. Our biggest challenge was deploying the model on Android which required quite a bit of configuration and debugging.
Accomplishments that we're proud of
We're proud that our model worked and that we got experience deploying neural nets on mobile and web. It has been a really great learning experience for us.
What we learned
Practical experience with Tensorflow and neural nets, deploying neural nets, process and teamwork on data science projects.
What's next for Lego Fit
With more data we'd like to replace the figure classifier with a neural net for creating LEGO structural embeddings. This way, random figures can be created and evaluated in-app.