While the entire world is fighting Covid-19, everybody is struggling to work from home in one or the other way. Especially families who are handling their kids/babies along with their hectic jobs. This scenario inspired me to find a way to monitor our Babies/toddlers while we are busy working or handling calls and take automated actions with the help of deep learning library PyTorch and Raspberry-pi to calm crying babies while we excuse us from every one and come to their rescue.
What it does
It classifies baby as happy or crying and if crying plays Tom and Jerry video to make them smile.
How I built it
- Data collection and prototyping in Google Colab
- Training on Azure Machine Learning
- Deployment on Raspberry-Pi 4 and as Web App on Azure Containers(Both are independent)
- Taking Webcam stream as input to predict Baby mood and play an audio file.
- Audio file acts as a voice command to Google Home to control the TV for playing Tom and Jerry on YouTube.
In this project I downloaded crying babies and happy babies from google images and quickly tested them in google colab for initial baseline. Check my Google colab notebook.
Training on Azure Machine Learning:
Deployment on Raspberry-Pi 4
For Raspberry Pi 4
Installing Pytorch and fastai on ARMv7l Architecture was a challenge. This was the toughest part of all the steps. Understanding compute requirements for doing inference on Pi using Pytorch and fastai was tedious.
For the Web App
We spin Azure web container instance from local conda environment. Included all the bottlenecks in the blog for easy reference.
Taking Webcam stream as input to predict Baby mood and play an audio file
We used IP Webcam app in the absence of a Raspberry Pi Camera and captured live stream every 5 sec for predicting the mood of a baby.
Web App - Consuming the web service from front end deployed as a Python flask Azure Web app
Finally, to consume the web-service we wrote a flask app with html and java script front end and deployed it as Azure Web App using local git and Kudu(Kudu is the engine behind git deployments in Azure Web Sites).
Challenges I ran into
-Understanding deployment architecture on Raspberry Pi while meeting the compute requirements for implementing PyTorch inference on ARMv7l.
-We were not having a USB webcam for Raspberry Pi so we used IP Webcam app to stream video from our android phone cam. To Fix this we lost time.
Accomplishments that I'm proud of
End-to end deployment of Pytorch deep learning model on Azure Cloud and Raspberry Pi was a major accomplishment for us.
What I learned
Raspberry Pi, Docker, Containers, Deploying web services, Python Flask web app development
What's next for Baby Vibes
Integration with Google Voice Search/Google Home for automatically playing Tom and Jerry/baby's favorite cartoon on predicting baby crying without playing an audio file.
When my 4 year old was a baby she used to stop crying on watching Tom and Jerry. As the world is fighting COVID-19, We thought Tom and Jerry can help us bring some smiles and try making babies stop crying for a few minutes while we pay attention to them and come to their rescue.
Github link -