Inspiration
Hospitalization constitutes an unpleasant experience for children who suddenly have to leave the familiar place of their home and the persons who are important for them, and stop their favorite activities include play.
What it does
To solve this situation, we combine chat program and puppet play for hospitalized children. They can speak and share their emotion with other friends through their virtual puppet playing.
How we built it
We take a picture of a user's hand and send it to the python flask server. In the server, we classify the image into three different labels using PyTorch model and send the output back to the playground web page. In this way, user can chat with other people with puppet playing without revealing their private information like their face.
Challenges we ran into
Much of our time was spent collecting data and training the PyTorch model. Also instead of using simple RESTful server, we use websocket to speed up sending images and receiving output to/from the python flask server.
Accomplishments that we're proud of
Using websocket library, we can greatly improve performance. Also we can make our own customized computer vision model using PyTorch and AlexNet.
What we learned
PyTorch is a great framework to build computer vision model. It was super easy to download Alexnet and slightly modify the model to classify shapes of hands.
What's next for Chatty-Puppets
We want to improve the frame rate of the output and add more characters which children love. Additionally, to prevent inappropriate and/or insulting words, we want to use speech recognition function to detect and filter these words.
Log in or sign up for Devpost to join the conversation.