Hospitalization constitutes an unpleasant experience for children who suddenly have to leave the familiar place of their home and the persons who are important for them, and stop their favorite activities include play.

What it does

To solve this situation, we combine chat program and puppet play for hospitalized children. They can speak and share their emotion with other friends through their virtual puppet playing.

How we built it

We take a picture of a user's hand and send it to the python flask server. In the server, we classify the image into three different labels using PyTorch model and send the output back to the playground web page. In this way, user can chat with other people with puppet playing without revealing their private information like their face.

Challenges we ran into

Much of our time was spent collecting data and training the PyTorch model. Also instead of using simple RESTful server, we use websocket to speed up sending images and receiving output to/from the python flask server.

Accomplishments that we're proud of

Using websocket library, we can greatly improve performance. Also we can make our own customized computer vision model using PyTorch and AlexNet.

What we learned

PyTorch is a great framework to build computer vision model. It was super easy to download Alexnet and slightly modify the model to classify shapes of hands.

What's next for Chatty-Puppets

We want to improve the frame rate of the output and add more characters which children love. Additionally, to prevent inappropriate and/or insulting words, we want to use speech recognition function to detect and filter these words.

Share this project: