Many of our members have worked with children with autism, either personally or through school-related projects. We quickly noticed that the teaching resources for these children, especially when it came to teaching emotions, were wildly outdated and inefficient. For a condition that applies to over 1.5 million children in the US, this is unacceptable. We created Copycat with the hopes of providing a more fun and engaging social-emotional learning experience.

What it does

Copycat is a mobile app that teaches the user how to recognition and replicate basic emotions. First, the user is prompted with an image of a person and identifies the emotion displayed in the image. Then, they hold up their phone and snap a picture of them doing the same emotion. This process repeats with new images until the time runs out. At the end, an AI algorithm analyzes the accuracy of the user’s facial expressions and outputs a score. The goal of the game is to recognize and accurately copy as many of the emotions as possible.

How we built it

  1. Conducted research on the background area and identified areas for improvement to ideate features. This included reading literature in the field and learning about user stories.
  2. Designed low and high-fidelity mockups in Figma before coding our user interface using React Native.
  3. Devised a logo and visual identity in Procreate and Adobe Illustrator for the app frontend.
  4. Applied a facial emotion recognition machine learning model to analyze the emotions of photos submitted by the user.
  5. Designed a cloud computing architecture for the backend. In specific, we utilized AWS services like Lambda and Elastic File Store to apply algorithms and run our model. We then used S3 and API Gateway to connect the backend to our app.

Challenges we ran into

  • Emulator and the webcam: the emotion recognition model runs very well on our computer. However, the emulator is not compatible with the laptop’s webcam, so we had to use Expo Go on our iOS device. However, this meant that our model was unable to run locally on the mobile device. Our solution was to move to cloud!
  • Limitations of Cloud: moving to cloud was not as simple as we were prepared for. The first problem we ran into was the serverless design of Lambda. We had to shift to do all our processing to occur in one large batch at the end because the time it took to import the heavy models negatively impacted the user experience. Despite this, we learned a lot about cloud architecture, and we would design a better one in the future.
  • Uploading images to S3: after implementing it 5+ different ways, we were never successful in creating a smooth pipeline between the phone camera and cloud storage. We tried sending the binary file itself, to converting to base64 and sending through an API, to using AWS SDK itself. This was the broken link that tripped us up for a long time.
  • Balancing friendly atmosphere while having a good user experience: we thought that user experience was especially important, especially for a game catered towards children.

Accomplishments that we're proud of

  • Great teamwork- we were able to delegate and take ownership of different tasks quickly
  • Quick ability to adapt- despite facing derailing challenges (like not having a webcam for a very computer vision heavy app), we found ways to get around it, even if they were outside of our comfort zone
  • Artwork and visuals were pretty aesthetic :)

What we learned

  • Got more familiar with front end, design, APIs.
  • Learned about various Procreate integrations
  • Cloud computing, React Native, and working with images.
  • There's a lot of cool people here to mentor us!

What's next for Copycat

  • Unlimited game content generation: we apply the same emotion recognition algorithm on images from the web to generate infinite content
  • Less waiting time: move Lambda processes to EC2 = shorter wait time for the user.
  • Broaden: Target more complex emotions (such as guilt, uncertainty, confusion, etc.)
  • Complexity: Tell a story, explain why people feel certain emotions in certain situations
  • Game stack: add different games like a memory match or real-time emotion recognition
+ 7 more
Share this project: