Many of us find it difficult to react to other's emotions. The challenge of recognizing and responding to emotion is amplified for people with autistic spectrum disorder. For this reason, people with autism often isolate themselves, which can damage their mental health. We wanted to design an 'easy to use' device that acts as a portable companion, which helps people communicate in social situations.

What did I build .

We have designed a hat for people with autistic spectrum disorder, which recognises emotions and offer helpful advice to the wearer about how to respond. The pi camera at the front of the hat captures the face of the person who the wearer is talking to and their emotion is detected using Microsoft cognitive service. An audio input is connected to a raspberry pi on the top of the hat, which allows the wearer to receive information on the type of emotion the other person is experiencing. The audio also gives the wearer advice on how to respond to specific emotions.

How I built it

The project is a combination of software and hardware. It utilises a raspberry pi running python to fetch frames from a camera mounted at the front of hat. These frames are put through the Face API of Cognitive Services and emotions are ranked. The ranked emotions are then fed into a dialogue generating module that creates an appropriate social cue based on the emotion observed. The selected cue is then put through the text to speech API of cognitive services to return an audio file that can be played over the audio-jack/bluetooth.

Challenges I ran into

1. Terrible internet

  1. Authentication on Azure is quite convoluted. Multiple keys are required and the dashboard is not intuitive.
  2. Thread synchronisation issues
  3. Audio synchronisation issues
  4. Latency # 6. Terrible internet

Accomplishments that I'm proud of

It works, it's on a hat, it says some encouraging words, what's not to like!

What I learned

Details about threading in Python, interfacing the various components on a Pi, working with file streams across a variety of libraries, leveraging Cognitive Services using Python.

What's next for Dora the Fedora

Ideally an app to observe the results, perhaps expansion into feedback for conference speakers. Smaller boards so that the overall footprint of the device is reduced.

Built With

Share this project: