Meg has an unhealthy relationship with emojis, and does the facial expression whenever she wants to send one. We realized that facial expression detection could be a cool way to quickly send emojis to your friends while on your laptop.
What it does
Laptop camera translates facial expressions into emojis. If it finds a facial expression that it can recognize, it matches it up with the appropriate emoji and sends it to one of your friends.
How we built it
We used the Clarifai API to train a model on different facial expressions, and built a macOS applications that sends keyboard strokes to the Messages app based on the output from Clarifai.
Challenges we ran into
- Training a prediction model to recognize small differences such as a wink was difficult, but we were able to tackle more distinct actions like :P - Emojis aren’t keyboard strokes, how do we send that info to the Messages app if it only has a private API? - How do we create an appropriately sized bitmap to create an image we can send to the Clarifai API?
Accomplishments that we're proud of
Had to do some low-level system calls stuff to send the appropriate data, never done that in application development before which was pretty cool. Learned what a bitmap was, and got to apply some machine learning concepts.
We also rewrote the Clarifai client for macOS, we’ll submit a pull request for that soon.
What we learned
How video data is actually represented, some of the application aspects of machine learning, how to do some low-level calls in Objective-C, how to do server-side work in Objective-C.
What's next for :P
The emoji recognition is cute, but it would be really cool if we could control our whole laptop session with gestures. We already implemented a dictation trigger, but it would awesome to train a model to recognize gestures to open up applications, navigate to the appropriate conversation in Messages, send a message, etc.