Holographic Emotional Recognition in Real Time

Using only openCV, ML and no other API to detect human emotions then project them in a DIY pyramid hologram

Comment

Inspiration:

All team members wanted to build an emotional recognition FROM SCRATCH (No prebuilt stuff, no apis like google cloud,etc). We planned to use OpenCV to process the webcam data in REAL TIME and to utilize Machine Learning as well as communicate with surrounding "hackers" to help providing us data (making happy/sad/angry faces). All of us were beginner with Unity, VR, and hologram so we wanted to pick up some new technologies building the project, too.

What it does

The system can detect and process human emotions in real time via a webcam utilizing openCV and machine learning. Then it sends json data to Unity which renders a 3d face that mimics the user's facial expression. Thanks to Unity, the 3d face is projected onto a pyramid hologram (a DIY and affordable hologram).

How we built it

We built our own emotional recognition program/algorithm from scratch. After hours of tinkering with openCV and machine, we got a training program that yields improving results. We then asked fellow hackers around us to help trained our ML program by making happy/sad/angry faces in front of the webcam. The program was then trained to better detect emotion. To deliver updated emotion data to Unity, we set up a web server using C#. As it records input, the Python script sends requests to this service, which in turn updates the values used by Unity to set the expression of the hologram face. We built the hologram face and shape keys using Blender then imported it in Unity.

Challenges we ran into

Unity sometime crashes and doesn't cooperate with what we expected. Computer Vision and Machine Learning require a good amount of dataset and long time to be trained properly.

Accomplishments that we're proud of:

A working and reasonably accurate emotion recognition system Real time data transferring between Python (CV, ML) and Unity through json+ c# webserver Affordable DIY hologram pyramid

What we learned

openCV, machine learning, statistics, communication skills with other people

What's next for Holographic Emotional Recognition in Real Time:

  • Added more supports for other complex emotions (fear, upset, ...)
  • Better hologram construction
  • Real time streaming of other image, VR objects
  • Instead of hologram, stream to other device like Oculus Rift, Gear VR thanks to Unity crossplatform support

Built With

Share this project:

Updates