Emotional-Cardiography (ECG)

Inspiration

The inspiration for ECG comes form the tremendous labor shortage Canada is currently having in the medical industry and the desire to make safety and security accessible to everyone whos staying in medical facilities.

What it does

ECG uses a ML model we trained and python OpenCV to be able to detect a humans emotions based of facial expressions. Then using a heart rate sensor hooked up to an Arduino to simulate a professional grade cardiac event monitor, we take in a patients bpm using C++. Then it sends both pieces of data to a web application made in JavaScript in real time using various libraries and techniques. Finally, the web application made using react creates a friendly user interface for the doctor to be able to see the probability of the patients possible emotions, the bpm and a suggestions section which gives the doctor recommendations based off our data.

How we built it

First we created a sequential model using 8 convolutional layers based on the fer2013 data set we found from Kaggle. Which includes about 24000 training images and 3000 validation images, after various transformations we end up having about 4.7 million parameters to train our model. Then after some optimization, we sent the model to our website by using the external flask library to return the live footage into an html file.

On the hardware side, we used an KY-039 heartbeat sensor and an Arduino to sense heartbeats, then we created an algorithm to approximate the average BPM of the last 4 seconds and used the serial monitor to send the information to JavaScript to be processed.

Challenges we ran into

One of the main challenges we ran into was the ability to transfer data in real time, whether that be from python to JavaScript, sending a live streamed from python to html, or sending C++ readings to JavaScript. Most of our time ended up being spent debugging various different transferring techniques we found. Another challenged we had was our model program was inconsistent with successfully initializing, one attempt it would work and the next after no syntax changes it would fail. Another issue we ran into in the OpenCV side was getting all the versions to line up in the virtual environment we created, eventually after enough pip installs we found out about poetry and used that to standardize all of our version control.

What's next for Emotional-Cardiography (ECG)

Being made in under 36 hours, there is of course a lot more to be done with ECG and we have plenty of ideas of what we could add. Some updates we could make to the software are: convert to an actual private desktop app rather than website, develop model that can detect higher number of emotions, have a larger data set & more layers, develop a better back end that sends the data more securely and faster, and finally a better front end that has a nicer UI for the doctor to use. On the hardware side we could add more sensors so that we get more data on how the patient is feeling, ie a tempeture sensor.