Patient safety refers to the prevention of harm in the course of receiving healthcare services. Interest in this issue has been growing, and the vast majority of the interest has focused on physical health care than mental. However, findings based on physical healthcare can’t be entirely applied to mental health patients due to the different challenges presented in two specialized areas. Our teammates explored few common adverse events that occur within each mental and physical healthcare facility. We found that different patient safety issues are likely to occur in different settings and realized that patient safety in mental health received less attention from researchers perhaps due to the lack of interest or due to the difficulty to perform structural studies. We chose to make contribute in improving the patient safety in mental health. Major academic review papers point out few prominent threats such as seclusion, reduced capacity for self-advocacy, and self-harming. The authors suggest systematic approach to embed a culture of safety within a healthcare facility that includes effective communication during transitions of care and encourage patients . Our team envision more progressive approach that utilizes Internet of Things (IoT) to create an interactive platform for patient and caregiver and set up a preventive solution by monitoring the mental status of the patient in real-time.

What it does

Continuously, it takes photo, detect face, analyze facial expression, evaluate how patient feels, and notify/warn caregiver if any sign of threat to the safety of the patient is detected.

Note: photo is not saved in anywhere for privacy reason, and the access to the camera is not allowed from outside of the system. The image never leaves the device, only data since we finish E

Important: We took a webcam from our computer and mounted on helmet for the sake of the demo without any provided optical hardware. The software works the same way regardless of the type of camera.

How we built it

(Data Transfer) The real-time images from the camera is sent to linux-based Qualcomm processor in the helmet by one frame per second. Processor analyzes the facial expression and send the result to the mobile application via wifi.

(Image Processing) Qualcomm Snapdragon Processor runs a python script that detect human face attributes making machine learning-based predictions and draw rectangle on the face. The recognized face features are used to determine how the user is feeling.

(IoT) A small portable computing processor like Qualcomm Dragonboard simplifies the data transfer and processing. We were able to bypass several steps such as building backend server and implementing connection between node.js webserver and python algorithm calculator. Instead a computing unit directly received raw data from a webcam that lacks data transfer function to the server by itself. The processor analyzes it, compresses the result, and sends it directly to the mobile application. Building this network of things speeds up and simplify the entire process.

(Front-end) Built front-end using react-native framework that receives data from the Qualcomm Snapdragon Processor through WiFi via Firebase database platform and rendered it on iOS platform.

Challenges we ran into

No previous experience with IoT. We had to start learning and building from scratch.

Traditional Mental Health care already keeps track of the patient’s mental health. So, how is our solution different? Those measurements (i.e beck hopelessness scale) are based on a set of multiple choice questions. This inspired us to build our emotion analysis real-time and “non-invasive" based on monitoring than asking questions.

Regarding the privacy issues of the patients. HIPAA (Health Insurance Portability and Accountability Act) protects the security and privacy of all medical records and other health information used or shared in ANY form. Therefore, recordings of the patients have to be used and stored safely without risk of getting hacked. Also, a camera will follow patients throughout day so patients will be required to give a consent before being recorded.

Accomplishments that we're proud of

We grew up as a team by exchanging lunatic, unreal ideas that can bring innovations when realized.

What we learned

Coming from different backgrounds, each team member had different interests and expertise. Whenever we faced a problem, we offered several unique approaches to the problems. We could broaden our horizon and learn new skills through discussions. Our achievement after 36 hours of intense cooperation was beyond our expectation. Medhacks was such a valuable experience where we learned the power of cooperation.

What's next for F4cy Motion

We are often faced with a question “Please scale your pain from 1 to 10.” What is the correct answer for this question? What number should we say? If the number is too low, the doctor might not understand how much it hurts. If the number is too high, the doctor might think I’m exaggerating. Pain is subjective similar to feelings. Our next goal is to incorporate our system to switch pain into a quantitative date, so others can interpret the pain more objectively, thereby patients can get more appropriate and effective treatments.

Other than facial expressions, there are gestures that should be considered. The face detection is not at the finest when the patient puts his hand around his face. However, putting a hand on his face can also be a sign of frustration, surprise, etc. Also, in addition to the facial expression, heart rate could be used to detect change in people’s state.

Each person has different “default” face. That is, some people’s neutral face might be seen as non-neutral. Therefore, we can incorporate machine learning to set more appropriate neutral face.

Share this project: