Example Data Log
Data Creation Screen
Facial paralysis is a disfiguring condition that results in a number of aesthetic and functional sequelae, leading to severely negative emotional ramifications. The condition often creates significant facial asymmetry with loss of smile, loss of blink, as well as functional deficits that can include loss of vision. This project involves developing a novel way of objectively measuring facial movement using mobile applications and programming and investigating the outcomes of various surgical procedures in the management of facial paralysis.
What it does
A patient creates an account upon opening the application and signs in. Once they do this, they are prompted to either view the existing data generated or to capture a new photo (to generate new data). If they choose the latter route, the view of the camera screen itself is an optional feature exposed by clicking another button (as some patients are embarrassed when seeing their disfigured reflections). The user is guided to take a photo with maximum smiling, and then another with no expression. The app analyzes the level of deviation between various features on the face (eyebrow relative to center eye, oral commissure, etc.) and provides the patient with this data in graphical form. The ‘level of deviation’ values are presented on a scale of 100 pixels. This data can then be used to quantitatively assess levels of paralysis and improvement from past photos taken, optimizing the process of recovery rate analysis.
How we built it
We started by researching facial paralysis detection technology today. Using the latest research papers and analyzing industry leaders like Facegram, we selected a plan of action. We found one paper that served as a very firm basis for our mathematical model. Our strength was web development so we decided to build a progressive web app using React.js and Firebase. We then researched APIs and determined which APIs would give us the necessary information from the mathematical models in the research paper. We selected the Face++ API employed Material UI for our front-end UI.
Challenges we ran into
The facial recognition API was by far the most frustrating. It took me 6 hours to determine that the API did not take headers. With other issues like COORS, the API was very difficult to get working. Additionally, the process of converting a research paper into actual software was tricky. The language of the research paper was difficult for non-medical individuals as well as individuals who are not well versed with linear algebra. We had to learn a lot of unexpected mathematics and find ways around some of the solutions in the paper as well.
Accomplishments that we're proud of
I was really happy to see the mathematical model we abstracted from the research paperwork so well in the real world. After 6 hours of attempting and failing to get the Face++ API working, we finally figured out that it does not take any headers at around 4:30 AM. Though it was disappointing to know we had wasted so much time on a simple API call, it taught me a lot about how to debug API calls. Without the Face++ API, there would be no basis for our product so it was very rewarding too when it finally returned data.
What we learned
We learned a lot about AJAX, COORS, Firebase hosting and deployment, as well NoSql structure. Additionally, we really had to practice a lot of patience. There were many moments when we thought we could just build the project on our own time, but we persevered because we wanted to show people that even small tech can make a large impact on the world.
What's next for Amelius
We want to create a version of the application that allows practitioners to utilize the app in a clinical/hospital setting. This would organize the data in a patient-specific manner, allowing the physician to efficiently collect necessary data while seeing numerous patients one after another. We will also develop a push notification platform via text for these doctors to send patients reminders regarding when they should be taking photos to keep records updated between clinic/hospital visits.