Serverless Hands-free Allergy Checker

Saves and checks drug allergy information using face recognition to prevent anaphylactic shocks

Comment 3

Inspiration

I was shocked when I learned that drug allergies (caused by medication like aspirin, penicillin, ibuprofen, etc) can cause fatal anaphylactic shocks, especially in the children while I was searching the Internet about the grass allergy of our twins. Then I thought that we could use the face recognition technology in emergency services to check whether a patient has a drug allergy quickly using voice recognition.

What it does

The system saves the allergy information with the picture of the patient and this information can be retrieved by taking another picture using face recognition later. Also, it provides a quick fully hands-free interface for the personnel (doctors, etc) for quick and safe login operation and other patient management functions using voice recognition, letting them use their hands for their job while using the system.

How I built it

I have used Rekognition for indexing and searching faces of the doctors to let them log in with their face. Also, OTP codes are extracted using Rekognition text detection. Again, the faces of the patients are indexed to search allergy information by their face.

The voice interface is implemented using Amazon Lex with 4 intents for adding patients, checking patients, taking pictures and logging out.

Amazon Polly is used for talking to the users and S3 is used for storing the pictures of the users and patients. Allergy information is saved to a DynamoDB table.

In addition to Amazon SES for sending emails for OTP codes, Amazon Cognito is used for authentication.

AWS Lambda is used as a trigger to index the face of the patients when a record is added to the DynamoDB table. Also, Cognito trigger Lambda functions is used for generating an OTP code and sending to the email address of the personnel.

Challenges I ran into

I have been already familiar to the most of the AWS services but it took some time to learn Cognito custom authentication functionality to generate a custom authentication flow for sending OTP codes to the user and to bind Lambda triggers for it. I thought that creating an installation and uninstallation script for the users of the code would be very helpful. It turned out that writing the scripts and testing many times was a challenge.

Accomplishments that I'm proud of

I am proud of building the whole project. From coding front end in Javascript to providing triggers to tables and to generating a custom authentication. I am proud of finding an idea about providing a quick hands-free interface to the emergency services personnel. Also, I am proud of the social responsibility side of the project that it would decrease anaphylactic shocks in situations like a medicine given to the children in emergency services.

What I learned

I have learned to enable a custom authentication flow for Cognito. Also, this was my first hackathon which taught to me running in a quick time frame for binding various AWS services. I have improved my Bash scripting language skills. In addition, I have learned that if you try to do something for the good of the people you can find the power to overcome the challenges.

What's next for Serverless Hands-free Allergy Checker

The Lex bot can be improved for more options for the name of allergens and patient names. Also, for simplicity audio is recorded for 4 seconds and if is not fully silence sent to Lex for processing. It can be improved by using silence detection continuously and stopping only when the silence is detected.

Built With

Share this project:

Updates