RecallMe

RecallMe

Inspiration

We were inspired by the lack of resources available to help dementia patients remember their loved ones. It’s often difficult for family and friends to reintroduce themselves to the patient, so we’ve created an application to make this process easier!

What it does

The application has pages allocated for each of the family members that contains photos and audios of the respective member. Patients can listen to the recorded messages from their family members, reinforcing familiarity through voice recognition. The messages have a speech to text feature which allows the patient to read the messages. We also have a facial recognition that links the individual to their page. This is so the dementia patient themselves can use their camera to remember someone instead of that individual having to go through the struggle of re-introducing themselves.

How we built it

The application was built was accessibility in mind. We kept the UI simple as to not overwhelm the user and used pastel colors to maintain the theme.

For the face detection feature we are using the facial_recognition library in python in addition to FastAPI. We use this API to essentiallyt allow the React frontend to communicate with our facial recognition python script where the front end sends a Post request at a given URL sending the image that was taken and sending that to the python script to see if it was a recognized face. We have another python script which generates matrix encodings using a separate computer vision library, where the saved encodings are compared to the encoding of the picture that was just sent through the front end to see if there was a match.

We built the speech to text system by combining a Python backend with a React frontend. On the backend, we used Flask to create a REST API that leverages OpenAI’s Whisper model for transcribing audio files. The transcription logic is encapsulated in a Python function that processes the audio and returns the text as JSON. On the frontend, we built a React component that displays an image along with a button that plays the audio. When the audio is played, the component sends a POST request to the Flask API to fetch the transcription, which is then displayed to the user.

Challenges we ran into

One challenge was integrating the facial recognition algorithm into react. We initially experimented with the library by just creating a simple python app, but integrating into the frontend was much harder as we had to use the FastAPI in order to do so and set up that communication between the frontend and our script.

Accomplishments that we're proud of

We're proud to implementing all the features we planned to implement. We were initially intimidated by the facial recognition feature but thankfully everything works as expected.

What we learned

We've earned to implemented libraries such as face_recognition into the frontend. We also learned how to implement speech to text in react as we've never worked with something like this before.

What's next for RecallMe

Our future plans include integrating the application into IOS applications. It would be easier for the users to use the camera system via phone instead of computers.

Categories : 1 :Best Health/Wellness Hack 2: Best Accessibility Hack 3: Best AI/ML Immersion Hack