Mosaic

• 

  Info on a person being displayed in the corner of their screen. (Zoomed in)

Inspiration

One of our team members watched their grandparent slowly lose the ability to recognize the people they loved most. We built Mosaic because we want to provide a solution that helps others. What it does Mosaic serves as the memory layer for people with dementia. It recognizes faces of people who the person talked to and records conversations. These conversations are summarized into a 3-4 sentence briefing that captures the most important portions of previous conversations. When users talk to people they talked to before, our application recognizes those faces and displays the briefing so the user can quickly recognize what they like to talk about.

How we built it

For the frontend, our application uses these technologies: React as our frontend framework Typescript as our primary language Tauri to create the desktop application For the backend, our application uses these technologies Fastapi as our api framework Python as our primary language OpenCV to connect to the webcam Openai-Whisper to do voice to text transcription Face_detection to do face recognition For our database we used Mongodb

Challenges we ran into

There was several errors along the way, however the most pressing of which was one regarding memory usage. While running face recognition and audio transcription, we ran into issues where the transcription wasn't being written to a log file that we used to save to the database. This took several hours of debugging before figuring out that it was due to the face recognition taking up all the cpu resources. We solved this by moving this to a separate python process so it wouldn't block my audio transcriptions.

Accomplishments that we're proud of

We're proud that this is a smooth and functioning application that uses AI. It's very nifty, and desecrate as it runs in the background and allows for users to perform other activates as it silently and quickly updates based on the people recorded.

What we learned

We each learned something different. Colin learned how to create API hooks, learned better web-development, web structuring, how to be better and python, type-script, and using AI in practical applications. Vincent learned: how to work with openCV and face detection

What's next for Mosaic

We plan to clean this project up and make it look more presentable. One nice feature would be to have it work over meetings since currently, it only works on the user’s webcam. We may add new features along the way however nothing is set in stone.

Built With

• python
• rust
• typescript