CLARA

CLARA is a virtual nursing companion that answers health questions, logs symptoms, tracks medications, and builds healthy habits—helping you stay safe and informed at home.

Comment

Inspiration

According to the Canadian National Seniors Council, by 2052, almost one quarter of Canadians could be of 65 years or older, with many preferring to live in their homes and stay in their communities. The result of this trend is a growing need from elders to have companions in their homes. This is where CLARA comes in as an at-home nursing assistant helping to keep track of health so that everyone can enjoy life to the fullest.

What it does

CLARA is a next-generation health companion with quantum encryption, blockchain integrity, and zero-knowledge privacy protection. Throughout the day, you can talk to the CLARA plush and note down any symptoms that you feel, while your smartwatch collects your health data and provides it to CLARA as well. All of your health information is securely stored while you can sign in to view all past daily logs through the CLARA website. You also have the opportunity to generate a report summarizing your logs from a timeframe of your choice. This report can be sent to your physicians and emergency contacts to provide an accurate representation of your health, ensuring that you get the best support possible.

How we built it

We started with Health data synced to our backend with Google Fit and a website to view logs and see updates in real life as they reflect onto MongoDB. Then, we integrated in a virtual assistant, meant to be an easy way for our users to add to and interact with transcripts. For this, we used Picovoice to listen for CLARA, the wake word for our assistant, which then feeds audio into Deepgram, which will transcribe further speech back into text for our intent classifier powered by BERT trained with few-shot learning to break sentences down into programmatic statement that can be acted upon by our Flask and MongoDB backend and logged into our Nextjs frontend. On our frontend, we display summarizes and notes taken by the assistant as well as health data collected by Google Fit/Apple Health to give the user a place to get a complete understanding of their health. Additionally, users can generate reports of historical data, which will be fed into Vellum and GPT 4o so that the LLM can summarize the data and provide useful insights for the customer.

To support the greatest variety of users and platforms, The main services of CLARA are dockerized and connected via websockets and REST APIs to focus on modularity and cross platform support.

Challenges we ran into

We had initially wanted to use Apple HealthKit for importing medical data considering that all of us used iOS instead of Android. However we quickly realized that Health data is a sensitive scope and therefore would require a paid developer account and approval from Apple even if we were fine side loading it via a mac for only a week. Given the time constants and budget constraints inherent to a hackathon, we deemed this to be infeasible and ended up using the Google Fit iOS app to migrate some Heathkit data into Google Fit and then worked using their API!

Accomplishments that we're proud of

We have a working real-time speech to text model with wake word recognition that can classify intent and respond to your question or log your symptoms accordingly. We have also incorporated Vellum to analyze health data and provide valuable insights through the reports.

What we learned

We learned how to handle asynchronous code, utilize API routes for Google Fit, as well as incorporating elements of branding and utilizing color schemes to ensure cohesion and improve UI.

What's next for CLARA

Next steps include a physician dashboard on the CLARA website, where a physician can click into the daily logs of each of their patients to review all of their health information for the most accurate understanding of each patient's health situation. We would also improve CLARA's response time, and making it generally more realistic and responsive as a companion to chat with. Finally, we would also like to develop a RAG to provide more accurate information when answering user questions.

Share this project:

Updates