Cognisense

A conversational assistant that revitalizes memories, fosters deep emotional connections, and empowers Alzheimer’s and Dementia care with intelligent, personalized therapy.

Comment

Inspiration

Alzheimer's disease affects 5.8 million Americans, causing profound memory loss that impacts not only patients but also their loved ones and caretakers. Inspired by personal experience from one of our team members with a family member suffering from Alzheimer's, our team recognized the pressing need for an innovative solution to help patients reconnect with their memories and maintain crucial relationships.

What it Does

Cognisense is an AI-powered application that serves as a digital memory companion for Alzheimer's and dementia patients. It analyzes patient photo albums and facilitates interactive conversations about selected images. The AI is trained to engage patients like a therapist or caregiver would, gently guiding them to recall past experiences in a supportive, non-intrusive manner.

How We Built It

We developed Cognisense using a robust tech stack:

  • Frontend: Streamlit for a user-friendly interface
  • Backend: Python, containerized with Docker for consistent deployment
  • AI Integration: Google Cloud's Gemini-1.5-Flash and Gemini-Pro for advanced conversation and image processing
  • Speech Processing: Google Cloud Speech-to-Text and Text-to-Speech APIs for natural voice interactions
  • Data Storage: Google Cloud Firebase for secure storage of conversations, images, and metadata
  • Deployment: Dockerized container for scalable and consistent deployment managed by Terraform

Challenges We Ran Into

Our main hurdles included:

  • Orchestrating data flow, specifically real-time audio transcriptions, between Docker Container files, backend, Google Cloud, Google AI Models, and Streamlit
  • Implementing a seamless conversation pipeline within Streamlit's framework
  • Integrating real-time speech input from users
  • Maintaining conversation continuity across multiple exchanges
  • Optimizing the rendering process to prevent code breaks during extended conversations

Accomplishments That We're Proud Of

We are particularly proud of:

  • Successfully integrating cutting-edge AI technologies (Gemini and Google Cloud APIs) into a cohesive application that allows for multi-modal, back-and-forth conversations with large context bases
  • Developing a robust conversational pipeline that maintains context and fluidity
  • Creating an empathetic, user-centered design that prioritizes the dignity and autonomy of Alzheimer's patients, based on published research, contextualized memories of patients, and additional information
  • Building a solution with real-world impact potential, capable of improving the quality of life for patients and caregivers

What We Learned

This project provided invaluable insights into:

  • Full-stack development best practices
  • Advanced AI integration and API utilization using Google Cloud technologies, specifically with Gemini-Flash and Gemini-Pro, along with Google Cloud Speech-to-Text and Text-to-Speech APIs
  • User-centric design that is easy to use, especially for our target audience
  • The intersection of technology and healthcare
  • Collaborative problem-solving in a hackathon environment
  • Containerization using Docker and Terraform

What's Next for Cognisense

Our roadmap includes:

  • Developing a mobile version to increase accessibility for users
  • Fine-tuning the AI model with real caregiver conversations to enhance empathy and understanding, based on Gemini analytics on past user conversations
  • Exploring partnerships with healthcare providers and Alzheimer's research organizations

Cognisense aims to become an indispensable tool in Alzheimer's care, fostering connections between patients and their cherished memories while supporting caregivers in their crucial role.

Built With

Share this project:

Updates