CogniConnect

Smart memory companion that tracks cognitive patterns and preserves precious moments for Alzheimer's patients using advanced AI.

Comment

Inspiration

Watching family members struggle with Alzheimer's disease is heartbreaking. The gradual loss of precious memories, the inability to recognize loved ones, and the frustration of cognitive decline inspired us to build something meaningful. We wanted to create a tool that not only preserves memories but also actively monitors cognitive health through AI-powered assessments.

According to the WHO, over 55 million people worldwide live with dementia, and this number is projected to reach 139 million by 2050. Current early detection methods are expensive, invasive, and inaccessible to most families. We envisioned an AI system that could detect cognitive decline patterns through everyday interactions while simultaneously helping patients maintain their identity and connections.

What it does

CogniConnect is a full-stack AI-powered platform that combines memory preservation with cognitive health monitoring through three core modules:

1. Intelligent Video Memory Processing

  • Automatically analyzes uploaded video journals using GPT-4 Vision API
  • Extracts and transcribes audio for semantic search capabilities
  • Detects and recognizes faces using dlib-based facial recognition (128-dimensional face encodings)
  • Generates contextual Q&A pairs for cognitive assessment

2. Multi-Modal Cognitive Assessment System

Our platform implements three scientifically-backed cognitive tests:

  • Face Recognition Assessment - Measures facial recognition capability with scoring formula: $$\text{Score} = \frac{\text{correct} - \text{incorrect}}{\text{total}} \times 100$$ Tracks recognition accuracy over time as a biomarker for cognitive decline.

  • Name-Face Association Test - Evaluates semantic memory through interactive matching with dynamic distractor generation for consistent difficulty levels.

  • Contextual Memory Recall - AI-powered answer verification using GPT-4's semantic understanding. Individual question scoring \(q_i \in [0, 10]\) with aggregate score: $$\text{Activity Score} = \frac{\sum_{i=1}^{n} q_i}{n} \times 10$$

Composite Cognitive Score: $$\text{Final Score} = \frac{\text{Activity}_1 + \text{Activity}_2 + \text{Activity}_3}{3}$$

3. Cognitive Decline Detection Dashboard

  • Time-series performance tracking to identify declining trends
  • Week-over-week analysis with statistical visualization
  • Early warning system for caregivers when scores drop below baseline thresholds
  • Pattern recognition in quiz performance for early intervention

How we built it

Backend Architecture:

  • Python 3.13 + Flask 3.x RESTful API
  • OpenAI GPT-4 Vision API for intelligent video content analysis
  • face_recognition library (dlib) for facial encoding and recognition with 128-dimensional embeddings
  • MoviePy 2.x for video chunking and processing
  • OpenCV with ORB keypoint detection for unique frame extraction
  • SpeechRecognition for audio-to-text transcription
  • JSON-based storage for quiz data, performance metrics, and video metadata

Frontend Architecture:

  • Next.js 14.0.3 with App Router and React 18.2.0
  • TypeScript for type safety across the entire codebase
  • Tailwind CSS 3.3.0 with custom theming system
  • Chart.js 4.4.0 for cognitive performance visualization
  • Radix UI + shadcn/ui for accessible component library

AI/ML Pipeline: Video Upload → Chunking (10s segments) → Frame Extraction (ORB) → Face Detection → Audio Transcription → GPT-4 Vision Analysis → Q&A Generation → Performance Tracking → Trend Analysis

Video Processing Algorithm:

  1. Split videos into \(n\) chunks where \(t_{\text{chunk}} = 10\) seconds
  2. Extract frames using ORB keypoint detection (\(k > 30\) keypoints threshold)
  3. Compute face encodings: \(\mathbf{e} \in \mathbb{R}^{128}\)
  4. Match faces using Euclidean distance: \(d(\mathbf{e}_1, \mathbf{e}_2) < \tau\) where \(\tau = 0.6\)
  5. Generate embeddings for semantic search using OpenAI's text-embedding model

Challenges we ran into

Face Recognition Accuracy
Initial face recognition had high false positive rates due to varying lighting conditions and angles in home videos. We solved this by implementing ORB keypoint filtering to select only high-quality frames, tuning the distance threshold \(\tau\) through cross-validation, and adding a confidence scoring system for face matches.

Video Processing Performance
Processing long videos (>5 minutes) caused memory overflow and slow response times. Our optimization strategy included chunking videos into 10-second segments for parallel processing, implementing frame deduplication using ORB keypoint similarity, and reducing GPT-4 Vision API calls by batching frame analysis.

Quiz Scoring Fairness
Early quiz scoring heavily penalized incorrect guesses, discouraging participation. We redesigned the algorithm to balance correct/incorrect ratios in Activity 1, implement AI-powered semantic matching for Activity 3 answers, and weight recent performance more heavily than historical data.

Real-time Performance Tracking
Calculating cognitive decline trends required handling irregular quiz intervals and missing data. We implemented week-based aggregation with interpolation for missing weeks, moving average smoothing: \(\bar{x}t = \frac{1}{w}\sum{i=0}^{w-1} x_{t-i}\) where \(w = 3\) weeks, and statistical outlier detection using z-scores: \(z = \frac{x - \mu}{\sigma}\).

Dark Mode Consistency
Ensuring theme consistency across 15+ pages and 50+ components required CSS variable architecture with hsl() color space, next-themes integration for system preference detection, and comprehensive testing across all UI states.

Accomplishments that we're proud of

  • ✅ Fully functional end-to-end platform with 7 major features deployed and tested
  • ✅ AI-powered cognitive assessment combining 3 scientifically-backed test modalities
  • ✅ Real-time face recognition achieving >85% accuracy on diverse video content
  • ✅ Intelligent video analysis generating contextually relevant questions automatically
  • ✅ Performance tracking system detecting cognitive patterns over time
  • ✅ Responsive, accessible UI with complete dark mode support
  • ✅ Semantic memory search enabling natural language queries across video libraries

What we learned

Technical Skills:

  • GPT-4 Vision's capabilities and limitations in analyzing personal video content
  • Face recognition engineering - from encoding generation to similarity matching algorithms
  • Video processing optimization techniques - balancing quality versus performance
  • TypeScript advanced patterns for type-safe React component design
  • Next.js App Router architecture for scalable full-stack applications

Healthcare Domain Knowledge:

  • Cognitive assessment methodologies used in clinical Alzheimer's diagnosis
  • The importance of longitudinal data in detecting early cognitive decline
  • How semantic memory, episodic memory, and facial recognition degrade differently in Alzheimer's progression
  • Ethical considerations in building AI-powered healthcare tools for vulnerable populations

User Experience Insights:

  • Elderly users need large, clear interfaces with high contrast ratios
  • Caregiver dashboards must balance detailed metrics with digestibility
  • Gamification increases engagement but must avoid patronizing users
  • Accessibility is non-negotiable - WCAG 2.1 compliance is essential for healthcare applications

What's next for CogniConnect

Immediate Enhancements:

  • Baseline cognitive profiling - Establish individual cognitive baselines for more accurate decline detection
  • Machine learning model - Train LSTM networks on performance time-series: \(h_t = \text{LSTM}(x_t, h_{t-1})\) to predict cognitive decline trajectories
  • Voice biomarker analysis - Extract acoustic features (pitch variance, speech rate, pause patterns) as early Alzheimer's indicators
  • Multimodal data fusion - Combine quiz scores, voice patterns, and video engagement metrics for comprehensive risk assessment

Advanced AI Features:

  • Emotion detection in videos using facial expression recognition
  • Object recognition for richer memory context and semantic search
  • Conversational AI - ChatGPT-style memory chat for natural interaction
  • Predictive analytics - Forecast cognitive decline curves using ensemble methods

Clinical Integration:

  • HIPAA-compliant data storage and end-to-end encryption
  • Clinical report generation for healthcare providers in standard medical formats
  • Integration with EHR systems (Epic, Cerner) via HL7 FHIR APIs
  • Randomized controlled trials to validate efficacy as an early detection tool

Platform Expansion:

  • Mobile applications (React Native) for iOS and Android
  • Offline-first Progressive Web App for areas with limited connectivity
  • Multi-language support starting with Spanish, Mandarin, and Hindi
  • Caregiver collaboration tools - shared family accounts and notification systems
  • Cloud infrastructure migration to AWS/Azure with PostgreSQL and S3 storage for scalability

CogniConnect represents our commitment to leveraging AI for meaningful healthcare impact - not just preserving memories, but actively monitoring cognitive health to enable earlier interventions and better outcomes for millions affected by Alzheimer's disease.

Built With

Share this project:

Updates