SentimentMD

Title Page
details

About SentimentMD: Reconnecting Humanity with Healthcare

Our Inspiration: The Empathy Gap in a Pixelated World

Telemedicine is a modern marvel, but in its quest for efficiency, it often loses its heart. We've all been there: the sterile video calls where a patient's subtle frown of concern or a doctor's reassuring glance gets lost in digital translation. This growing "empathy gap" was our call to action. We were inspired by a simple yet profound question: "What if technology could do more than just connect? What if it could actually help us understand?"

SentimentMD was born from this mission: to weave the human touch back into the fabric of digital healthcare, making every virtual consultation as insightful and empathetic as an in-person visit.

What We Learned: Building for Trust, Not Just Tech

This wasn't just a coding challenge; it was a crash course in human-centered design and medical-grade engineering.

Emotion is a Data Stream: Our biggest lesson was learning to treat emotion not as a vague concept, but as a rich, measurable data stream. We discovered how a single conversation contains layers of information in tone, word choice, and micro-expressions, all waiting to be understood.
Security is the Foundation: We learned that for any healthcare tool, trust is non-negotiable. Building a HIPAA-compliant platform from the ground up taught us that security isn't a feature, it's the bedrock upon which everything else must be built.
AI as an Augment, Not a Replacement: We solidified our belief that AI's best role in healthcare is to augment a doctor's intuition, not replace it. Our goal was never to tell a doctor how a patient feels, but to provide them with data points they might have missed, empowering them to ask better questions and connect more deeply.

How We Built It: An Orchestra of Cutting-Edge Services

SentimentMD is an orchestrated system where each component plays a crucial role. We followed a three-act development structure:

Act I: The AI Core - Capturing Emotion: The heart of our project is Hume AI. We built a resilient WebSocket architecture to stream live audio and video to Hume's empathic voice and expression models. This was our first major breakthrough, turning raw pixels and soundwaves into a structured, real-time feed of over 20 distinct emotional expressions.
Act II: The Secure Stage - Building the Fortress: With the AI core in place, we engineered the secure video infrastructure. We chose Twilio Video for its robust, encrypted WebRTC capabilities, ensuring all communication is private and peer-to-peer. We architected our backend with Express.js, implementing meticulous access protocols to allow both user's to interact consistently.
Act III: The Compassionate Interface - Bringing It to Life: We wrapped our powerful backend in a beautiful, intuitive UI crafted with Next.js, TypeScript, and Tailwind CSS. We intentionally avoided a sterile, clinical feel. Instead, we used subtle animations with GSAP to make the emotional data visualizations feel organic and alive. The dashboard doesn't just display numbers; it breathes, making technology feel less like a tool and more like an empathetic partner.

Leveraging Gemini for Intelligent Clinical Reporting

Beyond real-time emotional insights, we've integrated Google Gemini to elevate the post-consultation experience. After a session concludes, Gemini processes the aggregated sentiment data, key conversational points, and patient-reported symptoms. It then intelligently synthesizes this information to generate concise, actionable clinical reports for the doctor. This not only saves valuable time in documentation but also provides a holistic summary that highlights emotional trends and potential underlying concerns, ensuring no critical detail is overlooked.

Challenges We Overcame

Building the future of healthcare isn't easy. We tackled significant hurdles head-on:

The Latency Dragon: Our greatest challenge was synchronizing multiple real-time streams, video, audio, and AI analysis, without introducing noticeable lag. We obsessively optimized our data pipelines, ensuring that the emotional insights appeared on the doctor's dashboard in perfect sync with the live conversation.
The HIPAA Mountain: Navigating the complexities of healthcare data privacy was a formidable task. We dedicated countless hours to research and implementation, attempting our best to ensure every packet of data was handled with the security and respect patients deserve.
AI Service Orchestration: Getting multiple AI services (Hume for live analysis, and another LLM for pre-screening summaries) to "talk" to each other seamlessly required a sophisticated event-driven architecture.

Future Technical Enhancements

While SentimentMD already boasts a robust feature set, we envision continuous development to further enhance its capabilities and user experience: Robust User Authentication & Authorization: Implementing advanced authentication mechanisms (e.g., multi-factor authentication, OAuth) to ensure highly secure access for both patients and healthcare providers, alongside fine-grained role-based access control. Real-time Transcription Integration: Integrating a high-accuracy, real-time transcription service to provide live text of the conversation, complementing emotional insights and serving as an immediate reference for doctors. Electronic Health Record (EHR) Integration: Developing seamless integrations with popular EHR systems to automatically populate patient records with consultation summaries, emotional insights, and relevant data points, reducing manual data entry and improving continuity of care. Multi-language Support: Expanding the platform to support multiple languages for both video conferencing and AI analysis, making empathetic care accessible to a global patient base. Advanced Analytics Dashboard: Enhancing the doctor's dashboard with more sophisticated analytical tools, allowing for long-term tracking of patient emotional trends, symptom correlation, and personalized care planning. AI-Powered Pre-Screening & Triage: Utilizing AI to conduct initial patient screenings and assist with triage, identifying urgent cases or specific needs before the live consultation begins. Wearable Device Integration: Exploring integration with health-monitoring wearable devices to incorporate physiological data (e.g., heart rate variability, sleep patterns) into the holistic patient profile, enriching the AI's understanding.

What's Next for SentimentMD

This hackathon is just the beginning. We envision SentimentMD expanding to provide support for non-verbal patients, assisting in mental health triage, and even serving as a training tool for new medical professionals. We are building a future where technology doesn't widen the gap between us, but closes it, fostering a new era of empathetic, connected, and profoundly human healthcare.