MediTalk

Inspiration

A doctors job usually supplemented by other colleagues beside them. Imagine if they had a AI available 24/7 to assist in not only diagnosing, but providing detail about a doctor’s patients upper body x-ray scans. Studies show:

Pneumonia affects over 1 million people in the US a year, and so detecting, and providing a machines opinion to supplement a doctors knowledge is a powerful tool.
Over 50,000 die from pneumonia a year alone in the US,
Approximately 12% of patients diagnosed with pneumonia are misdiagnosed, preventing further care

What it does

MediTalk is an AI-driven platform designed to assist healthcare professionals and patients in analyzing chest X-ray images. It provides instant preliminary analysis, powered by computer vision (CNN models) and a voice-based AI assistant for interactive guidance.

How we built it

Frontend:
- Next.js (React-based)
- Tailwind CSS for styling
- next/image for optimized images
- @livekit/components-react for real-time audio/video and voice assistant
Backend:
- Express
- REST API endpoints for authentication and image analysis
- MonogoDB
AI/ML:
- Gemini
Real-Time Communication:
- LiveKit for audio and AI agent interactions

Challenges we ran into

Initially, we struggled to run the medgemma model using mps since the initial load time was very slow. We eventually switched to use colab t4 gpu

Accomplishments that we're proud of

Instant X-ray Analysis: Upload chest X-ray images to receive AI-assisted preliminary diagnosis.
Voice Assistant Interaction: Communicate with the AI agent for guidance, insights, and image analysis explanations.
Modern & Responsive UI: Clean, B2B-oriented design with smooth animations, intuitive navigation, and a mobile-friendly layout.
Real-Time Audio & Visual Feedback: Interactive audio controls and visualizations for seamless AI assistance.

What we learned

We gained additional hands-on experience in collaborating to produce a solution aimed at solving real-world issues. We were also able to learn from encountering issues and seeking help from one another throughout the development of the project.

What's next for MediTalk

Store user data to use as future reference for our AI to look back on for a more detailed analysis of the patient
Provide a interface for patients of doctors to look at their own results
Develop models for a variety of diseases, not just pneumonia