Patient in a Pocket

Inspiration

We recognized that effective healthcare extends beyond just diagnosis and treatment, as empathy and communication are critical components of patient care. Medical students and practicing physicians often lack adequate opportunities to practice bedside manner in a safe, judgment-free environment. We built this tool to bridge that gap, enabling doctors to refine their communication skills through realistic patient interactions while receiving constructive feedback on their approach.

What it does

This program simulates a medical consultation between a doctor (the user) and a virtual patient (AI). The doctor conducts a patient interview through voice, receiving real-time responses from an AI patient with various symptoms and concerns. After the conversation concludes, our system analyzes the interaction and provides detailed bedside manner feedback across four dimensions: empathy, clarity, active listening, and professionalism, with each scored 1-10 with constructive recommendations for improvement.

How we built it

We built it using OpenRouter API, OpenAI Whisper, and Eleven Labs. We used VS Code and the help of AI to develop this system.

Challenges we ran into

Our biggest challenge was integrating everything together in order for proper speech-to-text transcription. For all of us, this was one of our first major coding projects that involved the integration of LLM's or outside sources. So, we had to do a lot of background research to actually understand the process of how what we are doing.

Accomplishments that we're proud of

Ultimately, we are proud that we were able to get a working conversation going between the human and the AI chatbot voice. Even though it is not a seamless experience yet, and there is a lot of room for improvement, we are excited that we have at least something to show at the end of these past hours, especially since it is something that we will be able to build on in the future.

What we learned

This was an amazing learning opportunity, diving straight into this. While it may not seem outwardly complicated, it was a process figuring out how to go from a voice input, and then convert it to text, run that text through an AI to develop a response, and then subsequently have a voice play back the response to simulate a real-world scenario.

What's next for Patient in a Pocket

We are looking forward to continuing this project and be able to develop something that runs with fewer errors and create a website for a more seamless experience. There is much room for improvement and this is a solid starting point!

Built With

• elevenlabs
• openai
• openrouter
• python