-
-
Cover Image
Project Story
About the Project
SOFIA is a voice-first AI triage assistant designed to help healthcare clinics collect patient information before a doctor visit. The system talks with patients naturally, gathers identity and clinical information, measures basic vital signs using the camera, and generates a structured PDF report for medical staff.
The goal is simple: reduce the administrative bottleneck that happens before a patient sees a doctor.
Inspiration
In many clinics in Colombia and across Latin America, patient intake is slow and repetitive. Nurses often have to ask the same questions dozens of times every day: who the patient is, what symptoms they have, how long they’ve had them, and how severe they are.
This process takes time and adds pressure to already overloaded healthcare workers. I started thinking about a simple question:
What if patients could complete this process themselves through a natural conversation before seeing a doctor?
That idea led to SOFIA, a system that can guide patients through triage using voice interaction.
How the System Works
SOFIA runs on a phone, tablet, or kiosk in a waiting room. The interaction is fully conversational.
The process happens in four stages:
Identity collection The patient provides their name, ID number, date of birth, and sex. They can either say the information or show their ID card, which SOFIA reads using computer vision.
Clinical intake SOFIA asks about the main complaint, symptom location, duration, and severity on a scale from (1) to (10). It can also collect medical history and optional photos of visible symptoms.
Vital signs scan The system performs a 30-second camera scan to estimate heart rate and respiratory rate using remote photoplethysmography (rPPG).
Report generation A structured two-page PDF report is generated with demographics, symptoms, vitals, and a triage priority level. Doctors receive the report before meeting the patient.
The system does not diagnose or recommend treatment. Its purpose is to collect and organize patient information.
How I Built It
The project uses a serverless architecture on AWS.
- Frontend: Static web application hosted on S3 and delivered through CloudFront.
- API layer: AWS Lambda and API Gateway handle session initialization.
- Agent runtime: AWS AgentCore maintains the real-time voice session.
- Speech and conversation: Amazon Nova Sonic processes speech recognition, language understanding, and voice responses.
- Structured data extraction: Nova 2 Lite processes clinical information through a triage tool built with the Strands agent framework.
- Storage: Patient media and generated reports are stored in Amazon S3.
- PDF generation: ReportLab assembles the final medical report.
The triage pipeline is structured into stages to ensure reliable data collection and validation before moving to the next step.
Challenges
One challenge was structuring conversational data into reliable clinical information. Patients speak in different ways, so the system needs to normalize their answers into standardized medical language.
Another challenge was maintaining a smooth real-time voice interaction while orchestrating multiple AI services and tools in the background.
Finally, ensuring the system collects all required information before continuing required careful state management in the triage pipeline.
What I Learned
Building SOFIA taught me several important lessons:
- Real-world problems often benefit more from simple, reliable workflows than from complex features.
- Voice interfaces can significantly improve accessibility, especially for elderly patients or people with limited literacy.
- Using structured agents and staged pipelines helps maintain consistency when working with AI-generated outputs.
Most importantly, this project reinforced the importance of focusing AI on practical problems that improve everyday workflows, especially in healthcare systems with limited resources.
Log in or sign up for Devpost to join the conversation.