DermAssist Live Agent

“Your skin speaks — DermAssist listens, sees, and triages in real time. A multimodal agent analyzing conditions, detecting urgency, and surfacing evidence before physician review.”

Comment

🌟 Inspiration Dermatology is broken for the average person. In most countries, the wait time to see a dermatologist is 6 to 12 weeks. In rural areas, it can be longer — or simply impossible. Meanwhile, skin conditions are among the most common reasons people visit emergency rooms, and a significant portion of those visits could have been triaged and redirected far earlier if someone knowledgeable had just looked at the skin and asked the right questions. The moment that crystallized this project: a family member woke up with a spreading rash, mild fever, and no idea whether to call an ambulance or buy some antihistamines. They didn't know that fever + spreading rash is a clinical red flag that warrants emergency evaluation. They waited three days before seeing anyone. It turned out to be manageable — but it might not have been. Most people don't have a doctor friend they can send a photo to at 10pm. Most people can't tell the difference between contact dermatitis and early Stevens-Johnson syndrome. Most people don't know what questions to ask, what details matter, or how urgently to act. We wanted to build that doctor friend. One that is available instantly, speaks to you in real time, looks at your skin through your camera, asks the right questions one at a time, knows exactly when to say "go to the ER right now" — and knows when to say "this looks routine, here's what to discuss with your doctor next week." That is DermAssist Live Agent.

🩺 What It Does DermAssist Live Agent is a real-time, multimodal AI dermatology triage assistant that serves as the first point of contact for patient intake — before a physician is ever involved. A patient opens the app in their browser. They can:

Speak about their symptoms via microphone (real-time voice, not push-to-talk) Show their skin condition live via camera or upload a high-resolution photo Type if they prefer text

The agent conducts a structured 5-step clinical triage protocol: Step 1 — Structured Intake The agent greets the patient warmly and collects information conversationally — asking only 1 to 2 questions at a time, never an overwhelming form. It gathers: chief complaint, onset and duration, body location, symptom quality (itch / pain / burning), systemic symptoms (fever, fatigue, sore throat, joint pain), current medications, allergies, recent travel or exposures, and special population flags — age, pregnancy, immunocompromised status. Step 2 — Image and Video Analysis If the patient shares a camera frame or image, the agent performs a preliminary visual analysis. It identifies lesion morphology (macule / papule / pustule / vesicle / plaque / nodule / ulcer), color and border characteristics, distribution pattern, approximate body region, and secondary changes like scaling, crusting, or lichenification. It then asks targeted follow-up questions based on what it observes. Step 3 — Triage Classification Before anything else, the agent assesses clinical red flags: PriorityTriggerAction🚨 EMERGENCYRapid spread + fever, OR fever + mucosal involvement (mouth / eyes / genitals)Immediate ER referral. Provides brief differential for receiving clinician. Stops — no search, no summary.⚠️ URGENTSingle red flag: mucosal involvement, immunocompromised, pregnancy, rapidly spreading rash, fever with rashAdvises same-day or urgent care✅ ROUTINENo red flags, stable or chronic presentationStandard dermatology appointment Step 4 — Evidence-Based Search For URGENT and ROUTINE cases, the agent calls Google Search with the query: "First-line clinical management [condition] 2026 dermatology guidelines" — citing AAD, UpToDate, and BMJ where available. This step is skipped for EMERGENCY cases where speed matters more than context. Step 5 — Consultation Summary The agent produces a structured output formatted for physician handoff:

Suspected conditions ranked by likelihood with confidence level Evidence-based treatment overview (informational — Rx/OTC options, standard ranges, key side effects) Questions for the physician that would most refine the diagnosis Recommended next steps tailored to the triage level Mandatory disclaimer that this is preliminary and not a prescription

The entire interaction happens in real time — voice in, voice out, camera streaming — with natural interruption handling so the conversation feels human.

🔧 How We Built It The Stack LayerTechnologyAI modelGemini Live 2.5 Flash — gemini-live-2.5-flash-native-audioDeployment targetVertex AIAgent frameworkGoogle ADK — google-adkSearch groundinggoogle_search built-in ADK toolBackendFastAPI + Uvicorn (Python)Real-time transportWebSocket — bidirectionalFrontendVanilla JavaScript + Web Audio API + AudioWorkletDeploymentGoogle Cloud Run (source-based)

Built With

Share this project:

Updates