ERGO | Devpost

Welcome: patients pick English or French. Check-in takes about a minute, using voice and camera, with no video recorded.
ERGO asks out loud (ElevenLabs voice). Patients tap to talk, or type if they prefer.
Gemini filters out background chatter and shows a clean summary for the patient to confirm or redo.
Patients tap a 0–10 pain score, so the ranking uses an exact number rather than a guess from speech.
Webcam scan, calm patient: heart rate 67, breathing 14/min, alert, low pain expression → Non-urgent.
Same scan, patient in visible pain: pain expression 10/10 → Very urgent, and the kiosk tells them to alert a nurse now.
Check-in complete: Red-flag symptoms like chest pain send the patient straight to the front desk instead of the waiting room.

Inspiration

Last year, Canadians walked out of emergency departments 1.2 million times before a doctor ever saw them (CIHI, 2024–25). Among the sickest patients, the ones who should see a physician within 15 minutes under the Canadian Triage and Acuity Scale, 1 in 10 waited more than 4 hours (CIHI).

Triage guidelines say waiting patients should be reassessed regularly, but in a full waiting room no nurse can recheck everyone. People get worse in their chairs, and nobody notices until it's too late. We wanted to build something that keeps watch over the waiting room so nurses know who needs them first.

What it does

ERGO listens to patients, reads their vital signs, and tells nurses who needs care first.

A patient walks up to the ERGO kiosk and:

Picks a language (English or French).
Talks. ERGO asks out loud, "What brings you in today?" The patient answers by voice (or types). ERGO shows a clean summary of what it understood so they can confirm it, then asks them to tap a 0–10 pain score.
Gets scanned. The webcam estimates heart rate, breathing rate, alertness and facial pain expression, with no wearables and no video recorded.
Gets a check-in number (like A-07), spoken out loud. Patients with red-flag symptoms are told to go to the front desk immediately instead of taking a seat.

On the nurse dashboard, every waiting patient appears as a card, ranked by urgency, with the reasons listed ("Red flag: chest pain · Pain 8/10") and the patient's own words quoted. When a patient becomes urgent, the dashboard sounds an alarm and shows a banner until a nurse acknowledges it. An "I need help now" button on the kiosk puts a patient at the top instantly.

How we built it

Frontend: React + Vite. The kiosk and dashboard sync in real time through the browser's BroadcastChannel, with localStorage so no check-in is lost if the dashboard reloads.
Voice (ElevenLabs): Scribe v2 for speech-to-text and Flash v2.5 for fast, natural text-to-speech in English and French. The API key stays server-side in small serverless functions.
Understanding symptoms (Gemini): Gemini 3.5 Flash-Lite with structured JSON output turns what the patient said into fixed fields: symptoms, red flags (chest pain, breathing difficulty, stroke signs, severe bleeding, fainting, confusion), pain score and onset. It also filters out background chatter, so "can you hear me? Mike, get the car" doesn't become a symptom.
Camera vitals: MediaPipe face and pose landmarks, with the POS remote-photoplethysmography algorithm on forehead and cheek regions for heart rate, and shoulder and forehead motion for breathing. All of it runs in the browser.
Ranking: deterministic rules, not AI. Red flags, critical vitals or the help button make a patient red. Pain and rising heart rate add points. Ties are broken by the most severe red flag, then score, then who has waited longest.

Challenges we ran into

Webcam heart rate is hard. A pulse changes skin color by a fraction of a percent, and laptop cameras constantly auto-adjust exposure and white balance, which Chrome on macOS doesn't let you lock. We spent hours on signal windows and confidence gating so ERGO would show "unclear" rather than a wrong number.
Who should decide priority? Letting an LLM rank patients was tempting, but it's unpredictable and hard to audit. We settled on a split: Gemini understands the patient's words, and transparent rules decide the order.
Noisy rooms. Our first transcripts were full of other people talking. Adding a relevance check before the confirm screen fixed it.
Not over-promising. Our first confirmation screen said "a nurse will see you within 30 minutes." We removed it: promises the ER can't keep only cause frustration. Patients get a clear action instead ("If you feel worse, tell a staff member").

Accomplishments that we're proud of

A complete, bilingual, voice-first check-in that works end to end in under two minutes.
A ranking every nurse can audit: every card shows exactly why a patient is where they are.
A safety path that doesn't wait in line: red flags send the patient to the desk and alarm the nurse station at the same time.

What we learned

In healthcare, how you use AI matters as much as whether you use it. Keeping the model's job narrow made the whole system easier to trust.
Good kiosk UX means one thing per screen, a voice that leads, and never showing patients alarming medical labels.
Camera-based vitals are promising but sensitive to lighting and hardware, so honest confidence reporting matters more than always showing a number.

What's next for ERGO

"I feel worse" rechecks: patients rescan with their check-in number, and their card rises on the dashboard if they've deteriorated.
Clinically set thresholds, designed and validated with ER nurses and physicians.
More languages, starting with Canada's most common non-official languages.
Hospital integration, so ERGO's rechecks flow into existing triage systems.

ERGO is a hackathon prototype, not a medical device. Camera readings are estimates, and every decision stays with clinical staff.