ClinicQuick

ClinicQuick has your health story ready before you see the doctor: an AI intake assistant interviews you, and your clinician gets a one-page briefing with potential diagnoses

Comment

💡 Inspiration

It started with a number: 36% of Canadians who saw a specialist for the first time in 2024 waited three months or more for that appointment, and 64% said the wait affected their lives (Statistics Canada, Survey on Health Care Access and Experiences, 2024).

When the appointment finally comes, the patient's story is usually in pieces: an old lab report, a prescription label, months of watch data, and whatever they can remember on the spot. The clinician then spends part of a short visit rebuilding a history the patient has already told someone else.

We asked: what if the patient's story were already organized before they walked in, and the clinician got one page with exactly what they need?

That became ClinicQuick at Hack the Valley 11.

🎯 What It Does

ClinicQuick is a web app with a patient side and a clinician side.

How it works for a patient:

  1. Sign up and fill in a one-time health profile (conditions, medications, allergies, history)
  2. Start a check-in with Lena, our AI intake assistant, by voice or text
  3. Lena asks follow-up questions based on your record, the way a triage nurse would
  4. Get a plain-language summary and a suggested next step: pharmacist, walk-in, family doctor or ER
  5. Book an appointment or share the summary with a clinic

What the clinician gets:

  • 🧠 Symptom model: top 5: a LightGBM model we trained ranks the 5 most likely conditions with a probability for each, ICD-10 codes, "must rule out" flags for serious conditions, and the next questions that would best tell them apart. Never shown to the patient.
  • 📋 A one-page briefing: history, medications, red flags checked, and conditions to consider (clinician-only, labelled "not a diagnosis")
  • 📈 Patient trends: heart rate, HRV, sleep, steps, blood oxygen and blood pressure over time
  • 💊 Medications: current and past, with doses and a timeline
  • 🗂️ Records: uploaded lab reports and prescriptions, with values extracted

Personalization in practice: two patients both report blood in their urine. Robert, 68, takes a blood thinner and used to smoke, so Lena asks about other bleeding and his medication. Aisha, 23, had a UTI last year, so she gets asked about urinary symptoms and her cycle. Same complaint, different questions, different briefings, and a different ranking for the clinician:

Aisha, 23 Robert, 68
#1 Urinary tract infection 80% Enlarged prostate (BPH) 48%
#2 Menstrual bleeding 14% Bladder cancer 24% ⚠️ must rule out
#3 Kidney infection 3% Prostate cancer 15% ⚠️ must rule out

Also built: appointment booking, a walk-in clinic map with queue position (simulated data), Apple Health import, a physio camera coach, and phone fall detection.

All patient data is synthetic.

🛠️ How We Built It

Frontend: Next.js 15 (App Router), TypeScript, CSS modules

  • three.js for the 3D anatomy model and Lena's wireframe avatar
  • Leaflet + OpenStreetMap for the walk-in clinic map
  • MediaPipe Tasks Vision (Pose) for the physio coach, running in the browser
  • Web Speech API and the DeviceMotion API for voice input and fall detection

Backend: FastAPI (Python), Pydantic, psycopg 3

Database: Postgres on Tiger Cloud

  • TimescaleDB hypertables for wearable readings and check-ins
  • A continuous aggregate (daily_obs) for daily values and per-patient baselines
  • pgvector for embedding past visit notes, so Lena can retrieve them

AI: Google Gemini via the google-genai SDK

  • Gemini Live API for real-time voice conversations, proxied through a WebSocket on our backend
  • Structured JSON output for picking each question and filling the intake form
  • Multimodal extraction of lab values from uploaded PDFs and photos
  • Text-to-speech for the turn-based voice mode
  • Automatic switching between several API keys when one runs out of quota

Diagnosis model: LightGBM (gradient-boosted decision trees), trained with an autoresearch loop

  • Trained on DDXPlus, a large public dataset of synthetic patients (49 conditions, each with symptoms and medical history), plus 10 blood-in-urine conditions we generated, for 59 conditions
  • 462 inputs: age, sex, 454 yes/no symptom and history answers, and 6 numeric answers (e.g. pain 0–10)
  • The final model is 3,422 small trees (58 boosting rounds × 59 conditions, up to 15 leaves each)
  • Training data and every experiment result live in Tiger Cloud (a hypertable of results plus a continuous aggregate for progress)

Interoperability: FHIR R4 transaction bundle export

Deployment config: Docker Compose + Caddy (automatic HTTPS)

Key Technical Features

1. Slot-based interview engine. Each complaint type has required slots (onset, severity, associated symptoms, what they've tried…). Gemini picks the next question that fills the most important empty slot. Without an API key, it falls back to fixed question sets, so the app never stops working.

2. Rule-based safety layer. Red flags are regex rules with negation handling ("no chest pain" doesn't trigger). They run on every message, including live voice transcripts. The rules set a minimum urgency, and AI can raise it but never lower it.

3. Context and memory. Every prompt includes the patient's profile, conditions, medications and smoking history, plus past visit notes retrieved through vector search.

4. Validated screening scores. CANRISK, PHQ-2, GAD-2, AUDIT-C and STEADI are computed in plain Python with published cut-offs, not by the model.

5. Autoresearch-trained diagnosis model. Inspired by Karpathy's autoresearch: an AI agent (Gemini CLI) repeatedly edits the training script, runs a 90-second training, and keeps the change only if the score improves. The score is top-3 accuracy after a short interview: the model sees only the patient's history, their opening complaint and about 30% of the remaining symptom questions, like Lena a few questions in. We train on copies of each patient with most questions hidden, so the model works with incomplete answers. Final result: 96.1% of cases have the true condition in the top 3, averaged so every condition counts equally. A fairness guardrail checks every sex and age group separately; the weakest scores 97.7%.

6. Clinician-only decision support. The model uses everything we know: Lena's conversation (Gemini extracts symptoms and "no" answers, with a keyword fallback), the profile, medications and wearable data (for example, a step-count spike counts as "after intense exercise"). Only conditions that can cause the main complaint are ranked, as in DDXPlus, and probabilities are capped at 80% so the model never claims certainty. Lena receives a hidden hint so she asks sharper questions (in typed and live voice, between turns only), but she never names a condition. Patients only get safe next steps.

🧗 Challenges We Faced

Challenge 1: Late start Problem: We didn't settle on an idea until a day and a half into the hackathon. Solution: We picked one core flow (patient check-in → clinician briefing), built that first, and cut anything that didn't support it.

Challenge 2: Making personalization real Problem: Adding the patient's history to a prompt mostly changed the wording, not the questions. Solution: We moved to structured slots plus retrieved patient context, so the system decides which facts matter for this complaint and asks about what's missing.

Challenge 3: Gemini quotas Problem: We kept hitting daily and per-minute limits while testing. Solution: We cut the number of calls per turn, used a faster model for simple steps, added a 12-second timeout per question, and built an offline fallback so the demo never stalls. We also added automatic switching between several API keys when one runs out of quota.

Challenge 4: Live voice + safety at the same time Problem: A real-time voice session has to stay smooth while every transcript is safety-checked and the intake form fills in the background. Solution: We ran the safety rules and the slot-filling in parallel on the backend, separate from the audio stream.

Challenge 5: Not overstepping Problem: AI-generated medical output can't be treated as automatically correct. Solution: Emergencies are rule-based, AI can only raise urgency, the conditions list is shown only to clinicians, and every output is labelled as decision support.

Challenge 6: An honest metric for the model Problem: With every symptom answered, the model scored almost perfectly, which said nothing about real use, where a patient has answered only a few questions. Solution: We changed the target to top-3 accuracy after a short, partial interview, added hidden-answer copies to training, and added a per-sex and per-age fairness check that the agent couldn't trade away.

🏆 Accomplishments We're Proud Of

✅ Real-time voice intake with Gemini Live, with safety checks on every transcript ✅ A diagnosis model trained by an AI research agent: 96.1% top-3 accuracy after a short interview, shown only to clinicians ✅ Questions that actually change based on the patient's history ✅ A working patient app and a clinician dashboard, both on the same data ✅ Time-series health data and vector memory in one Postgres database ✅ FHIR R4 export, so the output can go into real clinical systems ✅ A rule-based safety layer that AI cannot override downward

📚 What We Learned

Technical Skills

  • LLM engineering: structured output, timeouts, fallbacks, and deciding what should never be left to a model
  • Streaming: WebSockets and real-time audio with Gemini Live
  • Data: TimescaleDB hypertables, continuous aggregates and pgvector retrieval
  • In-browser ML: pose estimation and joint-angle rep counting with MediaPipe
  • ML evaluation: choosing a metric that matches real use (partial interviews), macro averages so rare conditions count, and subgroup fairness checks

Domain Knowledge

  • How triage nurses structure an intake interview
  • Why screening tools use fixed, published cut-offs
  • Why the line between decision support and diagnosis matters

Soft Skills

  • Scoping under pressure: cutting features early beats half-finishing three
  • Parallel work: splitting frontend, backend and AI work cleanly across the team
  • Designing for two users: patients need a calm conversation, clinicians need a short summary they can check

🚀 What's Next

Short-Term

  • [ ] Test the briefing with clinicians and measure whether it saves time
  • [ ] Link every line of the briefing back to its source (visit, document or reading)
  • [ ] Run many synthetic patients through the system to measure question and summary quality
  • [ ] Validate the ranking on real (de-identified) cases, since DDXPlus is synthetic

Medium-Term

  • [ ] iOS and Android companion apps for live Apple Health, Samsung Health and Health Connect data
  • [ ] More complaint types and languages

Long-Term

  • [ ] Consent management, access controls and audit logs before any real patient data
  • [ ] Integration with clinic systems through FHIR

Built With

Share this project:

Updates

Submission history