Spot Check MD

Inspiration

During a medical visit, visual findings, dictation, and chart history can become disconnected. Audio-based AI scribes can help document what is said, but a clinician may notice physical findings without describing every detail aloud. We wanted to explore how image analysis could help capture those observations while keeping the physician in control.

What it does

SpotCheckMD is a physician-reviewed clinical documentation prototype. A clinician can select a fictional patient in the Demo EHR, confirm the patient and consent, capture or upload an image, and record doctor-only dictation. MedGemma 1.5 4B generates structured visual observations, while AssemblyAI Medical Mode transcribes audio separately. Finalized transcript text may provide bounded context to image analysis, but it is not treated as visual evidence.

The clinician can inspect each suggestion alongside its source image, then accept, edit, or reject it. Only accepted findings are assembled into a SOAP note. The clinician reviews and approves the exact note before it is submitted to the simulated EHR. The application is a research prototype, not a clinically validated diagnostic system or a connection to a live hospital EHR.

How we built it

We started by identifying the workflow needs we wanted to support: an intuitive interface, observations linked to their evidence, and clinician control at every step. We used AI coding agents to help build and refine the browser interface, backend services, API routes, and data persistence.

The application uses Next.js, React, and TypeScript, with server-side API routes and PostgreSQL for accounts, encounters, review state, and queued jobs. Physician passwords are hashed, sessions are secured, and encounter records are scoped to the physician’s workspace. Uploaded media is stored privately.

The image pipeline corrects orientation, removes metadata, preserves a lossless analysis copy, and checks image quality before processing. MedGemma 1.5 4B runs as the sole image-analysis model, self-hosted with llama.cpp and Q8_0 quantization. Audio transcription uses AssemblyAI Medical Mode as a separate path. Longer analyses run as persistent background jobs; the system serializes inference for the available model service and checks encounter revisions so stale results are not applied. Response structure and evidence references are validated before suggestions reach the review workflow. SOAP notes are assembled with deterministic application logic from clinician-approved findings.

Challenges we ran into

Connecting a self-hosted vision model to a web application involved more than sending an image to an endpoint. We had to handle image preparation, request formats, structured model responses, and service failures while keeping the clinician’s workflow understandable.

We also had to coordinate separate image and audio pipelines without treating transcript text as visual evidence. Long-running inference required background jobs and serialized requests, and draft revisions needed checks to prevent outdated analysis from being attached to newer encounter content. Throughout the workflow, we worked to make consent, evidence review, and physician approval explicit.

Accomplishments that we're proud of

We built a working end-to-end prototype that connects image capture, doctor-only dictation, model processing, evidence-linked review, SOAP note creation, and submission to a simulated EHR.

We are especially proud of the human-review workflow: model suggestions remain pending until a clinician evaluates them, and the final note is built from findings the clinician accepts. MedGemma failures do not trigger a silent switch to another image model.

What we learned

A useful clinical AI workflow depends on much more than model output. Image quality, evidence links, uncertainty, reliable processing, data access, and clear review steps all affect whether clinicians can understand and evaluate a suggestion.

We also learned to keep the roles of each component distinct: image analysis proposes visual observations, transcription captures speech, application logic assembles the note, and the physician makes the final decisions.

What's next for SpotcheckMD

Next, we want to evaluate the workflow with clinicians and appropriately governed, clinician-reviewed data; measure the quality and consistency of generated observations; and improve image-quality guidance and uncertainty handling. We also plan to strengthen privacy, security, retention, and audit controls before considering use with real patient information.

Longer term, we would explore validated measurement and comparison features, broader deployment options, and carefully scoped integration with real EHR systems. Any clinical use would require validation for its intended purpose and appropriate clinical and regulatory review.

Majors and Departments:

CISE (CS): Promit Roy, Aneal Singh, Ishaan Singh,

CLAS (Micro Bio- Cell Science): Rohan Patel

CLAS (CS): Andrew Dominguez

Built With

• css
• dockerfile
• javascript
• plpgsql
• python
• shell
• typescript