InsightRX

One no-needle eye photo tells a clinician what to treat, which protein and drug to target, and who to engage next: a specialist, a trial or the manufacturer.

Comment

Inspiration

Diabetic retinopathy is a leading cause of blindness in working-age adults, yet about a third of people with diabetes miss their yearly eye exam. Portable fundus cameras now let a primary-care clinic photograph the retina in a minute, without dilation drops or a needle. But the photo usually leads nowhere: the result sits in a chart, the referral is a fax, and nobody knows whether the retina specialist ever saw the patient.

The Impiricus challenge asked us to invent the next way we engage HCPs, with no SMS and nothing Impiricus already offers. We saw that the retina is the one place where blood vessels and nerve tissue can be seen directly, and that one eye photo at the point of care is a clinical signal strong enough to trigger a real engagement between the clinician and a specialist, a trial or a manufacturer, exactly when the therapy decision is made.

What it does

Insight Rx is a new HCP engagement channel triggered by a clinical signal: one no-needle eye photo tells the clinician what to treat, which protein and drug to target, and who to engage next (a specialist, a trial or the manufacturer).

  1. Home Screen: An operator uploads photos of both eyes. A quality gate rejects unusable images, then a fine-tuned vision model scores referable diabetic retinopathy, ICDR grade (0–4) and macular edema, with attention heatmaps and a "why this result" panel. If either eye has no usable image, the result is incomplete, never reassuring.

  2. Whole-body view (oculomics): The same photos score systemic conditions (heart, kidney, nerves, metabolism). Each score carries its evidence tier (research, exploratory or near-chance) and what moved it, and recorded history always takes precedence over the model.

  3. Therapy and targets: Each finding maps to guideline therapy classes (ADA, AAO, KDIGO), checked against the patient's medicines for eye-specific safety alerts (for example semaglutide with retinopathy, pioglitazone with edema). Insight Rx then ranks personalised protein targets. VEGF-A, for example, opens in its real PDB drug complex and its AlphaFold structure, with the confidence measured at the exact drug-contact residues, plus ChEMBL drugs and recruiting ClinicalTrials.gov studies near the clinic.

  4. Engage: The clinician signs a consultation package (every statement traced to case data, with an evidence brief) and sends it to a specialist. The specialist accepts and responds, a coordinator schedules the visit, and the handoff only closes when the referrer acknowledges the answer. Outside the network, the clinician can refer through the CMS NPI Registry with a printable letter.

  5. Ask the manufacturer: From any therapy option, the clinician can send a question to the manufacturer's medical-information desk. The desk sees only an age band and finding labels, never the patient, and nothing it sends can change scores, ranking or referrals.

  6. Quality and billing: Each encounter shows where it stands against CMS131 / MIPS #117 and HEDIS EED, and which retinal-imaging CPT code (92227/92228/92229) fits.

  7. Copilot and patient explainer: A clinical copilot remembers each clinician's preferences and follow-ups across sessions. For the patient, the result is rewritten in plain English or any other foriegn language and read aloud.

Every role (operator, referring HCP, specialist, coordinator, manufacturer desk, admin) sees only what it should, and every action lands in an append-only audit trail.

Why it matters Diabetic eye disease is the number-one cause of blindness in working-age adults. One in four people with diabetes already has it, and a third skip their yearly eye exam. It's also preventable: caught early, it can be treated before sight is lost. The problem isn't the camera, it's what happens next. Most eye photos end up as a line in a chart, and the family doctor is left without a clear next step. Insight Rx turns that photo into action during the visit: what to treat, which drug to target, and which specialist to send the patient to. Because retinal vessels mirror the heart, kidneys and nerves, one no-needle photo can flag risks across the whole body. It works on a handheld phone camera, so it can reach rural and underserved clinics first. The impact is measurable. For every 10,000 patients, that's about 3,500 missed eye exams closed, $300,000 in billable readings, and 2,600 people with eye disease found before they lose their sight.

How we built it

  • Retinal model: DINOv2-L at 392 px with LoRA (rank 16) and a multi-task head (unusable image, referable DR, ICDR grade via CORAL ordinal loss, macular edema), trained on mBRSET, a real Brazilian portable-camera dataset from PhysioNet. We used a frozen patient-grouped 70/10/20 split, a 2-seed ensemble with flip test-time augmentation, per-head temperature scaling, and thresholds frozen on validation.

  • Results on held-out patients: (258 patients, 1,032 images): referable DR AUROC 0.980 (95% CI 0.959–0.996), 81.7% sensitivity and 98.4% specificity; ICDR quadratic kappa 0.874.

  • Systemic models: Frozen RETFound and DINOv2 embeddings plus metadata, 5-fold patient cross-validation over all 1,291 patients. A head is released only if its AUROC is at least 0.65 and its bootstrap confidence interval clears 0.55; everything else is labelled exploratory.

  • Explainability: Gradient × activation attention maps masked to the fundus, per-eye margins against the frozen threshold, occlusion contributions for systemic scores, and reliability curves.

  • App: FastAPI with server-rendered Jinja pages and a hand-built design system (no front-end framework, strict CSP), SQLAlchemy on Neon Postgres, deployed on .tech domain. Inference runs on a live GPU worker reached through a Cloudflare tunnel; we also export a portable ONNX bundle that matches PyTorch to within 5e-7.

  • Public knowledge: AlphaFold DB, RCSB PDB, UniProt, ChEMBL, ClinicalTrials.gov, CMS NPPES NPI Registry and openFDA, cached with snapshot fallbacks. Only a condition term and the clinic ZIP ever leave the app. The 3D viewer is self-hosted 3Dmol.js, and PDF reports are built with reportlab.

  • Sponsor integrations:

    • Gemini API: cited copilot answers, evidence briefs and the bilingual patient explainer. Output that adds numbers or identifiers is rejected in favour of a safe template.
    • ElevenLabs: reads the explainer aloud for patients with low vision or low literacy.
    • Backboard: persistent copilot memory, with a private clone per clinician and a thread per patient.
    • Tiger Data: a TimescaleDB hypertable and continuous aggregate of de-identified findings (the "eye-detected demand over time" series).
    • Solana: SHA-256 digests of signed reviews and consultation packages written to the Memo program on devnet, so records are tamper-evident.

  • Quality 84 automated tests cover tenant and role isolation, the consultation state machine, signature invalidation, safety rules for incomplete cases and every partner integration. The demo video is recorded live on the deployed app with Playwright.

Challenges we ran into

  1. Our first DR threshold was set on only 28 validation positives and gave 73% sensitivity on test. We changed the rule, documented that the change came after seeing the test result, and kept both sets of metrics rather than re-tuning on the test set.
  2. Retinal images added a statistically supported signal only for neuropathy, and a first gradient-boosting baseline overfit rare conditions below chance. We rebuilt it with regularised logistic regression and a release gate, and show every other condition as exploratory instead of hiding it.
  3. Serving a 400 MB ensemble for free. Vercel can't host GPU inference, so the web app runs serverless while a GPU worker registers itself through a Cloudflare tunnel. When the worker is off, the site keeps working and clearly shows simulated mode.
  4. Pharma engagement without pharma influence: The manufacturer desk needed to be useful but firewalled. We restricted it to de-identified context, labelled sponsored content, and wrote a test proving sponsorship never changes ranking.

Accomplishments that we're proud of

  • AUROC 0.980 on unseen patients from a real portable-camera dataset, with calibrated, frozen thresholds and a quality gate.
  • Carrying a retinal finding all the way to a protein target, a drug, a trial and a named physician, including AlphaFold confidence measured at the actual drug-contact residues.
  • A complete two-clinician handoff with signatures, due times, an audit trail and tamper-evident anchoring, not a one-way "refer" button.
  • Five sponsor integrations that each do real clinical work, all optional, all de-identified and all tested.
  • A full-stack deployment on free tiers, with a live GPU model and a portable ONNX bundle.

What we learned

  • Medical AI earns trust through what it refuses to claim: an incomplete result must never look reassuring, and a weak signal should be labelled weak.
  • Engagement is strongest at the moment of decision. A finding on screen is a far higher-intent moment than any broadcast message.
  • Compliance is a design constraint, not a feature. The firewall, de-identification and audit trail shaped the data model from day one.
  • Foundation models (DINOv2, RETFound) with LoRA get strong results from a few thousand images, but calibration and threshold discipline matter as much as the backbone.

What's next for Insight Rx

  • External validation on other cameras and populations (APTOS, Messidor-2, IDRiD), plus a prospective pilot in primary-care clinics.
  • EHR integration through FHIR, so screenings and consults live in the chart.
  • An FDA pathway for autonomous reads (CPT 92229).

Built With

Share this project:

Updates

Submission history