EyeSpy

I spy with my little eye… a missed dose! Eye Spy pairs a smart eye-drop dock with a joke-cracking AI voice to keep cataract patients’ recovery on schedule.

Comment

Inspiration

After cataract surgery, patients usually go home with three different eye drops: an antibiotic, a steroid and an anti-inflammatory. Each has its own schedule, the steroid tapers week by week (4× a day, then 3×, 2×, 1×), and drops need a few minutes between them so one doesn't wash out the other. That's a hard schedule to follow even with perfect eyesight. These patients are mostly older adults whose vision is blurry right after surgery, so they're asked to read small labels and keep track of all of it exactly when it's hardest to do.

Missed or mistimed drops can slow recovery or raise the risk of infection and inflammation, and the surgeon usually has no idea how well the patient is keeping up until the follow-up visit. We wanted to build something that reminds the patient, guides them through each dose, and shows the care team how things are going, without asking the patient to read a screen.

What it does

EyeSpy has three parts:

  • A bedside dock (ESP32 + LCD + two LEDs + a switch). It shows the next dose, alerts when it's time, and the patient presses the switch after each drop. A red LED counts down the safe wait before the next drop, and a green LED shows when it's ready.
  • A low-vision-friendly iPad web app for patients. It has large text, dark and high-contrast modes, and one big "Play reminder" button. A spoken reminder plays automatically when a dose is due. Gemini writes the reminder in a warm, gently funny, grandparent-style tone, and humor can be set to full, mild or off. Patients can also keep an eye recovery journal: they save photos of their eye and can get an AI description of visible features (redness, irritation, cloudiness). It's always labeled as non-diagnostic.
  • A provider dashboard where the doctor sees only their own patients: 7-day adherence, missed-dose flags, a 14-day results chart, recent doses and the patient's eye-photo history.

How we built it

  • Firmware: MicroPython on an ESP32 DevKit V1 with an LCD1602 over I²C. The dock checks the backend every 15 s (GET /device/status) and posts every switch press (POST /device/events). With no WiFi config it runs a standalone offline demo instead.
  • Backend: Python + FastAPI. It's the only part that holds secrets (database URL, Gemini key), so nothing sensitive ships to the browser or the device.
  • Database: Tiger Data (PostgreSQL + TimescaleDB). Dose events and raw device events are hypertables. A continuous aggregate (daily_adherence) pre-computes daily counts per patient, so the dashboard never scans raw rows.
  • Auth: Auth0 with role-based access (patient / provider). Every request needs both the role claim in the token and a matching database row. A provider asking for someone else's patient gets a 404, not a 403, so the API doesn't even confirm the patient exists. Docks authenticate with a device token that is stored only as a SHA-256 hash.
  • Frontend: React + Vite, added to the iPad home screen so it opens full-screen.
  • AI: Gemini for reminder text, Gemini TTS for the voice, and Gemini vision for the eye-photo descriptions.

Rules decide, AI only adds personality

We didn't want an LLM deciding anything medical. Each dose is flagged by deterministic, unit-tested rules. With $\Delta = t_{\text{lifted}} - t_{\text{scheduled}}$:

$$ \text{flag}(\Delta) = \begin{cases} \text{on time} & |\Delta| \le 30\ \text{min} \ \text{early} & -2\ \text{h} \le \Delta < -30\ \text{min} \ \text{late} & 30\ \text{min} < \Delta \le 3\ \text{h} \ \text{missed} & \text{no press within } 3\ \text{h} \end{cases} $$

A press that matches no scheduled dose is flagged too frequent. The dashboard's adherence rate only counts doses that have already been resolved:

$$ \text{adherence}{7\text{d}} = \frac{n{\text{on time}} + n_{\text{late}} + n_{\text{early}}}{n_{\text{scheduled}} - n_{\text{pending}}} $$

A reminder is built in layers:

  1. A fixed template states the facts (medication, number of drops, which eye) straight from the database.
  2. Gemini writes only the opener and closer around that sentence, and it never sees the instruction itself, so it can't rewrite it.
  3. Guardrails reject any generated line that contains a number, a medication name, medical wording, praise after a missed dose, or too many words. A rejected line is retried once and then replaced with a safe built-in line.

We also wrote an eval set of 12 patient/schedule cases, each run at all three humor levels, that checks every reminder for an exactly preserved instruction, a tone that matches the flag, and no added medical advice.

Challenges we ran into

  • The ESP32 has no real-time clock. If WiFi drops, a queued button press would be saved with the wrong time. So the dock sends how long ago the press happened (age_ms), and the server computes $t_{\text{event}} = t_{\text{server}} - \text{age_ms}$. Presses made offline still get the right timestamp when they're finally sent.
  • Retries vs. real second drops. A deduplication window stops a retried request from logging two doses. At first it was 60 s, which also swallowed the patient's genuine second drop after the 20 s demo countdown. We cut it to 10 s.
  • iPad Safari blocks sound until the user taps the page, so automatic reminders played silently. We added a "Tap to start" screen that unlocks audio once, and every later reminder reuses that same audio element.
  • Auth0 refuses to run over plain HTTP on a LAN IP, which is exactly how you demo on an iPad at a hackathon. We built a clearly labeled local-only dev bypass that is off by default and logs a warning on startup whenever it's on.
  • TimescaleDB won't create continuous aggregates inside a transaction. Our schema runs as idempotent statements, one at a time in autocommit mode.
  • Changing stacks mid-hackathon. We first sketched a SwiftUI app and Arduino C++ firmware, then rebuilt them as a React web app and MicroPython so the whole team could work in languages we shared.
  • Keeping the AI honest about eyes. The photo-analysis prompt took several rounds of tuning so it describes what's visible, admits when an image is blurry, never claims an eye is "healthy", and always points to a professional for anything concerning.

What we learned

  • Separating what is true from how it's said is a simple pattern that makes LLM features safer to ship: let deterministic code own the facts, and let the model own the tone.
  • Hardware without a clock or a reliable network changes how you design an API.
  • Accessibility goes beyond a contrast checker: one action per screen, voice first, and never color alone (every result is a symbol and a word, like ✓ On time or ✕ Missed).
  • Time-series databases fit adherence data well. Hypertables and continuous aggregates let the dashboard stay fast as the data grows.

What's next

  • A load cell under the bottle cup to estimate drops used and predict when a bottle runs low
  • One slot per medication, so the dock knows which bottle was used
  • Push notifications, so reminders arrive even when the app is closed
  • A caregiver view, and review of the timing rules with ophthalmologists

EyeSpy is a hackathon prototype using demo data only. It is not a medical device and not HIPAA compliant.

Built With

+ 1 more
Share this project:

Updates

Submission history