IRIS | Devpost

3d mask

Inspiration

Over 2.2 billion people worldwide live with vision impairment, and nearly half of those cases were preventable. In the U.S. alone, 4.2 million Americans aged 40+ have uncorrectable vision loss, and 6.8% of children under 18 have a diagnosed eye condition. Yet most people never get screened until it's too late. We believe the eyes are the window to the soul, and they deserve better care. That's what inspired us to build Iris: accessible, AI-powered eye health from the comfort of home.

What it does

Iris is an at-home eye care platform powered by a personalized AI assistant. It has three core features:

Eye Disease Detection - Users upload a photo of their eye and Iris analyzes it for signs of potential external eye conditions, returning a prediction with a confidence score. (Results are informational and not a medical diagnosis.)

Eye Screening - Iris guides users through an at-home vision screening using any device, phone or computer. Using eye tracking and interactive tests, it estimates indicators of myopia and astigmatism without a clinic visit.

Personalized AI Assistant - After both exams, Iris synthesizes the results into a personalized health report. Users can ask follow-up questions in a chat interface, listen to a voice-narrated summary of their findings, and receive a tailored care plan including eye exercises, contact lens and glasses care tips, and daily habits, all based on their individual results.

Another feature that further support the user experience, when Iris detects a potential condition or disease, it doesn't just stop at the report. It helps users locate nearby eye care specialists so they can take the next step toward professional treatment, bridging the gap between at-home screening and clinical care.

Iris exists to make eye care accessible to everyone. Children often can't articulate when their vision is blurry or something feels wrong, Iris gives parents a simple way to screen early. In rural and underserved communities where eye exams are expensive or hard to reach, Iris provides a first line of awareness. From detection to education to finding professional help, Iris is a complete at-home eye care companion powered by personalized AI. We will go into details on our features that we are very proud of.

How we built it

Languages

TypeScript - Next.js pages, API routes, config JavaScript - Components, lib utilities, custom server Python - Standalone desktop eye-tracking script (exam/main.py) CSS - Tailwind v4 globals, CSS Modules HTML - Static pages in public/ (exam-app.html, astig-test.html)

Frontend Framework & Runtime

Next.js 16.1.6 - App Router, React Server Components, API Route Handlers React 19.2.3 + React DOM 19.2.3 Node.js - runtime for Next.js and the custom WebSocket server

Styling & Animation

Tailwind CSS v4 - via @tailwindcss/postcss Framer Motion 12.35.1 - motion, AnimatePresence, useMotionValue used across pages CSS Modules - ExamTracker.module.css Google Fonts (Geist + Geist Mono) - via next/font/google Inline styles used heavily for the custom design system

Database

MongoDB Atlas - cloud-hosted cluster (HackAI database) mongodb npm package v6.21.0 - official Node.js driver, singleton clientPromise in src/lib/mongodb.js

AI / ML APIs

Google Gemini (gemini-2.5-flash) Gemini powers three distinct features: Eye Disease Detection: The user uploads a photo of their eye. The image is converted to base64 and sent to Gemini Vision along with a clinical prompt asking it to identify signs of external eye disease (conjunctivitis, cataracts, styes, etc.) and return a confidence score with reasoning.

Eye Screening Analysis: After the user completes the vision tests (Tumbling E chart, astigmatism clock dial, myopia near-point test), the raw results, acuity scores, axis readings, diopter estimates, are passed to Gemini with the user's age, screen time, and known conditions. Gemini synthesizes this into a structured clinical report with a differential diagnosis, risk scores across 6 dimensions, and a 5-year projection.

AI Chat Assistant: Users can ask follow-up questions about their report. Each message is sent to Gemini with the full report as context, so answers are personalized to their specific findings rather than generic eye health advice.

All Gemini calls are routed through a server-side Next.js API route (/api/gemini) so the API key is never exposed to the client.

ElevenLabs TTS, Voice Narration After the report is generated, Iris reads it aloud using ElevenLabs text-to-speech. The report is broken into individual lines and each line is sent to the ElevenLabs API (/api/tts) using the eleven_turbo_v2 model, which returns an MP3 audio buffer. The app chains these audio clips sequentially with a live narration bar at the bottom of the screen showing the current line being read. This is triggered by a manual play button to comply with browser autoplay policies.

The voice narration is also used during the eye screening process, Iris speaks each instruction aloud, acting like a doctor guiding the user through the exam step by step.

Computer Vision / Eye Tracking (Browser)

@mediapipe/tasks-vision v0.10.32 - During the eye screening, the front-facing camera is activated and MediaPipe Face Mesh tracks the user's eye position and gaze. This enables the strabismus check (detecting eye misalignment during the cover test) and estimates the user's viewing distance for calibrating the visual acuity test, the same methodology validated in a Harvard Medical School / Massachusetts Eye & Ear smartphone vision screening study published in JMIR Formative Research. FaceLandmarker - 468-point face mesh + refined iris landmarks (points 468 & 473) HandLandmarker - 21-point hand skeleton tracking WASM binaries served locally from public/mediapipe-wasm/ (SIMD + non-SIMD) Runs on GPU delegate inside requestAnimationFrame loop

Challenges we ran into

Building the Eye Tracking System One of our biggest technical challenges was implementing the eye screening feature. Rather than guessing, we grounded our approach in peer-reviewed research, specifically a Harvard Medical School pilot study on smartphone-based vision screening published in JMIR Formative Research. The study validated smartphone apps for measuring ocular alignment and refractive error without requiring specialized equipment, which gave us a scientific foundation to build from. Translating that research methodology into a real-time, browser-based implementation, handling camera access, eye detection, and movement tracking across different devices, proved to be one of the most technically demanding parts of the project.

Responsible AI Output Designing the AI-generated report required careful thought. We needed results that were informative and actionable without crossing into clinical diagnosis territory. Striking that balance — giving users meaningful insight while clearly communicating that Iris is a screening tool, not a substitute for a professional exam, required multiple iterations of our prompting and UI design.

Accomplishments that we're proud of

Eye Screening & Disease Detection Both models are fully functional. Users can go through a real at-home vision screening and upload an eye photo for disease detection, with results that reflect genuine analysis rather than placeholder output.

Google Gemini Vision We made deep use of Gemini's multimodal capabilities, using the vision model to analyze eye images and generate personalized, medically-informed reports from screening results. Gemini doesn't just power the backend, it's the intelligence behind every insight Iris delivers.

*ElevenLabs Voice Integration * This is what we're most proud of. We didn't just use ElevenLabs to read a report out loud. Iris uses voice to guide users through the entire experience, narrating the eye screening process step by step, coaching users through eye exercises, and delivering their personalized report as a spoken summary. It transforms Iris from an app you read into one that feels like having a knowledgeable companion by your side.

What we learned

Eye Screening Technology This was our first time building a computer vision model of this complexity. Implementing real-time eye tracking and vision estimation from scratch, grounded in clinical research, taught us how much engineering goes into what looks like a simple camera interaction. We came away with a deep respect for the science behind vision screening. Eye Health We started this project knowing the statistics. We ended it understanding the conditions behind them. Researching myopia, astigmatism, strabismus, amblyopia, and external eye diseases gave us a genuine appreciation for how much goes undetected simply because screening is inaccessible. Building Iris made us more informed about our own eye health too.

What's next for Iris

Increased Screening AccuracyOur eye screening model is functional, but accuracy is everything in health technology. The next phase of development focuses on refining the computer vision model with more robust testing, broader device compatibility, and validation against clinical benchmarks. The goal is a screening tool users can genuinely rely on, not just a prototype that demonstrates the concept.

A Deeper, More Personalized Care Plan Right now Iris gives every user a care plan informed by their results. We want to go further, tracking changes over time, adjusting recommendations as users complete exercises and report back, and building a plan that evolves with the user rather than being a one-time output. Think less like a report and more like a personal eye health coach that gets smarter the longer you use it. These two improvements, accuracy and personalization, are the foundation Iris needs to grow from a hackathon project into something people genuinely depend on for their eye health.