MLH Tech: ElevenlLabs, Presage, Gemini API
Inspiration
Concussions are massively under-diagnosed in youth and amateur sports. On the sideline, the person deciding "can they go back in?" is usually a coach or parent with no clinical tools, just a gut check. Many concussions have no dramatic symptoms and slip through standard checks. Sending an athlete back in while their brain is still vulnerable is dangerous: a second impact can trigger Second-Impact Syndrome, and repeat trauma sharply raises the risk of long-term cognitive damage.
We wanted to bring real clinical screening tools to the sideline using hardware a team could actually carry with them. RapidGlass runs a SCAT5-aligned test battery across four devices, flags whether a player should see a doctor, and generates a report that medical staff can pull up instantly via NFC.
What it does
RapidGlass runs on four connected devices:
- Smart glasses: inward-facing IR camera plus mic/speaker, streamed from an ESP32-Sense webserver, capturing pupil and gaze data.
- Laptop: runs the Presage SDK on incoming video frames to extract heart rate and HRV.
- Raspberry Pi (QNX): the brain. Runs pupil detection, scores Stage 1, orchestrates Stage 2 when needed, fuses all signals, narrates the report with Gemini and ElevenLabs, and serves the report page.
- Tablet: shows visual stimuli and tracks head pose via camera for the balance and VOR sub-tests.
The screening flow:
Stage 0: A coach or guardian runs the standard field assessment for obvious, severe injuries. If the player isn't hospitalized, they move to Stage 1. Stage 1 (glasses + tablet + Pi): measures pupil diameter/variance, blink rate and duration, PLR amplitude, and HR/HRV, scored against fixed reference thresholds. If the result is clearly normal or clearly abnormal, the system goes straight to the report. Stage 2 (tablet + glasses + Pi), only if Stage 1 is ambiguous: four sub-tests, balance (20s of head sway tracked by the tablet camera), smooth pursuit (gaze velocity tracked against a moving dot), VOR (gaze stability during head motion), and recall (five words spoken by ElevenLabs, recalled after a delay).
If Stage 1 is confident, it drives the report alone. Otherwise Stage 1 and Stage 2 are combined. The final report is a web page hosted on the Pi, pulled from an encrypted database and opened via NFC tap on the patient's or medical personnel's mobile device.
How we built it
Networking: every device (glasses, laptop, Pi, tablet, phone) joins one personal hotspot. The Pi hosts the database and report page. The glasses' ESP32-Sense hosts its own capture endpoint that the laptop and Pi pull frames from. QNX + TensorFlow Lite: to meet the requirement of building on a published oss.qnx.com AI module, we built pupil detection on the QNX TensorFlow Lite + QSF sample, testing MediaPipe live with OpenCV as a fallback. Presage on the tablet: Presage's SDK doesn't run on QNX, so HR/HRV extraction runs on the tablet instead of the Pi. Tablet reuse: the tablet reuses the same head-tracking script pattern as the Presage pipeline for balance and VOR, instead of a separate stack. Gemini + ElevenLabs: Gemini analyzes the fused metrics to generate plain-language narration. ElevenLabs handles all spoken prompts, including the recall word list. Report delivery: a report page served from the Pi, pulled from an encrypted database, with ngrok as a fallback tunnel if a device needs access from outside the hotspot.
Challenges we ran into
- Syncing four devices, each capturing a different signal on a different clock, over one shared hotspot with no wired backbone.
- Getting CV tooling to run on QNX, which has little native support for MediaPipe or OpenCV, and needing a real fallback ready in case MediaPipe failed.
- Moving Presage's HR/HRV pipeline onto the laptop after discovering it doesn't run on QNX, without breaking fusion downstream.
- Scoring sub-tests without per-patient baselines, since the demo uses fixed thresholds instead.
Accomplishments that we're proud of
- Built a fully complete product/system and hit every goal we set, with minimal scope cuts. Synced four physical devices into one pipeline using nothing but a phone hotspot. Got a real CV pipeline running on QNX, a platform none of us had touched before this weekend. Built a genuine staged clinical flow (Stage 1 gate, conditional Stage 2, fusion) instead of one flat scoring pass.
- Enforced the no-diagnosis safety rule end to end, with hard overrides for anisocoria and severe VOR deviation.
What we learned
- Multi-device hardware systems carry real coordination overhead.
- The practical limits of running CV/ML tools on QNX.
- How to design scoring (like gaze-velocity cross-correlation) that works without calibration we didn't have time to build.
- Why safety constraints need to be designed in from day one, not added later.
What's next for RapidGlass
- Additional IR camera for better recognition of uneven pupil sizes.
- Introduction of a more compact design and usability.
- Balance the weight of the glasses, i.e polish design.
- Add an accelerometer to measure for abnormal vertigo.
- Add additional clinically backed data and validate.
Log in or sign up for Devpost to join the conversation.