ShiftSense

Contactless 15-second wellness check-ins for clinicians using camera-based vitals sensing and Gemini AI — no wearables, no friction, built for real clinical shifts.

Comment

Inspiration

Clinician burnout costs the US healthcare system an estimated $4.6 billion per year, yet most wellness interventions require dedicated wearables, scheduled appointments, or significant friction to use during an active shift. As someone with a background in clinical and health informatics, I wanted to build something that translates complex physiologic data into clear, actionable insights without adding burden to an already demanding workflow.

What it does

ShiftSense is an iOS app that gives clinicians a 15-second contactless wellness check-in during their shift using only their phone's front camera. No wearables. No extra hardware.

Each check-in measures pulse rate and breathing rate via the Presage SmartSpectra SDK, then feeds those readings into the Gemini 1.5 Flash API to generate a personalized clinical-style interpretation contextualized to the clinician's shift type and prior strain history.

Results are classified as "Calm," "Elevated," or "Strained" and stored persistently across the shift. ShiftSense tracks cumulative patterns (not just single readings) so it can detect when strain is repeating rather than isolated. If a clinician scans as Strained, the app automatically surfaces an adaptive recovery plan with breathing resets, hydration prompts, and micro-movement suggestions scaled to their current wellness level.

Given the wide scope, ShiftSense can be applied for burnout prevention for roles outside of clinicians too (including roles like students, remote workers and many more).

How I built it

  • Presage SmartSpectra Swift SDK for camera-based pulse and breathing rate measurement
  • Google Gemini 1.5 Flash API for real-time AI interpretation of vitals in clinical context
  • Swift / SwiftUI for the full iOS UI including live camera feed, countdown, result cards, and trend charts
  • Swift Charts for shift-level pulse and breathing trend visualization
  • UserDefaults for persistent check-in history across sessions

Challenges I ran into

Presage only provides native mobile SDKs (there is no web SDK) so the entire app had to be built as a native iOS application deployable to a physical device. Camera-based vitals measurement also cannot run in the simulator, which made every iteration require a full build and deploy cycle to hardware. The limited amount of free tokens was also a constraint that needed to be taken into consideration throughout the developmental process.

Gemini API rate limits on the free tier were exhausted during testing, requiring a model switch mid-build from "gemini-2.0-flash' to "gemini-1.5-flash-8b" and a fresh API key to restore functionality.

Keeping API keys out of the public repository while maintaining a clean setup experience for judges required careful ".gitignore" configuration and a companion "Secrets.example.swift" pattern.

Accomplishments that I'm proud of

The end-to-end flow works on a real device: a clinician opens the app, selects their shift, runs a 15-second scan, and within seconds receives a Gemini-generated plain-language interpretation of their vitals that accounts for how many times they have been strained that shift.

That full pipeline Camera → vitals → AI insight → recovery action runs in under 20 seconds with no wearables and no manual data entry.

What I learned

Translating raw physiologic signals into clinically meaningful language is exactly the kind of problem health informatics exists to solve. Pairing a sensing SDK with a large language model creates a surprisingly effective layer of interpretation. The numbers alone mean little to a busy clinician, but a two-sentence contextual summary they can act on immediately changes the utility entirely.

What's next for ShiftSense

  • Backend proxy for API keys to remove client-side credential exposure
  • Push notifications for repeated strain patterns across a shift
  • Anonymized shift-level dashboards for charge nurses or department wellness coordinators
  • Apple Watch integration for passive continuous monitoring between active check-ins
  • Validation study with clinical staff to calibrate wellness thresholds against self-reported burnout scores

Built With

  • combine
  • google-gemini-api
  • ios
  • presage-smartspectra-sdk
  • swift
  • swift-charts
  • swiftui
Share this project:

Updates