Vital Steps

Inspiration

70% of cardiac arrests outside hospitals go untreated — not because bystanders don't care, but because they were never taught what to do. Medical training today is locked behind classrooms, textbooks, and expensive certifications. We asked ourselves: what if anyone could practice life-saving skills just by putting on a pair of glasses? VitalSteps was born from the belief that medical literacy shouldn't require a medical degree but should be as accessible as the device on your face.

## What it does VitalSteps is an augmented reality medical training platform built for Snap Spectacles that teaches users essential clinical skills through hands-on practice. The app includes three training modules for now:

Pulse Checking — guides users through proper radial pulse technique using real-time hand tracking, times their count, and delivers instant clinical feedback on their BPM reading
Vein Detection — scans the user's forearm to identify the optimal vein for IV access or blood draws, projecting an AR marker directly onto the detected site
Prescription Validation — uses AI-powered image recognition to identify medicine bottles and prescriptions, reading back drug names, dosages, and usage instructions

As users complete each module, they earn items for a virtual 3D first aid kit registered as a verifiable credential on the Solana blockchain. The 3D first aid kit can be inspected, rotated, and scaled with two-hand pinch gestures on the Progress screen.

How we built it

VitalSteps runs entirely on Snap Spectacles, built with TypeScript in Lens Studio 5.15. We leveraged the SpectaclesInteractionKit (SIK) for robust hand tracking and SpectaclesUIKit for intuitive interfaces.

Pulse & Vein Detection: Our pipeline reads 25 keypoints per hand at 60fps. We use finger-to-wrist proximity calculations for pulse detection and wrist-to-knuckle vector extrapolation to project AR markers onto the forearm. For vein detection, a YOLOv7-based model runs through SnapML to analyze the camera feed, identify subcutaneous vein patterns, and isolate the median cubital vein. A 3-second stabilization gate filters out noise and prevents false positives.

Prescription Validation: We capture high-resolution still frames from the Spectacles, encode them to base64, and tunnel them through Cloudflare to our Node.js/Express backend. There, Google's Gemini 2.0 Flash vision model analyzes the frame to verify the medicine label and extract critical drug data.

Voice & Verification: We integrated Spectacles' native ASR (Automatic Speech Recognition) for voice commands and Text-to-Speech (TTS) for audio guidance. Finally, we mint session-specific Solana token IDs to create tamper-proof, portable credentials for completed training modules.

Challenges we ran into

Network Constraints on Wearables: The deprecation of the RemoteServiceModule API in favor of the HTTPS-only InternetModule meant local testing broke on-device. We circumvented this by routing our backend through Cloudflare tunnels, but diagnosing the resulting HTTP status 0 chains required heavy debugging.

Training and Setting up the Yolo7 model on hardware was a big challenge and took most of our time.

Accomplishments we're proud of

Both of us came in with zero XR experience and had to go through a steep learning curve. We had to pivot our project idea completely on day 2 because we failed to get our hardware running the first time. Having a fully functional application and learning so much about AR technology.