IrisAgent

Inspiration

Millions of people with ALS, spinal cord injuries, or locked-in syndrome lose the ability to move and speak. Existing assistive devices are expensive or rigid. We wanted to see if a standard dual-camera setup could restore mobility, voice, and environment control using only the eyes.

What it does

IrisAgent turns eye gaze, blinks, and mouth movements into control:

• Mobility: Gaze left, right, or straight steers a wheelchair.
• Agentic Vision: Hold a gaze on a room zone for 5 seconds. The system identifies the object and asks for confirmation before acting.
• Morse Code: Blinks become text, spoken aloud.
• E-Stop: Eyes closed for 10 seconds alerts caregivers by SMS.

How we built it

A laptop runs the eye and room tracking with YOLO11 and a gaze classifier. An AWS EC2 Graviton3 server runs the COOL-optimized OpenCV 5 pipeline for faster inference. Amazon Polly, SNS, and DynamoDB handle speech, alerts, and logging.

Results

COOL reduced preprocessing latency by 36% and end-to-end server latency by 13% compared with standard OpenCV on the same instance. The eye-state detector reached 85% mAP50, and the gaze classifier reached 90% accuracy.

Challenges

• Mixed performance: COOL sped up preprocessing but slowed postprocessing slightly, so we report both results.
• Lighting: Low light and glare reduce tracking accuracy.
• Cloud rate limits: Rapid requests triggered HTTP 429 errors, which we solved with a cached response window and a fallback.

What we learned

Measure every stage, not just the total. Good camera setup and lighting matter as much as the model. Time-critical functions like the E-Stop must work even when the network doesn't.

Built With

• accessibility
• amazon-dynamodb
• amazon-polly
• amazon-sns
• amazon-web-services
• blink-detection
• computer-vision
• cool
• ec2
• edge-ai
• eye-tracking
• flask
• gaze-detection
• gemini-domain-assistive-technology
• graviton3
• healthcare
• human-computer
• onnx
• opencv
• tech-python
• yolo