GozVision

An autonomous agentic vision copilot transforming mobility for 2.2 billion low-vision individuals through closed-loop OpenCV 5 depth perception, scotoma-adaptive contrast, and clinical intelligence.

Comment

Inspiration

Over 2.2 billion people globally live with vision impairment (WHO). For individuals diagnosed with Age-Related Macular Degeneration (AMD), Glaucoma, and Cataracts, navigating ordinary sidewalks, curbs, and stairs is fraught with danger.

However, assistive navigation systems have long suffered from two critical technical bottlenecks:

  1. The Classical 2D Computer Vision Failure Mode: Conventional edge detectors (Canny, Sobel, Laplacian) operate purely on luminance gradients. To a 2D filter, a 15 cm physical step-down curb looks mathematically indistinguishable from a flat painted bicycle lane stripe or a dark tree shadow. Both produce high-gradient edge contours. For low-vision patients, this causes constant false alarms or catastrophic missed drops.
  2. The Clinical Disconnect & Privacy Compromise: Generic computer vision algorithms apply uniform contrast enhancement to every user, completely ignoring the patient's individual ophthalmic profile. An AMD patient with a central scotoma needs peripheral contrast; a glaucoma patient with tunnel vision needs central clarity. Furthermore, sending continuous video frames of patients' private living quarters or exam rooms to third-party cloud GPUs creates unacceptable HIPAA and privacy exposure.

We built GozVision to solve both challenges: an autonomous, clinical-grade Agentic Vision Copilot built natively on OpenCV 5.0, Universal Intrinsics 2.0 on AWS Graviton, Air-Gapped Local WebGPU AI, and Closed-Loop Spatial Audio Feedback.

What it does

GozVision unites real-time edge computer vision with clinical EHR intelligence across two connected surfaces:

  • Goz Patient Copilot (B2C PWA / Mobile):

    • OpenCV 5 DNN Spatial Depth Disparity: Uses monocular depth estimation and perspective geometry to compute the vertical derivative of disparity ($\frac{\partial D}{\partial y}$) in the walking corridor ($y \in [0.55H, 0.95H]$). It flags physical elevation drops ($15.0\text{ cm}$ curbs and steps) while mathematically proving that flat painted lines and shadows are coplanar ($\Delta D \approx 0$).
    • Spatial Directional Acoustic Cues & Voice: Translates the 3D hazard vector $(\Delta x, \Delta y, \Delta D)$ into directional auditory cues and spoken warnings ("Caution: 15cm step down to your left"), allowing patients using open-ear smart audio frames (such as Solos AirGo 3) to intuitively perceive obstacle lateral angle and drop depth.
    • Specular Glare Suppression & CLAHE: Attenuates specular highlights caused by wet pavement and headlights for cataract patients while boosting local luminance contrast in LAB color space.
    • Morphological Reading Zone Isolation: Automatically isolates high-contrast text regions on medication bottles and charts for text-to-speech audio readout.
    • Agentic Fall Safeguard: OpenCV 5 hazard detections alter the agent's goal state in real time. If wearable accelerometer telemetry registers a stumble immediately following a hazard alert, GozVision autonomously dispatches an encrypted emergency SOS alert with GPS coordinates and spatial telemetry.

  • Goz Clinic Suite (B2B Practice Management Web Hub):

    • Ambient Clinical Scribe with Local WebGPU Privacy Sovereignty: Synthesizes doctor-patient dialogue into structured FHIR R4 and SOAP notes. Doctors can toggle seamlessly between Cloud AI (HIPAA BAA zero-retention) and Air-Gapped Local WebGPU AI (100% on-device inference with zero network packets) for complete clinical privacy.
    • Diagnosis-Adaptive Vision Enhancement (DAVE): Ingests the patient's ICD-10 diagnosis codes (e.g., AMD H35.31, Glaucoma H40.11, Cataract H26.9, RP H35.52) and Humphrey Visual Field (HVF) mean deviation from the EHR to dynamically configure the patient's computer vision filters to their exact functioning retina.

How we built it

GozVision was architected as a modular, high-throughput monorepo deployed on AWS and Supabase:

  1. OpenCV 5 Vision Pipeline (backend/gozvision_vision/cv_pipeline.py):

    • Stage 1 (Optical Pre-processing): Converts BGR frames to LAB color space. CLAHE is applied exclusively to the Luminance ($L$) channel with condition-adaptive clip limits, preserving chromaticity. Glare masks are attenuated via morphological dilation and highlight dimming.
    • Stage 2 (DNN Depth Disparity Engine): Loads ONNX depth models via cv2.dnn.readNetFromONNX with DNN_BACKEND_OPENCV.
    • Stage 3 (Vertical Drop vs. Shadow Classifier): Computes the vertical depth disparity gradient: $$\nabla_y D(x, y) = \frac{\partial D}{\partial y}$$ A discontinuous depth step ($\frac{\partial D}{\partial y} > 0.08$) confirms an abrupt physical drop-off, triggering drop height estimation in centimeters. Flat shadows exhibit continuous coplanar depth surfaces ($\frac{\partial D}{\partial y} \approx 0$), eliminating false positives.
    • Temporal Hazard Coherence (TemporalHazardCoherenceTracker): A 5-frame rolling window enforces geometric consistency ($\ge 0.60$), provably suppressing transient shadow flicker without introducing perception latency.

  2. Cloud-Optimized OpenCV (COOL) on AWS Graviton (backend/gozvision_vision/cool_pipeline.py):

    • Tuned for ARM Neoverse cores on AWS Graviton3 (c7g.xlarge) and Graviton4 (c8g.xlarge) using Universal Intrinsics 2.0 with ARM NEON and SVE SIMD vectorization.
    • Supports CV_8U, CV_32F, and CV_16BF datatypes with zero-copy frame buffering in Linux/Arm64 containers.

  3. Closed-Loop Agentic Vision & Spatial Web Audio (backend/gozvision_agent/agentic_vision.py & apps/patient/lib/services/):

    • Implements multi-step perception-to-action workflows where OpenCV 5 feature extraction directly controls downstream tools (send_sos_alert, log_adherence_event, and clinical SOAP note drafting).
    • Generates directional audio cues via the Web Audio API (AudioContext, gain envelopes, and synthetic speech).

  4. Cloud Infrastructure & Deployment:

    • Amazon ECS Fargate: Runs containerized FastAPI services (gozvision-cluster).
    • AWS CloudFront: Provides global TLS 1.3 edge termination (d7icpdzxenshi.cloudfront.net).
    • AWS Amplify: Hosts responsive Flutter PWA web apps (main.d1lr4zsb46he1m.amplifyapp.com).
    • Supabase PostgreSQL 16: Enforces multi-tenant Row-Level Security (RLS) for clinical and patient data isolation.

Challenges we ran into

  1. Distinguishing Real Shadows from Curbs: Single-frame 2D edges produced identical edge signatures for curbs and shadows. We overcame this by moving to a monocular depth disparity gradient $\nabla_y D$ and layering a 5-frame temporal coherence tracker. Geometrically stable steps maintain $1.00$ coherence, while fluctuating ground shadows drop to $0.20 - 0.40$ and are suppressed as coplanar_clear.
  2. Lateral-Only Curb Boundary Dropouts: Initial testing revealed that curbs situated entirely in the left or right third of the pedestrian corridor produced zeroed bounding boxes when averaged over full corridor widths. We resolved this by implementing an active drop mask (active_drop_mask) that isolates lateral hazard clusters independently, calculating precise lateral centroids for directional panning.
  3. Evaluating Vision Quality for Scotoma Patients: Standard full-frame SSIM actually penalizes adaptive enhancement because it averages pixel changes over blind retinal regions. We formulated the Sighted-Region SSIM and Contrast Improvement Index (CII), weighting metrics only across functioning retinal areas derived from patient visual field tests.
  4. Balancing Cloud Ingress Costs for Video Streaming: Streaming continuous frames to cloud GPUs is economically non-viable for non-profit and healthcare settings. By compiling OpenCV 5 with Universal Intrinsics 2.0 on AWS Graviton3, we achieved sub-6ms latency on CPU alone, processing 1 million frames for less than 24 cents.

Accomplishments that we're proud of

  • Empirically Proven AWS Graviton Economics: Our automated benchmarks (backend/benchmark_results.json) demonstrated:
    • Spatial Depth Disparity: 199.3 FPS at $0.2369 per 1M frames
    • Enhanced Contrast (CLAHE): 167.0 FPS at $0.2827 per 1M frames
    • Glare Suppression: 506.8 FPS at $0.0932 per 1M frames
    • Text Reading ROI: 873.1 FPS at $0.0541 per 1M frames
    • Graviton3 (c7g.xlarge) vs. Intel Xeon (c6i.xlarge): Achieved 1.34x throughput gain, 25.4% latency reduction, and a 36.8% net cloud cost reduction.
  • Real-Frame Empirical Evaluation Harness: We built a reproducible evaluation harness running against 69 real-world street and sidewalk images with zero synthetic placeholders (backend/evaluation/run_evaluation.py).
  • First Clinical-to-Vision Adaptive Feedback (DAVE): Patients with AMD achieved a 2.51x Edge Energy Ratio (EER) boost targeted specifically in their peripheral annulus, while Glaucoma profiles concentrated contrast gain exclusively within their central visual field.
  • Privacy Sovereignty with WebGPU: Providing an instant runtime toggle between cloud HIPAA BAA processing and 100% on-device WebGPU execution, ensuring zero biometric or clinical video frames ever leave the room when required.
  • Production-Ready Live Infrastructure: The system is deployed and active on AWS CloudFront, ECS Fargate, and AWS Amplify with verified HTTP 200 health endpoints.

What we learned

  • Perception Must Drive Actuation: Assistive computer vision cannot stop at displaying visual overlays; patients with severe vision loss cannot interpret complex visual bounding boxes. True accessibility requires an agentic closed loop that converts spatial perception into immediate directional auditory, haptic, and emergency actuation.
  • CPU Vectorization Matches or Beats Edge GPUs for Preprocessing: With OpenCV 5 Universal Intrinsics 2.0 on ARM Neoverse Graviton cores, optimized CPU SIMD execution delivers sustained 199+ FPS for depth disparity without the power draw or cost penalty of discrete GPUs.
  • Clinical EHR Context Transforms CV Accuracy: When computer vision algorithms know a patient has central scotoma or diabetic retinopathy before processing the first frame, enhancement parameters can be targeted with surgical precision.

What's next for GozVision

  • Multi-Center Clinical Trials & FDA SaMD Pre-Submission: Partnering with low-vision clinics and optometry schools to measure real-world reduction in the 29% 90-day assistive device abandonment rate and patient fall incidents.
  • Direct Micro-Firmware AR Waveguide Integration: Compiling our OpenCV 5 edge CLAHE and drop-disparity pipelines directly to emerging ultra-lightweight smart glass microcontrollers (e.g. customized firmware for Solos AirGo and Brilliant Labs Frame).
  • Expansion to Pediatric & Neuro-Ophthalmic Visual Rehabilitation: Extending our DAVE condition profiles to support hemianopia from stroke recovery and pediatric amblyopia contrast therapy.

Built With

Share this project:

Updates

Submission history