The Blind Spot

Live Camera Vision-impairment Simulator: Experience 59 different vision conditions. Understanding is the first step toward empathy; Share what you experience with others!

Comment

Inspiration

I'm a blind developer. I lost my vision to a stroke a few years ago, and had to adjust from being an avid competitive gamer, Redditor, marathon runner, computer user… to coming back online to an inaccessible internet. I struggled alone during the COVID shutdown, unable to find in-person rehabilitation, therapy, orientation classes, or anything. Looking online, I found next to nothing regarding stroke-related vision loss at the scale I experienced it. Accurately describing how I "see" the world was painfully difficult, and only added to the isolation.

I created The Blind Spot website a couple of years ago, and am now a first-time mobile app creator because I want to create a mobile-first design tool so that people are better equipped with tools to more accurately articulate and represent their blindness and/or vision conditions.

Image-heavy platforms became walls of nothing. Screen readers could only help when someone had bothered to add alt text, and almost no one had.

That experience led me to build AltText Guardian, a Reddit moderator tool that auto-generates alt text for image posts using computer vision. It won Best New Mod Tool at the Reddit Mod Tools Hackathon. But it solved only one side of the problem: making content accessible to people with vision impairments. The other side, the one nobody was building for, was making sighted people actually understand what vision impairment feels like.

According to the WHO, 2.2 billion people worldwide have a vision impairment, and at least 1 billion of those cases were preventable. It is a global health crisis, one that persists in part because the sighted majority has no frame of reference for what these conditions look and feel like.

The Blind Spot exists to close that empathy gap. If you can see through someone else's eyes for even a few seconds, you start to understand. And with understanding comes better design decisions, better policy, better care.

The RevenueCat Shipaton Peace Prize category, "greatest social good," gave me the perfect reason to bring the web-based simulator to mobile, where the phone's camera turns the simulation from a static demo into a lived experience.

What it does

The Blind Spot is a real-time vision-impairment simulator. It renders 59 medically-sourced vision conditions through your phone's camera at 30fps, letting you see your actual surroundings as someone with glaucoma, macular degeneration, cataracts, color blindness, or dozens of other conditions would see them.

It's web-native version lives at The Blind Spot

The core features:

  • Live camera simulation of 59 conditions across 5 families: color vision, acuity and clarity, field loss, distortion, and other visual phenomena
  • Severity slider that lets you experience conditions on a spectrum from mild to severe, because no two patients are alike
  • Overlay builder for composing multiple conditions together (e.g., cataracts + glaucoma, which frequently co-occur in elderly patients)
  • Voice-guided exploration via an ElevenLabs conversational AI agent that can switch conditions, adjust severity, and explain what you are seeing, all by voice
  • Educational profiles of 210+ famous people with vision conditions, searchable and filterable
  • Condition glossary with prevalence data, advocacy links, and medical sources
  • "See What I See" sharing that generates a link others can open to experience the same condition
  • 26-language support with WCAG-compliant accessibility throughout

The web version at theblind.spot supports image and YouTube video input. The mobile app takes it further: the camera makes the simulation visceral and immediate.

How I built it

The ConditionSpec contract

Everything in The Blind Spot flows from one abstraction: the ConditionSpec. Each of the 59 conditions is a JSON file describing an ordered pipeline of image transforms. The schema (v0.1) defines 11 operation types: colorMatrix, blur, fieldLoss (12 shapes), warp (3 kinds), contrast, directionalBlur, ghost, noise, floaters, shimmer, and starburst.

A single spec is the source of truth consumed by three independent interpreters:

  1. Mobile (Skia): TypeScript compiler generates a RenderPlan with SkSL shaders, applied via VisionCamera frame processor worklets
  2. Server (Elixir/libvips): Same semantics, different backend, used for sharing and the MCP surface
  3. Web (GLSL/CSS): Three.js shaders and SVG filters on theblind.spot (source on GitHub)

Golden tests assert perceptual similarity across interpreters. If any interpreter drifts from the schema's semantics, it is the one with the bug.

The rendering math

Operations compose with well-defined algebra. Gaussian blur sigmas combine in quadrature:

$$\sigma_{\text{combined}} = \sqrt{\sum_{i} \sigma_i^2}$$

Standard 4x5 color matrices compose via multiplication ($R = B \circ A$). Machado 2009 color vision deficiency matrices use linearized 3x3 transforms that operate in linear RGB space, requiring sRGB decode/encode in the shader:

$$c_{\text{linear}} = \begin{cases} \frac{c_{\text{sRGB}}}{12.92} & \text{if } c_{\text{sRGB}} \leq 0.04045 \ \left(\frac{c_{\text{sRGB}} + 0.055}{1.055}\right)^{2.4} & \text{otherwise} \end{cases}$$

Contrast amounts multiply and lifts add. The strongest warp wins (stacking reads as noise). Every parameter can be a severity curve, interpolated between 0 and 1.

The mobile stack

Layer Technology
Framework Expo 52, expo-router
Camera react-native-vision-camera 4.6.3
Graphics @shopify/react-native-skia 1.6.0
Worklets react-native-worklets-core 1.5.0
Animation react-native-reanimated 3.16.3
State Zustand 5
Payments react-native-purchases (RevenueCat)
Voice AI @elevenlabs/react-native
Monitoring @sentry/react-native
Notifications OneSignal

The camera pipeline runs entirely on the GPU. The compile.ts interpreter takes condition specs and severity, generates SkSL shader source and a flat uniform array, and the frame processor worklet applies it every frame. No JavaScript bridge crossing in the hot path.

Raxol-powered MCP agent

The Elixir backend uses Raxol to expose the condition library as an MCP (Model Context Protocol) tool surface. The Blindspot.MCP.Tools module defines four tools, annotated with @mcp_tool for auto-derived definitions via raxol_mcp:

  • list_conditions - browse all 59 conditions
  • get_condition - retrieve a full spec by ID
  • list_families - conditions grouped by mechanism family
  • compile_overlay - describe what a multi-condition overlay would look like

This means any MCP-compatible AI agent (Claude, GPT, or custom) can query the condition library, compose overlays, and reason about vision impairments programmatically. The Raxol integration made this nearly declarative: define the tool, annotate it, and the MCP surface is automatically derived.

ElevenLabs voice guide

The mobile app includes a conversational AI voice guide powered by ElevenLabs. The @elevenlabs/react-native SDK establishes a WebRTC session directly between the device and ElevenLabs with no backend in the audio path. The agent registers four client tools (show_condition, adjust_severity, save_overlay, list_conditions) that dispatch directly to the Zustand store, which triggers the Skia pipeline to update in real time.

You can say "show me what glaucoma looks like" and the camera view updates instantly while the agent explains the condition. This is especially valuable for educators and medical students who want a hands-free guided tour.

Monetization

RevenueCat manages a single non-consumable purchase (bs_overlay_builder) that unlocks the full 59-condition library and the overlay builder. Six of the most prevalent conditions (myopia, cataracts, deuteranomaly, hyperopia, glaucoma, diabetic retinopathy) are always free, ensuring the educational core is accessible to everyone.

Web donations go through Stripe Checkout on the share pages, entirely separate from the mobile IAP flow. The "Support the Cause" button on the "See What I See" LiveView page lets web visitors contribute without installing anything.

Challenges I ran into

The biggest problem was ironically the administrative blockers of getting the app live. I'm submitting it via the NextGen track, but still want to enable The Blind Spot in every app store I can, but the administrative processes are really inaccessible. I'm working to add it to platforms such as Uptodown.dev, and soon Apple.

Mobile Testing. This was far and away the most difficult part of the build for me. I've been building out The Blind Spot for a couple years, but have never built a mobile app before. It was always in the plan to have an app, since most the world is mobile-first, so this hackathon pushed me to do so. But frankly, being a blind developer, and testing myself has been a hiccup.

Color accuracy for color vision deficiency. Early versions used simple heuristic color matrices. The results looked wrong to ophthalmologists I consulted. I switched to Machado 2009 simulation matrices, which require operating in linearized RGB space (sRGB gamma decode, matrix multiply, gamma encode). This added complexity to the shader but made the simulations scientifically defensible.

Three interpreters, zero drift. The web simulator, the mobile app, and the Elixir server all interpret the same spec files. Any semantic difference between interpreters is a bug that would silently produce wrong simulations. We solved this with golden tests: run a reference image through all three pipelines and assert perceptual similarity. This caught several subtle issues, like the server applying blur before color matrix while mobile did the reverse.

Worklet closure semantics. VisionCamera frame processor worklets capture variables by copy, not by reference. Passing the compiled render plan into the worklet required shared values from Reanimated, and rebuilding Paint objects inside the worklet on every plan change. This was not documented anywhere and cost a full day of debugging.

Accomplishments that I'm proud of

  • 59 medically-sourced conditions running at 30fps through a phone camera, each backed by cited research
  • The ConditionSpec abstraction is genuinely portable: one JSON file drives rendering on three completely different graphics backends (Skia, GLSL/CSS, libvips) with verified consistency
  • The web platform at theblind.spot already supports 265+ conditions, profiles 210+ famous people with vision conditions, and is available in 26 languages, all MIT-licensed and open source on GitHub
  • The MCP integration via Raxol means any AI agent can programmatically explore and reason about vision conditions, opening doors for research and educational tooling we have not imagined yet
  • The voice guide makes the simulator accessible to people who cannot easily use a touchscreen, which matters deeply for a tool about vision impairment
  • WCAG AA compliance throughout the app, because a tool that teaches accessibility should itself be accessible
  • AltText Guardian (Previous project) won Best New Mod Tool at the Reddit Hackathon. The Blind Spot continues that mission: not just making the internet accessible, but building empathy for why it needs to be

Privacy can be a feature. The Blind Spot has no accounts, no first-party analytics, no personal data collection, and overlays are stored as local JSON. A vision simulator that records what you look at would feel invasive, so the absence of tracking is part of the trust.

What I learned

Other than the unexpected pain of mobile testing, there were a couple:

Technology outpaces accessibility. This was the lesson from AltText Guardian, and it is even more true for vision simulation. The tools to build empathy have existed for years (color matrices, field masks, blur), but nobody packaged them into something a designer or medical student could pick up and use in 30 seconds. The gap is not technical. It is prioritization.

What's next for The Blind Spot

  • Expand the mobile library from 59 to 265+ conditions, matching the web platform's full catalog
  • Narration system with ElevenLabs TTS: condition-specific empathy scripts that play while you experience each simulation, turning it into a guided educational moment
  • Photo mode for capturing and sharing "before/after" comparison images
  • Classroom mode for educators to present simulations to a group, with a teacher dashboard that controls what the audience sees
  • AR overlay using ARKit/ARCore to label real-world objects that would be difficult or impossible to see under each condition
  • Research API extending the Raxol MCP surface to let vision researchers run batch simulations programmatically
  • Partnership outreach with ophthalmology programs, design schools, and accessibility advocacy organizations who could integrate The Blind Spot into their curricula
  • Open-sourcing the mobile app to match the web platform's MIT license, so anyone can contribute conditions, translations, or interpreter implementations

Built With

Share this project:

Updates

Submission history