BrailleVision

"BrailleVision: Making tactile literacy accessible to the sighted."

Comment

Inspiration

Braille is an essential reading and writing system for visually impaired users. However, the support network surrounding these individuals—caregivers, teachers, volunteers, and family members—often lacks the touch-based literacy required to read physical Braille. Our team of four realized that while modern accessibility tech heavily focuses on digital text-to-speech, there is a massive gap when it comes to translating physical embossed documents in the real world. We wanted to build a privacy-first, accessible tool that turns any standard webcam into a real-time translator, bridging the tactile literacy gap.

What it does

BrailleVision is a zero-dependency web application that translates physical embossed Braille into English text and spoken audio. Users simply hold a Braille document up to their webcam. The app captures the video feed, processes the shadows of the embossed dots in real-time, translates the detected Braille cells into English characters, and immediately reads the output aloud. It features a high-contrast dashboard, a customizable shadow sensitivity slider for varying light conditions, and works entirely locally without needing expensive cloud AI APIs.

How we built it

We engineered a full-stack application optimized for ultra-low latency.

  • Frontend: React and Tailwind CSS power the high-contrast UI. We stream frames using navigator.mediaDevices and handle audio via the browser's native window.speechSynthesis API.
  • Backend: A local FastAPI (Python) server handles the heavy lifting using OpenCV (opencv-python-headless).
  • Vision Pipeline: We apply Grayscale and Gaussian Blur, followed by cv2.adaptiveThreshold to isolate the highlights/shadows of physical dots. We then filter noise using dynamic contour area constraints.

The Mathematics of Scale-Invariant Cell Grouping: Because users hold paper at varying distances, hardcoding pixel limits fails. We developed a scale-invariant grouping algorithm that dynamically calculates the median dot spacing to cluster dots into 2x3 Braille cells.

Once a cluster of dots is isolated into a bounding box of width $W$ and height $H$, we determine the precise index of a detected dot at coordinate $(c_x, c_y)$ by calculating its relative spatial position:

$$x_{rel} = \frac{c_x - x_{min}}{W}, \quad y_{rel} = \frac{c_y - y_{min}}{H}$$

These continuous relative coordinates are then quantized into a discrete $2 \times 3$ binary matrix. For example, the column assignment $C \in {0, 1}$ and row assignment $R \in {0, 1, 2}$ are determined by step functions:

$$C = \begin{cases} 0, & \text{if } x_{rel} \le 0.5 \ 1, & \text{if } x_{rel} > 0.5 \end{cases}$$

$$R = \begin{cases} 0, & \text{if } y_{rel} < 0.33 \ 1, & \text{if } 0.33 \le y_{rel} < 0.66 \ 2, & \text{if } y_{rel} \ge 0.66 \end{cases}$$

This mathematical mapping generates a 6-bit array (e.g., [1,0,0,0,0,0] for 'a') regardless of camera resolution, which is then mapped to our English dictionary.

Challenges we ran into

  1. Digital vs. Physical Artifacts: When testing with digital Braille images, the system detected the tiny visual placeholder dots in empty cells alongside the actual dots (resulting in false 6-dot cells). We solved this by implementing a dynamic area filter that calculates the maximum contour area in the frame and ignores any dots less than 30% of that size.
  2. Grade 1 vs. Grade 2 Braille: Physical Braille is often written in "Grade 2" (contracted Braille) to save space, where single letters represent entire words (e.g., the letter 'y' means "you"). We discovered this live when our engine translated a test image literally as "y c d x" instead of "you can do it". Building a context-aware NLP parser within a 2-hour sprint was impossible, so we optimized the prototype to flawlessly handle Grade 1 literal translation as a stepping stone.
  3. Variable Lighting: Webcams and physical paper introduce massive lighting variance. We overcame this by exposing a "Shadow Contrast Sensitivity" slider in the React UI, allowing users to manually calibrate the OpenCV thresholding constants in real-time.

Accomplishments that we're proud of

  • Building a highly resilient mathematical grouping algorithm that dynamically scales to the camera's distance from the paper.
  • Creating a 100% zero-dependency architecture. By using local OpenCV thresholding and the browser's native Web Speech API, our app is lightning fast, privacy-respecting, and costs $0 to run.
  • Designing an accessible, screen-reader-friendly React dashboard with a dedicated high-contrast mode.

What we learned

We gained a profound appreciation for classical computer vision. While it is tempting to throw a massive LLM at every image recognition problem, we learned that a well-crafted algorithmic pipeline using spatial mathematics is drastically faster and far more efficient. We also deepened our understanding of Braille complexities (Grade 1 vs. Grade 2) and how to effectively stream and throttle data between a React frontend and a Python backend.

What's next for BrailleVision

Our immediate next step is expanding our dictionary logic to support Grade 2 contracted Braille, allowing the app to parse shorthand like 'y' into "you" based on sentence context. Furthermore, we plan to package the React frontend into a Progressive Web App (PWA) and deploy the OpenCV backend to an edge-computing environment, allowing caregivers to use BrailleVision entirely offline on their mobile devices.

Built With

Share this project:

Updates

Submission history