Braille Vision Scanner

It's a system that takes an image of physical Braille dots and converts them to spoken English using computer vision

Comment

Inspiration

Many visually impaired people rely on Braille for reading, but caregivers, teachers, and volunteers often can't read Braille themselves is what I realized on researching on this topic. So, the theme of hackathon could bridge that gap which in itself is a noble cause. Also since I never worked on this topic ever before I was inspired to see if my contribution could help in even remote way.

So with that that thought I wanted to build something that treats Braille as what it is a spatial arrangement of dots and uses computer vision to decode it.

What it does

Braille Vision Scanner takes an image of physical Braille dots and converts it to spoken English through a multi-stage computer vision pipeline:

  1. Preprocesses the image (brightness normalization, noise reduction, adaptive thresholding)
  2. Detects individual dots using contour analysis with circularity classification
  3. Segments dots into 2×3 Braille cell grids using spatial clustering
  4. Decodes each cell pattern to English using a Grade 1 Braille lookup table with state machine for number/capital indicators
  5. Speaks the result aloud via Text-to-Speech

How we built it

Backend (Python + FastAPI):

  • OpenCV for all image processing — thresholding, contour detection, morphological operations
  • Custom dot classifier using three geometric metrics: Circularity, Convexity and Inertia Ratio
  • Lighting-aware thresholding that compares brightness across image quadrants:
    • Otsu's global threshold
    • Adaptive Gaussian threshold
  • Natural-break algorithm for cell segmentation
  • WebSocket streaming for real-time frame processing at 2 FPS

Frontend (React + TypeScript):

  • Web Speech API for voice commands and TTS output
  • High-contrast accessible UI\
  • WebSocket connection for live camera streaming

Challenges we ran into

The biggest challenge: bridging the gap between synthetic images and real camera input.

Our pipeline works perfectly on clean images (97% accuracy), but real webcam footage introduces:

  • Uneven lighting and shadows
  • Screen glare when showing images on monitors
  • Low contrast between dots and background
  • Paper texture creating false dot detections

We spent significant time building noise rejection, a grid-structure validator that checks if detected dots form regular rows with consistent spacing. This prevents the system from speaking garbage when pointed at faces or random objects.

Adaptive thresholding was tricky. A single threshold doesn't work for all lighting conditions. We had to implement a hybrid approach that detects lighting uniformity and switches methods accordingly.

Cell segmentation with variable spacing was harder than expected.

Accomplishments that we're proud of

  • 97% accuracy on camera-captured Braille images (all 26 letters, numbers, multi-line)
  • 27ms average processing per frame — fast enough for real-time
  • Smart noise rejection — the system stays completely silent when pointed at non-Braille content
  • Multi-strategy dot detection — contour analysis + Difference of Gaussians + Hough Circle Transform + morphological top-hat
  • Accessibility-first design — voice commands, audio tutorial, high contrast, screen reader support
  • The pipeline correctly handles shadows, paper texture, different backgrounds, and varying dot sizes

What we learned

  • Classical computer vision (thresholding + contours) is powerful but brittle — it needs near-perfect conditions
  • The gap between "works on test images" and "works on live camera" is enormous
  • Noise rejection is just as important as detection; a blind user hearing constant false reading is worse than hearing nothing

What's next for Braille Vision Scanner

  1. ML-based dot detection - Replace classical CV with a CNN trained on real Braille photographs. This would handle lighting/angle variations that thresholding can't.
  2. Grade 2 Braille - Support contractions and shorthand (currently only Grade 1)
  3. Mobile app - Native camera access via React Native for better performance
  4. Real embossed Braille — Train on actual textbook photos with side lighting

Built With

Share this project:

Updates

Submission history