Constella

Inspiration

Constella started from a simple accessibility gap: Braille can be present in front of someone and still be inaccessible to them.

Many caregivers, teachers, volunteers, family members, and accessibility workers encounter Braille in real situations but may not be able to read it by touch. A worksheet, label, note, or form can contain important information, yet still require a trained Braille reader to interpret it. This project explores how a regular phone camera could help bridge that gap.

Braille itself is beautifully compact. A standard Braille cell is a (2 \times 3) grid of raised dots, giving (2^6 = 64) possible patterns. Those patterns can represent letters, numbers, punctuation, or contractions. The hard part for this project was not translating already-digital Braille symbols. The real challenge was reading physical Braille from camera input: dots on paper, uneven lighting, imperfect framing, motion, shadows, and perspective distortion.

The goal became simple:

Build a tool that can point a phone camera at real Braille, recognize the dots, convert them to English, and speak the result aloud.

What We Built

Constella is an offline-first native Android app prototype for scanning real physical Braille. It is designed around a complete on-device pipeline:

Camera frame
  -> image preprocessing
  -> dot detection
  -> cell segmentation
  -> Braille pattern recognition
  -> English translation
  -> text and speech output

The app is built with Kotlin, Jetpack Compose, CameraX, OpenCV-style preprocessing, TensorFlow Lite detector integration, liblouis translation plumbing, and Android TextToSpeech.

The current implementation includes:

• A four-module Android architecture: :app, :pipeline, :runtime, and :domain.
• A pure Kotlin domain layer for models, confidence handling, alignment logic, segmentation, recognition, grade selection, result retention, and scan-status policies.
• A Compose UI with scanning, settings, and results screens.
• Design tokens for colors, typography, spacing, high-contrast support, and minimum touch targets.
• Alignment guidance logic that can tell the operator to move closer, move farther, hold steady, add light, flatten the document, or scan when ready.
• A debounced ready-to-scan state machine so the app does not rapidly switch between "ready" and "not ready" during camera jitter.
• Dot detection post-processing and scanning-mode parameter selection for embossed and handwritten modes.
• Cell segmentation that groups dots into (2 \times 3) Braille cells, orders lines, and inserts word boundaries.
• Pattern recognition that maps segmented cells to six-dot Braille patterns.
• Translation interfaces and liblouis JNI integration points for Grade 1 and Grade 2 English Braille.
• Grade auto-detection with manual override support.
• Offline text-to-speech selection and playback-state handling.
• Privacy gates that keep cloud features default-off and prevent image transmission unless explicitly allowed.
• Error and low-confidence handling that reports problems through available channels instead of failing silently.

How We Built It

The project is split into four layers:

:app
  Compose UI, screens, theme, accessibility semantics

:pipeline
  CameraX, TTS, privacy gates, Android-facing pipeline services

:runtime
  TFLite, OpenCV-style preprocessing, liblouis JNI/runtime integration

:domain
  Pure Kotlin models, policies, segmentation, recognition, confidence, orchestration

The most important architecture decision was keeping :domain pure Kotlin. It does not depend on Android, CameraX, Compose, OpenCV, or TFLite. That lets the core reasoning code be tested quickly with JVM tests.

The pipeline treats confidence as a first-class value. Instead of pretending recognition is always correct, the system carries a score (c \in [0,1]) through multiple stages. Low-confidence cells can be marked in the UI, and low-confidence scans can trigger a rescan recommendation.

For cell grouping, the segmenter estimates Braille layout from detected dots. Line grouping uses the median cell height (\bar{h}), with a tolerance like:

$$ \text{line tolerance} = 0.5 \cdot \bar{h} $$

Word boundaries are inserted when the gap between adjacent cells is larger than the normal spacing:

$$ \text{gap} > 1.5 \cdot \bar{s} $$

where (\bar{s}) is the median spacing between cells in a line.

For perspective correction, the image preprocessing stage uses the idea of a homography (H), mapping a tilted document quadrilateral into a flat rectangle:

$$ p' = H p $$

This matters because Braille dots need to be grouped spatially. If the paper is tilted, cell spacing and line spacing become distorted.

What We Learned

The biggest lesson was that this is not just a translation problem. It is a computer vision problem, an accessibility problem, and a trust problem.

A Unicode Braille translator is easy compared with reading physical dots from a camera. Real paper introduces shadows, blur, lighting changes, perspective, hand movement, and dot depth differences. Embossed Braille and slate-and-stylus handwritten Braille also behave differently, so the project separates them into different scanning modes and labels handwritten output as lower confidence.

We also learned that accessibility features cannot be added at the end. Voice guidance, large controls, screen-reader labels, high contrast, and clear failure messages are part of the main product, not decoration. If the app cannot tell the operator what went wrong, the recognition pipeline is not useful enough.

Another major lesson was privacy. A Braille scan might contain school work, names, medical labels, or personal notes. That pushed the design toward offline-first operation, bundled models, bundled translation tables, and no developer-controlled backend.

Challenges

Physical Dot Detection

The hardest technical problem is detecting real raised dots reliably. White paper on white dots gives very little contrast unless lighting creates shadows. That is why the design uses torch/raking-light behavior, perspective correction, and contrast normalization before detection.

Braille Cell Segmentation

Detecting dots is not enough. The app must infer which dots belong to the same cell, which cells belong to the same line, and where words begin. A valid Braille cell can contain only one raised dot, so the system cannot simply require multiple dots before accepting a cell.

Grade 2 Braille

Grade 2 Braille uses contractions, so one Braille cell may map to multiple English characters or an entire word. That makes character-span tracking more complex than Grade 1, where the mapping is closer to one cell per symbol.

Real-Time Guidance Without Audio Spam

Camera feedback changes constantly. If the app speaks every tiny change, it becomes frustrating. The alignment state machine holds the ready state through short fluctuations and only resumes guidance when the problem remains long enough to matter.

Offline Packaging

The product goal is that the app keeps working without the developer running a server. That means the model, translation tables, native runtimes, and offline speech path all need to be packaged with or available on the device. The repository has placeholders and integration code, but final production packaging still needs the real native binaries, trained model, and evaluation assets.

Why It Matters

Constella is for the moments when someone is trying to help, but Braille becomes the barrier. A parent, teacher, caregiver, or volunteer should not need full Braille fluency just to understand a page in front of them.

The long-term vision is not to replace Braille literacy. Braille literacy is valuable and should be supported. Constella is a bridge: a way to make physical Braille more understandable to the people around it, and a way to give learners and support workers faster feedback.

Raised dots become English text. English text becomes speech. The page becomes accessible to one more person.

Built With

• android-studio
• android-texttospeech
• camerax
• gradle
• jetpack-compose
• kotest
• kotlin
• liblouis
• tensorflow-lite