ASL Bridge

Fluent Hand Talk bridges the communication gap between Deaf and hearing communities through real time ASL translation.

Comment

ASL Bridge

Inspiration

Most digital systems — meetings, classrooms, interviews — are designed around spoken English and dense text. For Deaf and Hard of Hearing users, access often depends on interpreters or captions that preserve English structure, even though American Sign Language (ASL) is a distinct visual language with its own grammar and cognition.

We were inspired by the gap between translation and comprehension. Rather than asking ASL users to adapt to English-first systems, we asked: what would it look like if systems adapted to ASL cognition instead?

ASL Bridge was created to explore an ASL-first approach to live communication — one that prioritizes structure, context, and visual clarity over literal word-for-word translation.

What it does

ASL Bridge is an ASL-first live meeting companion that restructures spoken conversations into ASL-friendly visual explanations in real time.

Instead of translating every sentence, the system:

  • Segments live meeting speech into meaningful ideas
  • Identifies intent (instruction, decision, question, action item)
  • Presents simplified text alongside concept-based ASL explanation videos
  • Maintains a visual timeline so users can track context and progression

The result is a visual language layer that supports ASL users during live meetings, without claiming to replace interpreters or perform literal ASL translation.

Key features:

  • Real-time concept segmentation from live captions or transcripts
  • Two-panel interface with structured text and ASL explanation videos
  • Topic timeline for context recovery and reduced cognitive load
  • Optional opt-in voice output for short, user-approved messages in mixed Deaf–hearing collaboration settings

How we built it

We designed ASL Bridge as a browser-based prototype focused on clarity, ethics, and finishability.

Core technologies:

  • Google Gemini API for reasoning over live text:
    • Segmenting speech into conceptual units
    • Classifying intent (instruction, decision, discussion, etc.)
    • Generating simplified, structured explanations
  • Pre-recorded or placeholder ASL explanation videos mapped to high-level concepts, clearly labeled as prototype representations
  • Frontend UI emphasizing spatial layout, minimal cognitive load, and visual hierarchy

We intentionally avoided sign language recognition or word-level ASL generation to ensure responsible AI use and to respect ASL as a complete language rather than a dataset.

Challenges we ran into

  • Avoiding overclaiming: It was tempting to frame the project as “ASL translation,” but doing so would misrepresent ASL linguistics and raise ethical concerns.
  • Scoping live features: Fully integrating with live meeting platforms was unrealistic in the time available, so we simulated live input using real-time captions and transcript streams.
  • Balancing text and ASL: We wanted to support bilingual reference (ASL + text) without overwhelming users or reverting to English-first design.
  • Responsible AI framing: Ensuring that our AI usage focused on structure and intent — not people or identity — required careful design decisions.

Accomplishments that we're proud of

  • Building a live, interactive demo that shows real-time restructuring of meetings for ASL comprehension
  • Creating an ASL-first UX that avoids common accessibility clichés
  • Using AI for meaningful reasoning, not just surface-level translation
  • Clearly communicating ethical boundaries and limitations in both the product and presentation
  • Delivering a polished, judge-ready prototype within hackathon constraints

What we learned

  • Accessibility is not just about adding overlays — it’s about rethinking system assumptions
  • ASL access benefits from structure, pacing, and context, not just captions
  • Responsible AI design requires saying no to certain features, even if they seem impressive
  • Clear UX storytelling can matter as much as technical complexity in accessibility projects

What's next for ASL Bridge

Future directions include:

  • Expanding the ASL explanation clip library with community-informed content
  • Supporting additional meeting contexts such as classrooms and interviews
  • Improving topic tracking and context recovery across longer sessions
  • Collaborating with Deaf designers and interpreters to refine ASL-first interaction patterns
  • Exploring deployment as a companion tool alongside — not in place of — professional interpreters

ASL Bridge is a step toward systems that adapt to visual language, rather than forcing visual language to adapt to systems.

Built With

Share this project:

Updates