SignBridge - TSL Translation System

Breaking communication barriers with real-time, bidirectional Taiwan Sign Language translation powered by Gemma AI and 3D animation.

Comment

Inspiration

In Taiwan, over 140,000 individuals live with hearing impairments, with approximately 30,000 relying on Taiwan Sign Language (TSL) as their primary mother tongue. Yet, fewer than 400 active certified sign language interpreters serve the entire country—a stark ratio of roughly 1 interpreter for every 75 Deaf individuals.

In critical daily scenarios such as healthcare consultations, public administration, transit, or job interviews, Deaf individuals are often restricted to slow, ambiguous text messaging or pen and paper. Commercial accessibility tools are overwhelmingly one-way: either standard speech-to-text (which overlooks the distinct grammatical syntax of TSL) or isolated static vocabularies. We engineered SignBridge to dissolve this barrier: an accessible, bidirectional, real-time communication platform that dynamically converts spoken natural language and text into fluid 3D sign language animations, while simultaneously classifying and translating continuous physical sign language into coherent natural text.

What It Does

SignBridge delivers a full-duplex communication pipeline between Deaf and hearing users on mobile devices:

  • Natural Language to 3D Sign Animation: Users input spoken voice or text. The system leverages LLM semantic translation to restructure standard Chinese grammar into native TSL gloss order, dynamically concatenating 3D skeletal clips to render smooth, continuous signing on an interactive 3D avatar.
  • Continuous Real-Time TSL Recognition: Deaf users sign directly into the smartphone camera. The system tracks 55 skeletal keypoints in real time, extracts kinematic features, and executes temporal gesture classification via deep graph neural networks.
  • Dual-Platform Synchronized Rooms: Powered by WebSockets, hearing and Deaf users can communicate across separate devices with synchronized avatar animations and live conversational transcripts.
  • Community Data Crowdsourcing: Users can record and upload custom sign samples to alleviate the acute scarcity of public TSL datasets and expand regional vocabulary coverage.

How We Built It

We designed SignBridge around a cross-platform mobile frontend connected to an asynchronous, high-throughput AI backend.

1. Client Architecture

  • Frontend: Built with Flutter & Dart to maintain identical UI and logic across iOS and Android from a single codebase.
  • 3D Visualization: Integrated Google Model Viewer to render interactive .glb avatars with full 360-degree rotation, smooth skeletal interpolation, and touch-to-zoom capabilities for inspecting intricate finger articulations.

2. Backend & Semantic Translation Pipeline

  • Microservice Layer: Developed with FastAPI and Uvicorn, serving asynchronous REST endpoints for batch tasks and WebSockets for real-time bidirectional communication.
  • Syntax Transformation (Gemma + RAG): TSL operates under distinct grammatical principles (e.g., Topic-Comment structures). We deployed Google Gemma to bridge natural Chinese semantics into TSL gloss order.
  • Vector Retrieval via PostgreSQL & pgvector: To prevent hallucinations and enforce idiomatic phrasing, we built an Advanced RAG pipeline. Given an input query vector q and database corpus vectors d_i, relevant sign templates are retrieved via cosine distance:

$$\text{Cosine Distance}(q, d_i) = 1 - \frac{q \cdot d_i}{\Vert{}q\Vert{}_2 \Vert{}d_i\Vert{}_2}$$

The retrieved context grounds Gemma's transformation before the target gloss sequence is assembled.

3. Spatial-Temporal Graph Convolutional Network (ST-GCN)

  • Custom 55-Point Topological Topology: Rather than flattening coordinates, we extract 55 skeletal points (V = 55) via MediaPipe Holistic: 12 upper-body pose joints, 21 left-hand joints, 21 right-hand joints, and a synthetic neck anchor (midpoint of left and right shoulders) defined as the coordinate origin.
  • Multi-Modal Feature Engineering: Because raw z-coordinates from monocular RGB can be noisy, we focus on 2D space (x, y) enriched with kinematic attributes:
    • Frame-to-frame velocity vectors (Δx, Δy)
    • Detection confidence binary masks
    • Kinematic bone vectors directed toward parent joints
    • Bone-joint angular invariant features across kinematic chains
  • Graph Modeling & Dynamic Edge Importance: Spatial keypoint sequences are resampled to T = 64 frames. We model the body as a spatiotemporal graph G = (V, E) using ST-GCN (best.pt) structured with 4 graph blocks (64 -> 64 -> 128 -> 128 channels) and spatial partition configuration. Spatial graph convolution on node v_i is formulated as:

$$f_{out}(v_i) = \sum_{v_j \in B(v_i)} \frac{1}{Z_{ij}} f_{in}(v_j) \cdot W(l_i(v_j)) \odot (A_k + M_k)$$

where A_k is the normalized adjacency matrix partitioned by distance to the skeleton center, M_k is a learnable edge importance weighting mask, and W is the kernel weight tensor. Temporal convolutions with stride-2 downsampling capture dynamic signing transitions over a 5.0-second sliding window.

4. Dynamic 3D Animation Synthesis

  • Identified sign tokens are mapped to pre-rigged 3D bone tracks. The backend uses pygltflib to concatenate bone channel animation curves across a continuous timeline, performing quaternion spherical linear interpolation (SLERP) at action boundaries to avoid jarring transitional jumps.

Challenges We Faced

  • Corpus Scarcity & Cross-User Variance: Taiwan Sign Language lacks large public benchmark corpora. To prevent overfitting on individual signers, we engineered an aggressive Domain Randomization pipeline:
    • Limb-length random scaling (±15%) to emulate divergent body proportions
    • Stochastic joint and hand dropouts (15% drop rate) simulating optical occlusion
    • Temporal warping (±30%) and random temporal cropping (25%) to handle non-aligned sliding windows
  • Syntactic Discrepancy: Mapping Chinese into TSL requires linguistic restructuring rather than literal transcription. Integrating Gemma with verified pgvector exemplars prevented direct transcription errors.
  • Real-Time Inference Drift: Fixed sliding windows on unsegmented real-time video streams often capture incomplete signs. Calibrating our temporal inference window to match the median gesture span (5.0s, resampled to T = 64) combined with multi-frame confidence voting (confidence >= 0.75) eliminated false-positive frame flicker.

What We Learned

  • Linguistic Autonomy: Sign languages are complete, standalone languages with dedicated spatial-temporal grammars rather than direct visual encodings of spoken text.
  • Graph-Based Kinematics: Graph convolutions that preserve human anatomical topology vastly outperform flat temporal vector models in handling spatial hand interactions and expressive upper-body cues.
  • Robust Domain Randomization: When training data is limited, physically informed synthetic augmentations (joint drop, limb scaling) are essential for zero-shot generalization across diverse real-world users.

Built With

Share this project:

Updates

Submission history