Bridgelens: An AI-Based Sign Language Translation

Inspiration

Modern digital solutions and financial services systematically marginalize the non-verbal and Deaf community. While standard LLMs can process text or speech fluidly, real-time sign language processing has historically suffered from extreme computational latency, poor handling of temporal dependencies, and a complete absence of support for indigenous languages like Yoruba, Hausa, and Igbo.

We were inspired to build BridgeLens Engine to demonstrate that high-performance machine learning architectures can bridge both communication and economic gaps simultaneously—combining edge computer vision with secure payment rails without requiring massive, high-latency cloud infrastructure.

What it does

BridgeLens Engine is a high-performance machine learning backend and API designed to enable seamless, two-way communication and branchless financial inclusion for the Deaf and non-verbal community.

The engine processes raw data streams across three primary pipelines:

Real-Time Sign-to-Text Processing: Captures a sequential stream of video frames via a sliding window, extracts key dynamic body coordinate joints using computer vision, and uses an attention-driven deep learning model to classify the gesture into one of 57 distinct conversational or medical vocabulary signs (such as DOCTOR, BLOOD, HOSPITAL, or MONEY).

Linguistic Translation & Normalization: Features a robust NLP layer utilizing the Groq API (Llama 3.1). When a non-verbal user interacts using signs, or when an indigenous language speaker inputs text (in Yoruba, Hausa, or Igbo), the engine interprets and translates the expressions into grammatically correct sentences or standardized uppercase sign gloss strings.

Accessible Financial Rail Execution: Connects directly to secure, branchless financial endpoints. The API supports biometric-backed identity verification (KYC) using an 11-digit National Identification Number (NIN), a facial matching layer, and a unique 3-second Sign Password—allowing users to securely run value-added services (VAS) like paying utility bills or buying data completely independently.

How we built it

BridgeLens is engineered using a decoupled, high-throughput Engine-and-Shell architecture:1. Computational Feature Extraction (The Visual Pipeline)Instead of passing raw video frames ($1920 \times 1080 \times 3$) to a computationally heavy model—which introduces unacceptable $O(N)$ bottlenecks—our pipeline compresses video frames into spatial coordinate vectors. Using MediaPipe's Image Vision Mode, the engine scales 33 holistic pose landmarks and $2 \times 21$ hand landmarks into a highly dense, localized 1D matrix of exactly $225$ spatial features per frame.$$\text{Total Features} = (33 \times 3) + (21 \times 3) + (21 \times 3) = 225$$2. Deep Temporal Modeling & Attention LogicThe sequential engine processes spatial inputs using a fixed temporal window of $T = 30$ frames. The sequence matrix $\mathbf{X} \in \mathbb{R}^{30 \times 225}$ is fed through a multi-layered deep learning architecture:Bidirectional LSTMs: Two stacked Bi-LSTM layers (128 and 64 units) capture structural semantic patterns from both past and future frame contexts simultaneously.Spatial Attention Block: We implemented a custom Self-Attention over the Time Dimension using a localized Tanh activation. This assigns dynamic weights to frames containing critical kinetic hand shapes while down-weighting static transitional movements.$$\alpha_t = \text{Softmax}(\tanh(\mathbf{W}_a \mathbf{h}_t + \mathbf{b}_a))$$3. Data Engineering & Augmentation MatrixTo handle sample variations across 57 distinct sign classes, we engineered an explicit, automated mathematical data augmentation pipeline. It structurally manipulates sequences on the fly using:Gaussian Noise Injection: Adds deterministic noise vectors ($\mu=0, \sigma=0.01$).Random Spatial Scaling: Adjusts coordinate magnitudes by a factor of $1.0 \pm \delta$.Temporal Time-Shifting: Uses a random index roll ($\pm 3$ frames) along the time axis to simulate variation in user signing speeds.4. High-Throughput API DeploymentThe model is served using a containerized FastAPI backend designed for micro-second response rates. The API abstracts translation logic via three critical components:Sliding Window Stride ($S=5$): The inference engine runs predictions every 5 frames instead of every single frame, cutting redundant compute cycles by 80%.Majority Voting: A temporal stabilization algorithm (Counter().most_common(1)) filters out transient classification flips, returning a clean, stable prediction text.Linguistic Translation Integration: Connects to the Groq API running Llama-3.1-8b to automatically parse and normalize localized Nigerian language inputs (Yoruba, Hausa, Igbo) into standard upper-case sign gloss arrays.

Challenges we ran into

Data Imbalance & Overfitting: Managing 57 highly complex kinetic classes on restricted sequence counts meant the model originally gravitated towards dominant classes. We neutralized this by computing automated balanced class weights (compute_class_weight) and feeding them directly into our categorical cross-entropy loss function.

Accomplishments that we're proud of

99% Structural Data Compression: We are incredibly proud of optimizing the frame preprocessing loop. By extracting spatial coordinates instead of transmitting heavy video assets over the network, we reduced the data payload size by over 99%, making the tool highly functional in low-bandwidth environments.

Successful Attention Layer Integration: Implementing the custom spatial-temporal Self-Attention Mechanism allowed our Bi-LSTM networks to achieve stable convergence, boosting top-3 categorical accuracy significantly across a complex 57-class dataset.

Linguistic Accessibility for Native Dialects: Seamlessly engineering the prompt logic triggers for Yoruba, Hausa, and Igbo within a unified API framework allows the software to serve localized, multilingual communities that are traditionally ignored by Western-centric accessibility software.

Flawless End-to-End Core Decoupling: When faced with severe timeline bottlenecks, we successfully split our software's heavy components—transforming a massive training script and complex logic blocks into a clean, standalone, and lightning-fast FastAPI microservice that plugs effortlessly into low-code frontend shells.

What we learned

We learned that algorithmic efficiency trumps raw model size. By carefully extracting landmark coordinates instead of sending heavy video objects over the network, we reduced payload sizes by over 99%. We also deepened our mastery of sequence-to-sequence model optimization, hyperparameter tuning via early stopping callbacks, and building highly reliable fallback logic when working with sandbox financial APIs.

What's next for Bridgelens: An AI-Based Sign Language Translation

Transition to Edge-Native Deployment: Moving the MediaPipe feature extraction and coordinate mapping blocks out of the backend container entirely and running them directly on user devices via client-side JavaScript or WebAssembly to lower server compute overhead.

Expansion of the Medical & Financial Corpus: Collecting and training the model on a broader lexicon of localized sign variations, specifically focusing on complex diagnostic terms and branchless banking terminology to enhance triage and financial independence.

Production Deployment of Interswitch Live Rails: Graduating from sandbox API environments to live production connections, allowing actual non-verbal users to set up functional accounts with real-world deposit, transfer, and token-purchasing capabilities.

Continuous Multi-Agent Orchestration: Upgrading the backend into an autonomous, state-driven multi-agent framework capable of dynamically tracking a user's historical behavioral profiles to offer proactive predictive financial and health recommendations.

Built With

• cloudinary
• fastapi
• groq
• interswitch-connect
• javascript
• keras
• mediapipe-(0.10.13)
• medo
• numpy
• opencv
• pydantic
• python-3.10
• scikit-learn
• tensorflow-2.x