SignBridge

SignBridge Inspiration

There’s no personal story behind this one . We wanted to see whether a laptop could do speech recognition and neural translation into sign language with no internet.

I love to learn technoligies related to ai , ml and computer vision. I think it's the same attraction that made me start this project and it had fired my desire to see if ai can make communication more accessible.

What it does

You talk, and SignBridge shows a live transcript. When you pause, it translates the finished clause into Factored SignWriting (FSW) and draws the glyphs. It also drives a pose avatar through a remote service, and it can listen to tab audio through browser screen sharing.

How we built it

The shell is Tauri, the frontend is React, TypeScript and Vite, and the backend is FastAPI running as a Python sidecar. The browser streams 16-bit PCM chunks every 256 ms through a ScriptProcessorNode over a WebSocket at /ws/stream.

Two speeds. The fast one runs faster-whisper (tiny.en, int8, CPU) on incoming audio for partial transcripts, with a TranscriptStabilizer that freezes the prefix so text on screen stops flickering. The slow one waits until Silero VAD hears 0.8 s of silence, then hands the whole clause to Sockeye on PyTorch CPU, which outputs FSW.

There’s an optional Groq-hosted LLaMA step that simplifies long sentences first. If it’s off or fails, a rule-based fallback strips fillers like “um” and “like”. That’s the only cloud call in the text path.

Challenges we ran into

The audio lag. Our first version ran Whisper and Sockeye directly inside the async WebSocket handler. Audio stuttered, the transcript froze and then jumped ahead, and eventually clients dropped with WebSocketDisconnect from ping timeouts. We put time.perf_counter() timers around the socket reads and the inference calls. The logs showed chunks stopping exactly during WhisperModel.transcribe() and sockeye.translate(), 300 ms to almost a second at a time. The event loop wasn’t reading the socket, so chunks piled up and then arrived in one burst. We moved inference into loop.run_in_executor, added an asyncio.Lock so ASR and Sockeye stopped fighting over cores, and made partial ASR a create_task.

Translating too early. We wanted to translate continuously. Running Sockeye every 350 ms next to Whisper pinned every CPU core at 100% and the whole machine lagged. The output was worse. Feed it “Good”, then “Good morning”, then “Good morning every”, and the signs came out erratic and kept changing. Sign languages have different syntax and topicalization from spoken English, so half a sentence isn’t something it can translate. We dropped that idea and translate only after the 0.8 s pause.

The output looked broken. The first Sockeye result was a string like M500x500S14c20489x524S27106515x543S30a00482x482. In the browser it showed as raw text or empty rectangles until we loaded the Sutton SignWriting TrueType fonts through @sutton-signwriting/font-ttf in Vite.

Cold start. The first request took 7 to 10 seconds, roughly 4 to 6 loading models from disk and the rest PyTorch setting itself up on CPU. We now push a dummy 1-second audio array through Whisper and translate “hello” through Sockeye at startup. The first real clause is fast. The server takes about 5 seconds to boot.

The avatar, which we didn’t solve. Pose generation calls a remote Google Cloud Function . If we’re offline or it takes longer than 5 seconds, the backend catches the error and loads backend/default.pose. The avatar then plays a generic animation instead of what you said. The SignWriting glyphs aren’t affected.

Accomplishments

It works end to end on CPU: speech in, live transcript, SignWriting out. The repo has automated tests and a benchmark script. The script measured 312 to 321 ms for ASR on one second of audio, 800 to 900 ms for Sockeye, and 1.8 to 2.0 s from end of speech to signs. Those numbers roughly add up if you count the 0.8 s silence wait: 0.8 + 0.3 + 0.85. They come from our benchmark runs, so they won’t hold on every machine.

i would like tell the some stuff related to machine on which i did benchmark on :

What machine did you benchmark on? Hardware: Tested on a 13th Gen Intel® Core™ i5-13420H (8 cores, 12 logical processors, x86_64 architecture). Execution Environment: Windows 11, running Python 3.10 with PyTorch 2.0.1 on CPU. Inference Settings: faster-whisper was locked to cpu_threads=4 with int8 quantization; Hugging Face offline guards (HF_HUB_OFFLINE=1) were strictly enabled.

What we learned

Inference inside an async handler will stall your socket. We only believed that once the timer logs showed audio ingestion freezing at the same moments Whisper ran.

A partial sentence isn’t a smaller version of the full one. Sockeye gave nonsense for “Good morning every” because the input wasn’t a clause, so when to call the model mattered more than how fast it ran.

Warm-up doesn’t remove a delay, it moves it. Cold-start cost went from the first sentence to app boot.

What’s next Offline pose generation. There’s a plan doc (plan/phase13_offline_pose_generation.md), and no code yet. Native OS-level audio capture (WASAPI inside the Tauri Rust side), so nobody has to tick “Share tab audio”. Running on the Snapdragon NPU through Qualcomm AI Engine, QNN or ONNX Runtime. Everything is CPU-only right now. Showing the glyphs to signers. No signer has reviewed the output yet