SignalBridge Use case example

Inspiration

Someone who can't speak still knows exactly what they want to say. That's the part people get wrong. Aphasia, ALS, a stroke, an intubation tube, an autistic kid on a hard morning, the intent is fully intact, and the only thing missing is the channel out. What usually fills that gap is a board of laminated icons, or a caregiver guessing out loud and watching a face for a flinch.

The guessing is the part that bothered us most. When a person can't correct you, a confident wrong guess isn't a small error, it's words put in their mouth that they have to live with. So we set out to build something that reads gesture and context into a spoken sentence, and that stops and asks instead of guessing whenever two different meanings are both genuinely plausible.

What it does

SignalBridge watches a short nonverbal interaction through a laptop camera and speaks the sentence behind it.

Press the button. It captures six frames over about three and a half seconds, a sequence, not a snapshot , and sends them to Gemini with a prompt that's explicitly not asking for a description of body motion. It's asking for the message. "Hands together beside the cheek after a yawn" comes back as "I'm tired and want to go to sleep."

Alongside every frame goes its timestamp and a measured motion energy value between 0 and 1, computed in the browser. That number tells the model where in the sequence the gesture actually peaks, so six near-identical stills don't get weighted equally. The waveform scrolling across the screen is that same buffer, it's you moving, not a looping animation.

If the reading is clear, the sentence is printed as a two-line broadcast-style caption and spoken aloud through the browser's speech synthesiser, word-by-word highlighted as it goes. If two materially different meanings survive, it refuses to pick. It asks one short question , "What do you need?" , with two or three tappable options, and says the chosen sentence immediately.

Every confirmed reading — never a frame, never an image — is written to a Tiger Cloud hypertable, which answers the operational question a ward or a lab actually has: how often did this system have to stop and ask, by room, by hour?

How we built it

The capture path. A getUserMedia video feed plus a 64×36 off-screen probe canvas sampling every fourth pixel on an interval. Frame-to-frame absolute difference, normalised to 0–1, pushed into a 240-slot Float32Array ring buffer that both the on-screen trace and the frame metadata read from. Six full-resolution JPEG frames at 650 ms apart get base64'd and posted to /api/interpret.

The understanding layer. @google/genai against Gemini, with a hand-written JSON responseSchema and temperature: 0.25. The model returns a structured interpretation: actor, intent, a short semantic action label, ranked object candidates, location, up to three supporting observations, and either a final first-person sentence or a clarification question with full options. The system prompt carries hard rules — never infer appearance, race, age or gender; never diagnose; never claim sign-language translation; only include an object the communicator intentionally referenced, not every bottle visible in frame.

The honesty design. Confidence values are documented everywhere in the codebase as ranking signals, not statistics. They sort candidates and we render rank — never a percentage, because a percentage next to someone's words implies a certainty the model doesn't have.

Each clarification option ships with its own complete finalMessage already written by the model. So when someone taps "Water," resolution is local and synchronous — no second round trip. The person just told us what they meant; a network call to venue wifi would be slower without being any more accurate.

The data lane. A TimescaleDB hypertable on Tiger Cloud, partitioned on time, segmentby = 'room', plus a continuous aggregate bucketing readings hourly by room with a clarified count. The write is deliberately fire-and-forget: if the database is missing or down, /api/readings returns {stored: false} and logs it. A database outage is never allowed to tell someone their sentence failed. The GET path queries the continuous aggregate, and falls back to computing the same shape live with time_bucket() if the rollup was never created.

The analytics layer. A databricks_ro Postgres role with SELECT on two tables and nothing else, plus Unity Catalog Lakehouse Federation SQL and a JDBC notebook as an alternate path. Tiger answers one room in milliseconds; Databricks was meant to answer a campus over a semester.

Challenges we ran into

The model was describing bodies instead of reading people. Our first prompts produced anatomical play-by-play: "the subject raises their right arm and extends the index finger." Accurate, useless. The rewrite that fixed it made the instruction explicit — your job is NOT to narrate body motion, it is to infer the message — and gave the model a reasoning order and worked examples at the level of interpretation we wanted.

It clarified constantly, then not at all. Asking on every imperfect confidence made it unusable; asking never made it a liar. The clarification policy went through several drafts before landing on the rule that works: ask only when two or more materially different intended messages remain plausible and picking wrong would change what gets said. And the question has to be about meaning, not mechanics — "Are you pointing at yourself and then at me?" is a bad question. "What do you need?" is a good one.

A credential bug that looked like a vision bug. The Gemini client was constructed at module load, so a missing key threw while the module was still evaluating. Every request failed in about 15 ms, and the UI could only render it as "could not read that" — it looked like the vision pipeline was broken when nothing had ever reached the model at all. Moving construction into a lazily-built getClient() with an error message that names the actual fix cost ten minutes and saved hours of the wrong debugging.

Continuous aggregates can't be created inside a transaction block. Sending rollup.sql as one file fails, because multi-statement input runs in one implicit transaction. One statement per call. And by default a one-hour end_offset policy means a reading taken on stage doesn't appear in the rollup until an hour later — which is the entire demo. ALTER MATERIALIZED VIEW ... SET (timescaledb.materialized_only = false) unions the materialised buckets with newer rows and fixes it.

Accomplishments that we're proud of

What we learned

That the interesting engineering in an assistive tool is almost entirely in what it refuses to do. Not writing a confidence percentage on screen, not ingesting the frames, not guessing when two meanings are live, not letting a failed database write surface as a failed sentence — every one of those was a decision to make the system say less than it technically could.

We also learned how much cheap, local signal improves an expensive remote model. The motion values cost a 64×36 canvas and a loop over every fourth pixel, and they meaningfully sharpened which frames Gemini weighted. And we learned broadcast captioning practice the hard way: a caption read from across a room fails at three lines, so the wrapper widens until it fits two and then balances them so the second line isn't a single orphaned word.

What's next

Persistent per-person vocabulary — the intents someone actually recurs to, ranked ahead of the general prior. Unlocking the Databricks federation path to answer the semester-scale questions. And on-device inference, so the promise that frames are interpreted and discarded becomes a promise that they never leave the room at all.