NeuroBridge: LifeConnect & SocialOrbit

Wearer-controlled multimodal communication bridge and sovereign safety net translating conversational subtext bidirectionally for autistic young adults.

Comment

Inspiration

Inspiration

Over 40% of autistic students spend the majority or entirety of their school years in segregated classrooms or separate specialized schools (NCES Digest Table 204.60, IDEA Part B data). These structured environments provide support, but when autistic youth and adults move into unstructured spaces — workplaces, social settings, everyday conversation — that support vanishes overnight.

We were inspired by Dr. Damian Milton's Double Empathy Problem: communication breakdowns between autistic and neurotypical people aren't caused by a one-sided deficit, but by a genuine two-way mismatch in communicative norms. Most existing tools only address one side — AI chatbots train an autistic person's social skills before a conversation happens, AAC apps help them build sentences, wearables read a neurotypical partner's face for the autistic wearer alone. None of them work live, for both people, during an actual interaction — and none address what happens when someone can't call for help themselves during a fall or a moment of crisis. We wanted technology that equips both sides in real time, rather than trying to fix or surveil one of them.

What it does

NeuroBridge is a browser-based platform built around three connected modules:

Communication Bridge — live, two-way conversational clarification that works across four input channels: spoken audio, direct typed text, an on-screen switch-scanning keyboard, and calibrated hand gestures — all flowing into one shared conversation timeline. It flags likely friction points (sarcasm, idioms, bluntness, ambiguous questions) only when genuinely needed, stays silent for clear statements, and adapts within a session based on which types of clarification a specific pair finds helpful. Gesture Translation — a per-user calibrated system where the wearer teaches the app their own gestures (3-5 labeled examples), rather than relying on a generic "read autistic behavior" model that would misinterpret stimming as distress. Safety Net — fall and distress detection (via manual test triggers standing in for continuous sensors) plus one-tap SOS, with real-time alerts to a guardian dashboard and an actual email dispatch.

Every clarification card is badged transparently as either live-AI-generated or from the offline heuristic fallback, so nothing is presented as more automatic than it really is.

How we built it

Frontend: React + Vite, with the Web Speech API for voice input, a custom switch-scannable on-screen keyboard for AAC-style input, and @mediapipe/tasks-vision (HandLandmarker) for client-side, in-browser hand landmark tracking — no video ever leaves the device. Gesture matching: each captured gesture is normalized into a landmark vector relative to the wrist, and compared against the wearer's own calibrated examples using distance-based matching, with a rejection threshold enforcing "false silence" over guessing wrong. Backend: Node.js/Express, with a two-tier language model architecture — a live LLM call for clarification analysis, falling back automatically to a deterministic heuristic engine if the API is unavailable, so the demo never goes dark mid-conversation. Adaptive feedback: each clarification card carries a thumbs up/down; repeated negative feedback for a category (e.g., idioms) suppresses that category for the rest of the session for that specific pair. Safety alerts: Server-Sent Events push real-time alerts to a Guardian Dashboard, paired with an actual dispatched email containing location and vitals. Persistence: settings, feedback tallies, and conversation history persist per-browser via localStorage, so a page refresh doesn't silently reset the adaptive state. Deployment: live on Vercel, tested end-to-end against the production URL, not just locally.

Challenges we ran into

Balancing real-time transcription with unpredictable environments. The Web Speech API can be unreliable in noisy rooms or muted devices, so we built dual-input fallbacks — typed and scanning-keyboard input feed the exact same analysis pipeline as speech. Designing for wildly different input speeds. A spoken turn and a switch-scanned turn happen on completely different timescales; analyzing text the instant it's typed would interrupt someone mid-composition. We solved this by triggering analysis only on explicit send, regardless of which input channel produced the turn. Preventing clarification fatigue. Constant interruptions would break natural conversation, so the system stays silent for clear statements, auto-dismisses cards after several seconds, and learns to suppress categories a specific pair doesn't find useful. Being honest about what's simulated. Early in testing, we found language in our own drafts overstating automatic sensing as fully autonomous. We rewrote it to clearly disclose that fall/distress detection currently uses manual test triggers standing in for continuous sensors, while the full alert-to-guardian pipeline itself is genuinely live. Accomplishments that we're proud of True wearer sovereignty — every monitoring feature is visibly toggleable, and nothing runs invisibly. On-device gesture recognition — real landmark extraction and matching entirely in-browser, with zero video sent to any server. A genuinely unified, multimodal conversation — speech, typed text, scanning-keyboard input, and gesture-based turns all flow into one shared timeline rather than separate silos. Closed-loop adaptation — the Bridge changes its own behavior within a session based on what a specific pair actually finds helpful.

What we learned

Accessibility isn't one-size-fits-all, even within one diagnosis. The same person might speak sometimes, type sometimes, and gesture when overwhelmed — the system needed to treat all of these as equally valid, not accommodate one as an afterthought. Assistive tools should equip, not police. Many existing safety tools are built for anxious caregivers watching over someone; we tried to build one the wearer controls instead. Honest disclosure builds more trust than polish. Being explicit about what's simulated versus fully live turned out to matter more for credibility than making everything sound more automated than it is.

What's next for NeuroBridge

Extend gesture recognition from static held poses to dynamic, continuous motion (like rocking or repeated hand-flapping). Move from manual fall/distress test triggers to real continuous sensor integration on wearable hardware. Move from per-browser localStorage to a real shared backend so state persists across devices, not just one browser. Pilot with autistic self-advocates and speech-language pathologists to refine clarification phrasing before any real-world deployment

Built With

Share this project:

Updates

Submission history