SomatoSync

Inspiration

Concussion recovery does not stop when someone leaves a doctor's office.

People still have to read assignments, use websites, answer messages, complete work, and return to school, daily life, exercise, or sport. But symptoms such as headache, light sensitivity, difficulty concentrating, visual discomfort, dizziness, and mental fatigue can change depending on the task.

The problem is that the technology around the person usually stays exactly the same.

Most concussion tools focus on recording symptoms or giving static guidance. We kept coming back to a different question:

What if the digital environment could adapt to the person recovering, instead of constantly asking the person to adapt to the digital environment?

That became the idea behind SomatoSync.

We wanted to build more than a symptom tracker. We wanted recovery information to actually affect the experience in front of the user. If reading becomes difficult, SomatoSync can reduce how much information appears at once. If visual symptoms are present, it can lower visual intensity and motion. If dense text becomes harder to process, it can simplify the page structure, increase spacing, offer Plain Language, read content aloud, or create a more focused reading environment.

Our goal was to connect recovery tracking, neuroscience, accessibility, and AI into one system that helps people understand what they can tolerate and temporarily adapt their digital environment as those needs change.

What it does

SomatoSync is a privacy-first concussion recovery platform that connects symptom tracking, multidomain assessment, recovery planning, adaptive accessibility, AI, and communication in one system.

Users can complete a 22-item PCSS-style symptom check-in or speak naturally through a private Voice Check-In. They can also complete optional reaction-time, learning and delayed-recall, and experimental camera-based postural-movement assessments.

SomatoSync intentionally keeps these domains separate. Symptoms, reaction time, memory, balance, and real-world function measure different parts of recovery, so we do not collapse them into one "recovery percentage" or readiness score.

Instead, SomatoSync builds a longitudinal recovery picture through:

• symptom and assessment trends
• activity-tolerance tracking
• return-to-learning, daily-life, and sport pathways
• personalized supports
• Response Memory, which tracks which accommodations appeared helpful in different situations
• reports for sharing selected recovery information with people involved in care

For the first 24 to 48 hours after a recorded injury date, SomatoSync can also show a simpler Acute Mode focused on relative rest, manageable activity, and clear safety guidance. Later stages support gradual increases in cognitive and physical activity rather than prolonged complete rest.

Focus Mode

The most important part of SomatoSync is that the interface itself can become part of the support system.

Focus Mode starts from the user's confirmed symptoms and can temporarily adjust:

• text size and line spacing
• page luminance and saturation
• motion and moving media
• reading width
• information density
• visual hierarchy and secondary information
• language complexity
• Read Aloud support
• a manual Reading Spotlight

For stronger cognitive-load support, long ordinary paragraphs can become short previews with Show full text / Show less controls. Multi-column layouts can become a single reading lane, and nonessential interface elements can be reduced while navigation and important information remain available.

This means Focus Mode does more than recolor the page. It can change the amount, structure, and presentation of information.

Safety messages, alerts, forms, dialogs, and protected clinical text are excluded from automatic collapsing.

Optional on-device adaptation

If the user chooses to enable the camera, MediaPipe Face Landmarker runs locally in the browser.

SomatoSync creates a short same-session reference and looks for sustained changes across signals such as viewing distance, squinting, blink patterns, head movement, gaze variation, rereading, and extended pauses.

We were careful about what those signals mean.

Squinting does not diagnose concussion. Moving closer to a screen does not prove neurological fatigue. These are treated only as non-specific accessibility cues.

SomatoSync requires sustained patterns, multiple signal families, tracking-quality checks, grace periods, and cooldowns before offering stronger changes.

The user can accept, undo, pause, or turn off adaptations at any time. Raw camera frames and facial landmarks are not stored or uploaded.

After enough confirmed feedback, an optional local TensorFlow.js model can contribute to future personalization.

SomatoSync Shield

Recovery does not only happen inside SomatoSync.

SomatoSync Shield is a Chrome extension that carries selected reading accommodations onto external websites.

Shield can adjust text size, spacing, article width, motion, media intensity, visual intensity, and page structure. It uses site-specific consent, and the original page can be restored at any time.

Shield does not diagnose concussion from webpage behavior and does not send webpage content back to SomatoSync.

Recovery system

SomatoSync organizes recovery around function, not one score.

The Recovery area includes:

• Overview - Current recovery pattern, what seems to help, recent events, and supporting evidence
• Trends - Symptoms plus separate reaction, memory, and experimental camera-movement trends
• Plan - Current supports and pathways for return to learning, daily life, and sport
• Activity tolerance - Records what the user attempted, symptom change, and how long it took to return toward baseline
• Response Memory - Tracks which accommodations appear helpful across sessions

Return-to-learning, daily-life, and sport pathways remain separate because they involve different demands. SomatoSync supports gradual progression but never grants medical clearance.

Multidomain assessments

SomatoSync includes several browser-based recovery measures:

• PCSS-style symptom tracking - 22 symptoms rated from 0 to 6
• Reaction time - Practice and scored trials with within-person history
• Learning and delayed recall - An original 10-word list across multiple learning trials followed by delayed recall
• Experimental camera balance - MediaPipe Pose Landmarker estimates postural movement and head steadiness from a short eyes-open recording

These measures are displayed separately and compared mainly with the user's own earlier sessions.

We deliberately avoided turning them into one readiness score because that would suggest more clinical certainty than the prototype can support.

Voice Check-In

Voice Check-In allows users to describe how they feel naturally instead of manually adjusting every symptom slider.

Whisper Tiny through Transformers.js transcribes speech locally in the browser. SomatoSync maps the transcript into possible PCSS symptoms and asks the user to review the suggested ratings before anything is saved.

Speech is used as an input method, not as a diagnostic signal. Raw audio and transcript text are discarded after the user confirms the structured symptom record.

Possible danger-sign language is routed to urgent guidance instead of being handled like a normal check-in.

Research Assistant and on-device AI

We did not want the Research Assistant to behave like a generic health chatbot.

It retrieves from a built-in concussion evidence library and uses:

• BM25 lexical retrieval
• MiniLM semantic embeddings
• reciprocal-rank fusion
• MiniLM neural reranking
• Gemini Nano when available
• FLAN-T5 Small through Transformers.js as a local fallback
• deterministic grounding and safety checks

Generated answers are based on retrieved evidence rather than unrestricted generation.

Requests involving diagnosis, medical clearance, exact personal recovery dates, driving readiness, or urgent danger signs are routed away from generative AI or answered with explicit safety boundaries.

The Recovery system uses the same local generation stack only to rewrite already-calculated recovery facts into easier language.

Research foundation

A major part of SomatoSync was making sure the design reflected modern concussion guidance instead of outdated ideas about prolonged complete rest.

We grounded the recovery logic in sources including:

• Amsterdam 2022 International Consensus Statement on Concussion in Sport
• Living Concussion Guidelines
• Living Guideline for Pediatric Concussion Care
• CDC HEADS UP guidance
• PCSS-style symptom tracking concepts
• peer-reviewed research related to cognition, postural movement, graded activity, and concussion recovery

These sources shaped major design decisions.

SomatoSync uses relative rest followed by gradual return to activity, temporary symptom-based accommodations, separate return-to-learning and return-to-sport pathways, and multidomain tracking rather than one recovery score.

Focus Mode was also designed around the fact that concussion can affect attention, visual tolerance, reading endurance, mental fatigue, and cognitive stamina. Because those needs can change by task and over time, Focus uses temporary, reversible accommodations rather than one permanent "concussion theme."

How we built it

We built SomatoSync using React, TypeScript, and Vite, with several AI and machine-learning systems working together.

• Whisper Tiny + Transformers.js - Local Voice Check-In transcription
• MediaPipe Pose Landmarker - Experimental postural-movement assessment
• MediaPipe Face Landmarker - Optional interaction signals for Focus Mode
• TensorFlow.js - Local personalization from repeated user feedback
• Browser Speech Synthesis - Read Aloud
• Gemini Nano / FLAN-T5 Small - Local explanation generation
• BM25 + MiniLM + neural reranking - Evidence retrieval
• Chrome Extension APIs - SomatoSync Shield and site-specific permissions

A major technical challenge was making these systems feel like one recovery workflow rather than separate demos.

A Voice Check-In can create a structured symptom record. That symptom record can influence the starting Focus setup. Optional computer-vision signals can refine the interface. User feedback can update personalization. Useful supports can enter Response Memory, follow the user onto outside websites through Shield, or be shared through Recovery Relay.

Challenges we ran into

One of the hardest parts of building SomatoSync was making it technically advanced without making medical claims the technology could not support.

Computer vision without pretending it is a biomarker

MediaPipe can detect changes in viewing distance, facial movement, or head position, but those measurements are not specific to concussion.

We redesigned Focus Mode around multiple signal families, sustained observation windows, tracking-quality requirements, grace periods, cooldowns, user confirmation, and reversible changes.

The system can notice that a session may be becoming harder. It cannot determine why.

Reducing cognitive load without hiding important information

A simplified interface is not useful if it removes safety guidance or important recovery information.

Focus Mode can reduce long body copy and explicitly secondary information while preserving alerts, forms, dialogs, navigation, urgent guidance, and protected clinical content.

Avoiding a misleading recovery score

Combining symptoms, reaction time, memory, balance, and activity tolerance into one number would look simple, but it would imply that the system could calculate overall neurological recovery.

We decided not to do that.

Each domain remains visible separately, and unusual changes lead to rechecking or professional follow-up rather than an automated medical conclusion.

Making adaptation predictable

An interface that constantly changes could become more frustrating than helpful.

We added sustained-signal requirements, cooldowns, tracking-quality checks, user approval, Undo, pause controls, and re-trigger logic that requires a fresh sustained pattern after Undo.

Accomplishments that we're proud of

We are most proud that SomatoSync became more than a symptom tracker.

A symptom reported during a check-in can influence the starting interface. Sustained interaction changes can refine that setup. User feedback can help the system learn which adaptations appear helpful. Those supports can follow the user onto external websites through Shield or be shared through Recovery Relay.

That creates a loop between:

what the user reports -> what they are trying to do -> how the interface responds -> whether the support actually helped

We are also proud that Focus Mode changes more than color.

It can change:

• how much information appears at once
• how wide text is presented
• how dense a page feels
• how much motion is visible
• how complex language is
• whether content is read aloud
• how a user follows a section of text

The goal was to reduce actual visual and cognitive load, not just create a different theme.

SomatoSync also combines speech recognition, computer vision, local machine learning, semantic retrieval, neural reranking, local generative AI, browser accessibility adaptation, Chrome extension integration, and longitudinal recovery tracking into one connected product.

Safety and responsible design

Because SomatoSync deals with health information, we made the boundaries of the prototype as important as the features.

SomatoSync does not:

• diagnose concussion
• measure concussion severity
• predict an exact recovery date
• prescribe treatment
• determine neurological readiness
• provide return-to-school, work, driving, exercise, or sport clearance

Danger signs are treated as reasons to seek urgent medical evaluation.

Camera signals are treated as non-specific accessibility cues. Experimental assessment results are kept separate from clinical conclusions. AI is used for accessibility, organization, retrieval, and explanation rather than diagnosis.

Raw camera frames, facial landmarks, pose streams, and voice recordings are not stored.

Users can undo or disable adaptive changes at any time.

What we learned

The biggest thing we learned is that concussion recovery technology should not simply collect more data.

A symptom score, reaction-time result, memory task, camera measurement, or behavioral signal only tells part of the story.

We constantly had to separate:

What did the system observe?

from

What is the system actually justified in concluding?

That distinction shaped almost every part of SomatoSync.

It is why camera signals are treated as accessibility cues rather than biomarkers. It is why assessments remain separate instead of becoming one recovery score. It is why AI cannot provide diagnosis or clearance.

We also learned that accessibility can be dynamic.

Someone may need larger text while reading, less information during a mentally demanding task, reduced motion when visually sensitive, or no accommodation later in the day.

Instead of treating accessibility as one permanent setting, SomatoSync treats accommodations as temporary, reversible supports that can change with the task and the person.

What's next for SomatoSync

The next step is validation.

We want to work with concussion researchers, clinicians, caregivers, and people with lived experience to determine which adaptations are genuinely useful and when they should be offered.

We would like to:

• evaluate Focus Mode with real users
• measure whether specific interface adaptations reduce perceived visual or cognitive burden
• improve longitudinal personalization
• strengthen SomatoSync Shield across more websites
• study which supports consistently help in different tasks and environments
• improve recovery handoffs for clinicians, schools, workplaces, caregivers, and coaches

SomatoSync is still a prototype.

Our long-term goal is not simply to build a better concussion tracker.

We want to explore what recovery technology looks like when the digital environment itself becomes part of the support.

Built With

• chrome
• computer
• extensions
• gemini
• language
• learning
• machine
• mediapipe
• nano
• natural
• processing
• react
• retrieval-augmented
• tensorflow.js
• transformers.js
• typescript
• vision
• vite
• webassembly
• webgpu
• whisper