DexPlay

Dexplay turns any device with camera access into an offline/online dexterity coach for children with motor delays , replacing unavailable therapists with camera based games and clinical progress data.

Comment

Inspiration

I was inspired by the inequality in paediatric rehabilitation. In Kenya, there are only about 553 registered occupational therapists for millions of children. For a child with a fine motor delay, DCD, or cerebral palsy, seeing a specialist is often financially and geographically impossible.

What it does

DexPlay is an offline-first Progressive Web App (PWA) that turns a standard smartphone into a pediatric dexterity coach.

  1. Turns any piece of paper into a play mat
  2. Tracks the hand without touching the screen
  3. Uses "dwell" instead of "pinch" actions to enable child play
  4. Runs entirely oflline with online syncs for more functions
  5. Generates clinical data : syncs it to the cloud so a remote occupational therapist or community health worker can track the child's motor progression over time.

How we built it

  • The Architecture: Offline-First PWA + Edge AI : Designed from the ground up to be offline-first, meaning the app shell, the vision model, and the game logic are all cached locally on the device. A FastAPI (Python) backend handles optional cloud syncing, therapist reporting, and subscriptions using paypal.

  • Core pillars

  • Edge computer vision (MediaPipe)

  • Multi thread processing (Web workers)

  • Local first data storage (IndexedDB)

  • AI orchestrators , Tool calling Agents for session analysis. a. The Cloud Report Generator (The Therapist/Parent Agent) : generates a plain language report that the app can translate based off session b. Dynamic Difficulty Adjustment (The Adaptive Coach Agent) : reasons through the entire session history to recommend the optimal next step c. Resource Recommendation (The Content Curation Agent) : uses tools to query a catalog and match resources (videos, PDFs, guides) based on the child's specific condition

Challenges we ran into

  1. The core technical battle was a "page unresponsive freeze" : Our initial architecture used OpenCV.js running on the main browser thread to detect colored blocks via HSV thresholding. The moment a user tried to tap the calibration corners, the browser would completely freeze and throw a "Page Unresponsive" dialog. The main thread was so bogged down by the synchronous WASM execution and getImageData readbacks that it couldn't process pointer events.

  2. The "Pinch" Flaw (The Accessibility Pivot): To simulate picking up and placing a block, we initially designed the game to require a "pinch" gesture (touching the thumb to the index finger). We quickly realized this was backwards: we were asking children with fine motor delays to perform a complex pincer grasp.

  3. The MediaPipe Worker Module Nightmare : MediaPipe's internal WASM loader expects to use importScripts(), which is blocked in modern Vite module workers. I wrote a custom polyfill at the top of hand-worker.js that intercepts MediaPipe's fallback import logic.

  4. Initial data sourcing for recommended therapeutic exercises and resource content for experts and professionals.

Accomplishments that we're proud of

  1. Proving edge AI can be smooth on cheap phones. By killing the freeze issue via a multi-threaded Web Worker architecture that runs MediaPipe at a smooth framerate on commodity hardware.

  2. The "Dwell" Pivot to ensure we designed for the actual user and not a demo.

  3. Dexplay being an Offline-First PWA That Actually Works Offline.

  4. A multi agent orchestration for a clinical backend - a therapist can supervise progress for a group of children because agents handle the paperwork and flag anomolies

  5. The Data Generation in a "Data Desert" : paediatric developmental data in Sub-Saharan Africa is severely limited. DexPlay creates the first longitudinal motor-progression dataset for children in low-resource settings.

What we learned

  1. Accessibility is mostly an architectural decision and not a UI feature.
  2. Strict thread management for browser based computer vision.
  3. Ai agents need strict guardrails in healthcare implementation
  4. Specialist gap is real and measurable for dexterity coaching.

What's next for DexPlay

  1. Clinical validation and pilot deployment in Kenyan clinics or special needs schools.
  2. Integration with Kenya's digital health infrastructure.
  3. Expand beyond motor skills to daily living.
  4. Technical debt and product polishing.

DexPlay's next step is to prove it works clinically in a Kenyan pilot, then embed itself into the Community Health Promoters workflow the Ministry of Health is already building, while expanding from single-hand dwell tracking to bilateral coordination and adaptive difficulty.

Built With

Share this project:

Updates

Submission history