Inspiration
After a stroke, the hand isn't broken. The brain's wiring to it is. Recovery requires one thing: hundreds of repetitions where the patient tries to move and the movement actually happens. That pairing is what forces the brain to rewire itself. But a weak hand can't move on its own, so the patient can't practice. They compensate with the good hand, the weak one atrophies further, and a 45-minute therapy session fits maybe a few dozen reps.
Clinical intent-gated robots solve this. They cost $20,000+, live inside hospitals, and serve one patient at a time. We built the same closed-loop neuroplasticity engine from $30 in parts, deployable at a bedside or a kitchen table, with a therapist still in command, and then asked: what happens when that therapist has a thousand patients, or a hurricane disconnects all of them at once?
What it does
NeuroGrip's robotic hand refuses to move unless the patient's brain is actively trying to move it.
- The headband detects intent. A capacitive sensor at the temple picks up the involuntary muscle co-activation that fires every time the brain sends a motor command, even to a paralyzed hand. It answers one question: is the patient trying right now?
- The camera captures trajectory. A webcam tracks the patient's healthy hand through MediaPipe and extracts per-finger curl values in real time.
- The robot executes conditionally. Five servo-driven fingers on a tendon-actuated hand mirror the healthy hand, finger by finger, but only while the headband confirms active effort. Wave the good hand while relaxed and the robot does nothing. Focus, and it follows.
This is not passive mirror therapy. This is an intent-gated, vision-guided, AI-adaptive neurorehabilitation system built from four interlocking subsystems:
1. Live Session Engine. Real-time rehab with per-finger tracking, rep counting, activation rate, adaptive difficulty, and a live 3D hand visualization. It operates across five neurological conditions: stroke recovery, Parkinson's monitoring (tremor FFT + finger-tap decrement test), phantom limb mirror therapy, pediatric cerebral palsy, and cognitive decline screening via reaction-time degradation.
2. AI Clinical Reasoning (NVIDIA Nemotron). A multi-step agent with six callable tools that doesn't just summarize: it reads session metrics and patient history, checks live alert state, adjusts the difficulty threshold on the physical device, triggers patient feedback, opens clinical incidents, and explains every decision with full tool-call traceability on the dashboard.
3. Clinical Command Center. 10 automated detection rules generate incidents covering sensor loss, camera failure, tremor spikes, cognitive latency regression, failed sign-ins and unauthorized access. Each incident carries an owner, severity, status and live updates. Every action, login and blocked request feeds a tamper-resistant audit log. The entire system operates fully offline if the network drops.
4. Mass Casualty Surge Response. A disaster disconnects 1,200 rehab patients from care simultaneously.
- Every patient receives an explainable composite risk score. Hard-coded safety rules flag anyone with 3+ days without a session or a declining trend as urgent, regardless of what the AI says.
- Therapists triage to the highest-risk patients first. Remaining patients land on a ranked waitlist with automatic escalation alerts, ensuring zero silent drops.
- Nemotron generates a coverage plan to bridge the gap.
Patients access their own portal, where Gemini explains their progress in plain, encouraging language they can actually understand.
How we built it
Hardware. An ESP32 headband reads the temple capacitive pad and transmits wirelessly over ESP-NOW to an ESP32-S3 hand controller. The hand board drives five SG90 servos on a tendon-actuated mechanism (braided fishing-line tendons, rubber-band return springs) and relays sensor telemetry to the laptop at 50 Hz. If servo commands stop for 500 ms, firmware-level watchdogs open the hand automatically. If the headband signal drops, the system reports zero intent and the hand releases rather than staying clamped on a patient's fingers.
Computer Vision. MediaPipe extracts 21 hand landmarks per frame. Per-finger curl is computed as the ratio of fingertip-to-wrist distance versus knuckle-to-wrist distance.
Frame-to-frame exponential smoothing and a 3° deadband eliminate servo jitter from camera noise.
Intent Detection. The capacitive reading $x$ is compared against a per-patient relaxed baseline $b$. A two-phase calibration (relax, then maximum effort) measures each individual's full-effort delta $D
The difficulty level $\ell$ represents the fraction of the patient's own maximum effort required to activate the robot. This is the parameter Nemotron adjusts. Additional signal processing includes 80 ms debouncing, adaptive baseline drift correction, and artifact rejection for movements exceeding 2.5x the patient's calibrated effort range.
Clinical Signal Processing.
- Tremor detection isolates the dominant frequency between 3-12 Hz via FFT on wrist accelerometer or camera-derived motion data (Parkinsonian tremor: 4-6 Hz, essential tremor: 8-12 Hz):
$$f_{\text{tremor}} = \arg\max_{3 \le f \le 12 \text{ Hz}} \left\lvert \mathcal{F}{w \cdot \tilde{x}}(f) \right\rvert$$
- Finger-tap decrement test quantifies rate, rhythm, and amplitude decay from first to final third, a standard Parkinson's assessment metric:
$$\delta = \frac{\bar{A}{\text{first}} - \bar{A}{\text{last}}}{\bar{A}_{\text{first}}} \times 100\%$$
- Rep validation requires the hand to close past 60% while intent is active and reopen below 30%, preventing false counts from passive movement.
All outputs are screening flags, never diagnoses.
Software Architecture.
- A Python engine runs the core 50 Hz sensor-vision-actuation loop
- Flask + Socket.IO dashboard updates at 10 FPS with a Three.js 3D hand model
- Nemotron operates through NVIDIA's hosted API in a genuine tool-calling loop with full reasoning traces. Gemini powers the patient-facing view. MongoDB Atlas stores all session data
- Role-based access control with route-level enforcement and sign-in lockout for therapist and patient roles
- Three operational modes: full hardware, webcam-only, and fully simulated with synthetic patient data
- 180 automated tests cover the engine, agent tools, triage logic, incident rules and access control
Challenges we ran into
Capacitive sensing is deceptively hard. Bare foil on skin reads as constantly "touched," and head movement or gripping a power bank shifts the baseline unpredictably. We layered foam padding, per-person calibration, temporal debouncing, artifact rejection and continuous drift tracking to extract a reliable intent signal. One critical bug caused intent to spike to 100% at random intervals: the baseline had been captured with the headband off the head. That failure is why the Zero-recalibration button exists.
A silent camera failure. Frames occasionally arrived under 1 ms apart with identical timestamps. MediaPipe rejected the duplicates and the camera thread died silently while the dashboard still showed "Camera OK." We implemented strictly monotonic timestamps and automatic camera recovery.
Our AI model disappeared mid-build. The Nemotron variant we integrated was retired from NVIDIA's hosted catalog during the hackathon. We migrated to Nemotron Super with its reasoning configuration under time pressure.
The AI was reviewing empty sessions. Our first implementation saved and reset counters before triggering the AI review, so Nemotron confidently analyzed 2-second sessions with zero reps. We restructured the pipeline: review first, reset after.
Honest metrics matter. "Bilateral symmetry" always read ~100% because the robot is copying the healthy hand by definition. We replaced it with activation rate: the percentage of session time the patient's own effort was strong enough to move the hand. That number is honest and clinically meaningful.
Power under load. Five servos gripping simultaneously pull ~3 A. That required an isolated battery pack, a shared ground plane, and a smoothing capacitor, all on a mechanical hand held together with hot glue and fishing line.
What we learned
- The hard part of rehabilitation technology is not moving a motor. It is deciding when the motor is allowed to move, and measuring that decision honestly.
- Calibrating to the individual patient beats any universal threshold, because every person's "maximum effort" is different, and it changes as they recover.
- An AI agent proves its value when it changes something real, like the difficulty setting on a physical device, and shows its reasoning. A summary without action is a report, not an agent.
- At scale, patient safety rules belong in deterministic code that executes on every patient, every cycle, not only the patients the AI model happens to evaluate.
- Build for failure first: firmware watchdogs, self-recovering camera threads, graceful degradation modes. In a live demo, recovering from a failure is more impressive than any feature.
What's next
- Moving from an external robotic hand to a soft actuated glove worn directly on the patient's impaired hand.
- A clinical pilot with physical and occupational therapists to validate our metrics against standardized clinical outcome scores.