Remend

On-device movement tracking that helps people understand their range, consistency, and progress.

Comment

Inspiration

Movement practice outside a supervised setting can be difficult to understand and sustain. Most consumer fitness tools are designed around broad, able-bodied movements and universal targets. They rarely account for smaller facial or hand movements, one-sided mobility, or the reality that a meaningful range of motion is different for every person.

I built Remend around a simpler idea: start with what a person can comfortably demonstrate today, help them practice from that personal baseline, and make their progress visible without requiring wearables or uploading sensitive camera footage.

Remend is a personal activity tool, not a medical device. It does not diagnose conditions, prescribe treatment, or replace a qualified professional. Its purpose is to make independent movement practice more accessible, understandable, and private.

What it does

Remend turns an iPhone or iPad camera into a real-time movement companion. On-device computer vision tracks face, hand, upper-body, and lower-body movement. Exercise-specific evaluators convert derived landmarks into signals for repetitions, holds, tracking quality, and routine progress.

For supported exercises, users first establish a comfortable personal range. Remend then evaluates activity relative to that starting point instead of imposing a universal range. Exercises can also be configured for the left side, right side, or both sides when anatomically relevant.

Completed sessions become local activity summaries and privacy-preserving 3D replays. Users can compare their initial demonstrated range with later exercise attempts, review consistency and completion, and add optional context about comfort, effort, or assistance.

Remend also demonstrates how this derived data could support future assistive AI research. With explicit opt-in, the app can remove identifiers, media, free text, absolute timestamps, and face geometry before contributing normalized movement records. A prototype retargets human hand trajectories to a simulated LEAP robotic hand, making the pipeline visible:

human movement → derived 3D geometry → normalization → embodiment-specific retargeting → simulated robotic motion

This does not claim that recovery movements directly become robot control policies. It proves that privacy-preserving movement data can be structured, contextualized, retargeted, and inspected in a robotics workflow.

How we built it

The mobile application uses Expo SDK 57, React Native, TypeScript, Tamagui, React Native Reanimated, and a custom native iOS module. MediaPipe models estimate face, hand, and pose landmarks on the device. A shared exercise engine handles setup, tracking-quality gates, repetitions, holds, rest periods, personal targets, and routine completion while preserving modality-specific signal processing.

User history, personal targets, preferences, and derived replay data are stored locally with versioned persistence. Raw camera frames and video are not uploaded by default. Apple Foundation Models and an optional local Gemma model can generate evidence-grounded insights on supported devices, with deterministic summaries available as a fallback.

The deployed Google workflow uses Cloud Run, Vertex AI, Gemini 2.5 Flash, Firestore, Cloud Storage, Cloud Build, and Artifact Registry. Gemini analyzes structured robotics episode metrics and explains what an episode demonstrates and what validation is still missing. It also supports a human-reviewed outreach workflow by evaluating operator-supplied lead information and drafting personalized outreach. Firestore stores lead workflow data, while a private Cloud Storage bucket receives only validated, explicitly opted-in movement contributions.

Challenges we ran into

The largest challenge was creating one coherent experience across face, hand, and body tracking without pretending those modalities behave identically. Each requires different framing, smoothing, confidence checks, coordinate handling, and exercise logic.

Mirrored front-camera coordinates also made anatomical left and right more complicated than visual screen position. Replay performance required careful separation of durable session data from animation state. Routine completion exposed state-management edge cases where exercises could finish visually without fully releasing the active routine.

Privacy introduced productive constraints. The system needed enough longitudinal detail to show meaningful progress and support future research without turning into a repository of identifiable video. That led to an architecture centered on local inference, derived measurements, explicit consent, strict contribution schemas, and separate local video controls.

Accomplishments that we're proud of

Remend progressed from a standard Expo starter into a working TestFlight application with multimodal camera tracking, a broad movement library, personal-range calibration, side-aware exercises, custom routines, local history, 3D replay, and grounded insights.

The Google Cloud backend is deployed and has completed verified Gemini calls through Vertex AI. The research pipeline can validate and store de-identified contributions in a private bucket, and the LEAP Hand prototype demonstrates embodiment-specific retargeting rather than presenting a static robotics concept.

Most importantly, the core experience remains useful even without cloud access or a generative model.

What we learned

Personalization should begin with demonstrated ability, not an assumed ideal. AI-generated language is most useful when constrained by observable evidence and prevented from drifting into diagnosis or prescription.

I also learned that potentially valuable robotics data requires more than coordinates. Task context, personal range, exercise phase, tracking confidence, assistance, and outcome labels may be as important as the trajectory itself.

What's next for Remend

Next, I will improve replay comparisons, validate tracking across more devices and movement abilities, and organize the exercise library into cohesive goal-specific programs. I also plan to conduct structured user testing and pursue partnerships with qualified rehabilitation and assistive-technology researchers.

The longer-term goal is a privacy-first platform that helps people understand their own movement today while, through explicit and carefully governed participation, contributing to safer and more adaptive assistive AI in the future.

Built With

Share this project:

Updates