MetaCare

Inspiration

MetaCare started with the experiences of two of our teammates. Enzo lives in a rural community in Southern Virginia (SOVA), and Sabal lives in rural Nepal. In their communities, reaching the nearest healthcare can mean traveling roughly one to two hours.

That distance becomes even harder to manage for someone with chronic pain or limited mobility. We kept coming back to the same problem: how does someone get help when the condition they need assessed also makes it difficult to reach a doctor?

We wanted to build a way for patients to complete guided movement assessments at home. They could describe their pain, watch a movement demonstrated in VR, and share a recording and joint measurements with their doctor for review. That became the idea behind MetaCare.

What it does

MetaCare is a VR prototype for guided movement assessments, with an initial focus on the hips, knees, and ankles. It is designed to collect information a clinician can review.

In the workflow we're developing, a patient puts on the headset and hears, "Explain what pain you are experiencing." They answer aloud, and the assistant asks follow-up questions about their symptoms before guiding them through tasks from a predefined assessment library.

A green demonstration avatar shows each movement. Visible paths indicate where to move, and labeled color cues show progress: red for not started, yellow for in progress, and green for completed. Spoken instructions accompany the demonstration.

The movement recording is intended to capture estimated joint positions and angles. If a patient misunderstands a task, the system can demonstrate it again. If pain or limited mobility stops them from completing it, the assessment should record that limitation and let them stop.

The planned doctor portal brings together the symptom history, original recording, skeletal replay, and AI-generated observations. A clinician can review those records, consider possible causes, and decide whether further evaluation is needed.

How we built it

We organized the prototype design around the path from the patient's first answer to the doctor's review.

The proposed architecture uses Gemini Live for symptom intake and follow-up questions, with ElevenLabs providing spoken instructions and feedback. The VR interface handles demonstrations and motion paths. Movement tracking supplies the estimated joint positions used in the replay, while Tiger Data stores timestamped measurements and assessment records. Gemini's analysis would appear in the doctor portal alongside the source recordings, so the clinician can check the observations against the movement itself.

Joint angle

For a knee, let $\mathbf{h}$, $\mathbf{k}$, and $\mathbf{a}$ be the estimated hip, knee, and ankle positions in the same coordinate system. Define the two segment vectors:

$$ \mathbf{u}=\mathbf{h}-\mathbf{k}, \qquad \mathbf{v}=\mathbf{a}-\mathbf{k} $$

The angle between them, in degrees, is:

$$ \theta= \frac{180}{\pi} \cos^{-1}\left( \operatorname{clip}\left( \frac{\mathbf{u}\cdot\mathbf{v}} {|\mathbf{u}||\mathbf{v}|}, -1,1 \right) \right) $$

An estimated bend angle relative to a straight leg is $\phi=180^\circ-\theta$. For example, an included angle of $150^\circ$ corresponds to a bend of $30^\circ$.

The clipping operation prevents rounding errors from putting the input outside the valid inverse-cosine range. Missing points or zero-length segments must be excluded. With 2D keypoints, this measures a projected image angle; 3D measurements require 3D coordinates.

Observed range of motion

Across the valid frames of one movement:

$$ \mathrm{ROM}=\max_i(\phi_i)-\min_i(\phi_i) $$

For example, a recorded bend angle ranging from $10^\circ$ to $80^\circ$ gives an observed range of $70^\circ$.

Angular velocity

The estimated rate of angle change between successive valid samples is:

$$ \omega_i= \frac{\phi_i-\phi_{i-1}} {t_i-t_{i-1}} $$

When time is measured in seconds and angles in degrees, $\omega_i$ is measured in degrees per second. Its magnitude gives angular speed.

Difference between sides

For comparable left- and right-side assessments:

$$ D_{\mathrm{ROM}}= \left| \mathrm{ROM}{L}-\mathrm{ROM}{R} \right| $$

This expresses the difference in observed movement range in degrees. These measurements describe movement; they do not independently identify a condition or establish a diagnosis.

The skeletal replay is intended to visualize estimated joint movement. It is not a direct measurement of internal structures such as tendons.

Challenges we ran into

Lower-body tracking is the hardest technical problem. We need to establish which movements the tracking setup can measure reliably and how much uncertainty remains in the joint positions it produces.

An incomplete movement is also difficult to interpret. The patient might have misunderstood an instruction, the tracker might have lost a joint, or pain might have limited the movement. The system needs to distinguish these situations before deciding whether to repeat a task.

Cleaning up recordings creates another problem. Smoothing can reduce tracking noise, but it must preserve movement differences that a clinician may need to see.

Accomplishments that we're proud of

We're proud of how the design accounts for both the patient and the doctor. The patient gets a demonstration and spoken guidance. The doctor gets the symptom description and movement evidence together, with a way to review the recording behind an AI observation.

We also made voice interaction central to the design. Someone completing a movement task should be able to listen to instructions and ask for help without repeatedly navigating menus.

What we learned

A skeletal replay needs an explanation of what was measured and what was estimated. Its appearance alone tells us little about its accuracy. The same applies to AI output: an explanation or confidence score still needs evidence and clinical review.

We also learned to pay attention to why someone cannot complete a task. Difficulty following an instruction and difficulty moving require different responses. That affects the instructions, the option to pause or stop, and what the doctor sees afterward.

What's next for MetaCare

Our first priority is to make one assessment work reliably from beginning to end, from the symptom conversation through movement capture and replay to the doctor's review.

We then want to compare the tracking and calculated measurements against reference measurements. Working with clinicians will help us choose appropriate tasks, define when an assessment should stop, and establish when a patient needs further care. The doctor portal also needs more work, including patient consent controls and protection of patient data.

For rural use, we want to add multilingual guidance and support for limited connectivity. We also hope to explore programs where providers lend or ship headsets to patients. Clinical studies, applicable regulatory requirements, and potential insurance reimbursement would need separate work before MetaCare could be offered as a medical service.