Steplet

Inspiration

For the injured and the elderly, physical therapy only works if exercises are done correctly at home. Without proper guidance, small mistakes go unnoticed and can make things worse over time. Most patients leave their clinic with a paper prescription they barely remember how to follow. We wanted to close that gap.

What it does

Steplet turns any physical therapy prescription into an interactive at-home coaching experience. You upload your PT sheet and Claude reads it, extracting every exercise into structured data. From there, Steplet generates an animated guide showing exactly how each movement should look. A built-in voice assistant called Claude walks you through the exercise and answers any questions. As you move, Steplet detects your form in real time and flags bad reps before they cause further harm.

How we built it

We built Steplet on Next.js 14 with Tailwind CSS handling the frontend. Claude serves two core roles: parsing PT prescriptions from uploaded images or PDFs into validated structured data, and generating skeleton keyframe data for each exercise. Remotion takes those keyframes and renders deterministic template-based animations using a custom forward-kinematics rig. Voice guidance runs through the Web Speech API and the whole app is deployed on Vercel.

Challenges we ran into

Getting Claude to produce consistent and physically accurate skeleton keyframe data was the hardest part. We had to design a detailed forward-kinematics rig with joint angle references and few-shot examples to get reliable outputs. Mapping those angles into smooth and realistic Remotion animations without any AI-generated video required careful parameterization of every composition.

Accomplishments that we're proud of

We built a full pipeline that goes from a scanned PT prescription to a playable voiced animation in under 30 seconds. The skeleton rig and keyframe generation system can represent a wide range of exercises without any hardcoded video, making it genuinely scalable.

What we learned

Prompt engineering for structured spatial data is a fundamentally different challenge than working with text. Getting a model to reason about joint angles and body mechanics required rethinking how we communicated the problem entirely. We also discovered how powerful Remotion is for deterministic data-driven animation.

What's next for Steplet

The immediate next step is live camera-based form detection using pose estimation. Beyond that we want to build out a full library of PT compositions and adaptive exercise modifications based on user constraints, with a mobile app so patients can follow along anywhere.