Petey.ai — Your On-Demand AI Physical Therapist

Inspiration

We are sports lovers and athletes. We love to go out and play the games we love. From weekly casual basketball runs to training to run marathons, everyone has a physical activity that they love to do. But in our experience, engaging in regular physical activities comes at an inevitable cost–injuries. And whenever we did, we found ourselves stuck in the same frustrating loop: waiting weeks or months to see a physical therapist, all while our injuries got worse.

We started digging and realized this wasn’t just our problem:

38% of insured Americans delay or skip care due to cost — Gallup, 2022
42% of those people get worse while waiting — Kiplinger
Starting PT within 2 weeks cuts emergency visits by 50% and leads to far better outcomes — Johns Hopkins Medicine, 2022

But physical therapy is full of friction. It’s expensive, time-consuming, confusing, and out of reach for too many people. So we built something that we wished we had: Petey.ai, a live AI physical therapist you can talk to instantly, through video and audio, whenever you need help.

What it does

Petey.ai is a conversational AI physical therapist that works through your webcam and microphone. You describe your pain, show the movement, and get back clear, thoughtful guidance in real time. It feels like you're actually talking to a human therapist—because that’s the experience we optimized for.

Here’s what it can do:

Talk to you live through natural voice and video conversation
Ask diagnostic questions and offer guided self-assessments
Provide recovery plans based on real PT best practices
Summarize your session with clear, simple takeaways called Petey’s Points
Export your full recovery plan as a PDF so you don’t forget what to do next

How we built it

We used Gemini Live to stream video and audio from the user directly to a language model capable of responding in real time.

We designed system prompts by combining Google's system prompt framework with our own curated knowledge base of physical therapy protocols and best practices, sourced from the web and processed through NotebookLM.
We built a React frontend to capture and stream webcam and microphone input to the model in chunks for easy processing.
To reduce friction and improve retention, we added Petey’s Points—a GPT-powered summarization feature that extracts key instructions after each session.
We also uset GPT to generate PDF summaries automatically, so users can refer back to their plans or share them with care providers.

The entire experience—from tech stack to voice UX—was designed to feel as human and helpful as possible.

Challenges we ran into

Streaming video and audio reliably into Gemini Live was difficult. We ran into latency issues, dropped streams, and all sorts of small things that disrupted the feel of a real conversation. Optimizing the flow by correcting issues with our audio and video buffers and streams was integral to making the flow feel natural and authentic.
Making the AI feel like a real physical therapist required extensive prompt engineering. Early versions were either too robotic or too generic. We spent time refining the structure, tone, and flow of conversation to make it feel supportive, accurate, and actionable.
Summarizing conversations into useful instructions wasn’t easy. We had to balance clarity, accuracy, and tone, especially for users who may be overwhelmed or in pain.

Accomplishments that we're proud of

We got real-time voice and video conversations working end-to-end with a language model. Petey can not only hear about your injury but can watch and observe how you move to get a far better understand of your ailment than audio can accomplish alone.
We created a clean, easy-to-understand summarization experience with Petey’s Points—solving a real pain point around information overload during medical conversations.
We built an AI that doesn’t just answer questions, but actually guides users through a recovery process in a way that feels intuitive and compassionate.

What we learned

Prompt design is everything. The difference between a helpful therapist and a clunky chatbot came down to how we shaped the model’s behavior.
Live AI conversations are powerful—but fragile. It takes a lot of engineering to make them feel seamless.
Users need retention, not just interaction. People forget most of what their doctor or therapist says. Giving them a clear, structured summary afterward dramatically improved the experience.

What's next for Petey.ai

This is just the beginning. Here’s where we’re headed:

Adding computer vision features to analyze your movement and give real-time form feedback
Partnering with certified physical therapists to validate protocols and keep improving accuracy and safety
Building integrations with insurance and EMR systems to help people get reimbursed or share care plans with real providers
Bringing Petey to mobile so you can use it right from your phone

We believe everyone deserves immediate, high-quality care when they’re injured. Petey is our first step toward making that a reality.