Battle Scene and Menu Options
Starting Screen
Skill Selection and Video picker and Enemy Target
Chest Reward and Inventory/Item(Potion)
Setting Options and Credit Textbox
MediaPipe Tracker

Inspiration

Back in middle school, right in the middle of my mid-term exams, I suffered a severe knee injury where a split-off piece of bone actually shattered. I almost blacked out but tried to tough it out and tanked the pain until I got home. By the next day, my leg had swollen massively and I was rushed to the doctor.

Fortunately, the bone healed quickly, but the psychological block was much harder to fix. I was fearful to use that leg, and the sheer tedium of traveling all the way to a clinic every single week for hours of repetitive rehab completely drained my motivation. Because I avoided doing the physical therapy, one of my legs remains weaker to this day.

I don't want anyone else—especially kids and the elderly—to go through that. This is incredibly personal to me because my mom also broke her foot a few years after I broke mine, and she didn't even go to physical therapy recovery because of the high cost. That's why making this cheap and accessible was a massive priority. Inspired by fitness apps and the all-ages accessibility of Wii Sports, I realized we don't need expensive consoles to make recovery engaging. I wanted to turn tedious physical therapy into an accessible, personalized RPG adventure that you can play with just a standard camera/phone.

What it does

PhysRPG: Gamified Physical Rehab is a retro turn-based(task/time ATBbar) RPG that uses your real-world physical movements to drive combat. Instead of pressing 'A' to attack, I designed the game so players must physically perform therapeutic movements (like a cross-body punch or a high-knee march).

The game securely records the motion and evaluates the player's form locally using edge AI. Good form deals critical damage and rewards the player with loot (like health potions you can store in your inventory), while poor form results in a miss, encouraging proper posture and safe mobility scaling.

How I built it

I started by planning the idea into four distinct parts: the AI, Federated Learning (FL), the game, and the tracker, with the goal of linking all of them together.

The Tracker: I started on the tracker first because MediaPipe looked cool and getting it working early helped up my momentum. I implemented Google MediaPipe (Tasks API) in Python to track human skeletal landmarks.
The Game: Built in Godot 4 (GDScript). I reused an old game I made for a class to get a head start on the ATB (Active Time Battle) logic. From there, I built out the inventory, save states, animations, combat targets, and GUI focuses.
The Edge AI API: I used FastAPI to create a local microservice to receive the video from Godot.
Federated Learning (Architecture): To ensure biometric privacy (Responsible AI), I designed the system architecture to utilize Flower and PyTorch.

While the current hackathon prototype uses MediaPipe for instant on-device grading, the structural foundation is laid for a privacy-first training pipeline. To protect user privacy, raw video is never uploaded. Instead, I structured the system to use the Federated Averaging (FedAvg) algorithm to update a global model using only localized weight deltas. The central server is designed to compute the new global weights using this formula:

$$ w_{t+1} = \sum_{k=1}^K \frac{n_k}{n} w_{t}^k $$

(Where $w$ represents the model weights, $n_k$ is the number of local data samples on the client, and $n$ is the total number of data samples across all clients).

Challenges I ran into

Because I started with the tracker, and tracking the user's movement and pose is such a big part of the project, I tried to make it so Godot could accept both a live camera feed and video uploads. I looked up tutorials, set up the scripts, and compiled the shaders. But after all that work, I realized it couldn't pick up any camera feed—not even a virtual OBS camera. I finally looked deep into it, read the documentation, and saw that the whole CameraServer class simply can't run on Windows. I really should've read the docs first. I had to pivot entirely to an asynchronous file-upload system.

I also struggled massively with time management. I spent way too much time in Godot trying to make a parallax background, but it didn't work and took up too much time. Because I spent the vast majority of my time on the game stuff, I had to rush the AI and FL parts at the end, and I completely ran out of time to try putting the backend on Render or making it a App.

Accomplishments that I'm proud of

I am incredibly proud of building a fully functional end-to-end pipeline completely solo. Taking four disparate pieces of technology—a Godot game client, an asynchronous Python FastAPI server, Google MediaPipe, and PyTorch architecture—and successfully linking them together was a massive leap for me. Watching the game successfully trigger a retro RPG attack based purely on my real-world physical form score in the demo video made all the frustrating bugs worth it.

What I learned

Theory vs. Implementation: My college multimedia classes honestly didn't do much to teach me the basics of APIs and AI. I also learned the hard way that studying what FedAvg is and making academic research reports on it is not the same thing as actually programming it yourself. Building this forced me to learn the practical fundamentals on the fly.
Read the Docs: Assuming cross-platform engine features work identically on Windows cost me valuable development hours.
Scope and Time-boxing: A hackathon requires brutal limits. I should have set strict limits on the time I spent making the game stuff. Some of it took way too much time, which is why I couldn't deploy to the cloud.

What's next for PhysRPG: Gamified Physical Rehab

In the future, I want to replace the file-picker workaround with a native WebRTC live-camera integration so the game feels completely seamless. I also plan to fully deploy the central Flower Server to Render, allowing the AI to dynamically calibrate to the unique mobility limits of users with chronic pain across the globe—all without ever compromising their visual privacy.