Inspiration
Right before the hackathon started, I sprained my ankle at a football game. And while I sat there coming up with ideas for a "New World", the pain was consuming me, and all I could really think about was: "What if I hadn't messed up my ankle? What if I lived in a world where I knew in advance so I never hurt it in the first place?"
A New World where I knew what not to do, a world where I could continually optimise every physical activity I do through a continuous feedback loop.
What it does
Optimise brings the progress being made in education through AI to the physical world. It not only open doors to this New World where people can optimise everything thing they do and where the barrier to entry to all these activities is lowered. But it also opens doors for each and every person into a New World, they haven't experienced before whether that's a new physical activity or the next level to an activity they already do.
You can upload a video of yourself doing any physical activity. Optimise will run human pose estimation and perform a kinematic analysis (fancy terms for finding your joints and figuring out what angles they make as you move, while also looking at other factors, such as the speed of a ball over time).
A 3D mesh is then made and animated for the scene.
AI agents then run a thorough analysis and, utilising the movement measurements, selected video frames, coaching research, and previous attempts, figure out possible lines of advice for you to improve your form.
A counterfactual simulation (if one thing were changed, like the release angle of a basketball shot, how would it affect the outcome?) is then run on the physics model, and practical advice is recommended, linked to the different evidence points.
How I built it
I built Optimise using React, TypeScript, and Three.js, alongside React Three Fiber for the interactive 3D visualisations and animated athletes.
The backend uses Node.js and Express, with a Python pipeline built using OpenCV, FFmpeg, and MediaPipe BlazePose to estimate body landmarks, track movements, and calculate joint angles. I also used EfficientDet for ball detection.
I implemented custom physics simulations in TypeScript, including basketball trajectories and football ball-response experiments. For the AI coaching, I integrated the OpenAI Responses API and Exa, allowing AI agents to examine video frames, analyse movement measurements, retrieve relevant sporting research, and investigate possible improvements. I also built a personal learning system that compares previous attempts and their outcomes.
For more detailed body reconstruction, I integrated an optional DuoMo motion reconstruction pipeline, running on NVIDIA GPUs through Modal. I also implemented a phone-pairing system so users can record on their phones and analyse their performances on a computer.
The application is deployed using Vercel and Modal.
Challenges I ran into
I'd say one of the biggest challenges was turning a 2D video into a stable 3D representation alongside meaningful physics simulations.
Camera angles can vary, as can other factors like lighting, and then you have a lot of missing information, such as a body part that might not be in view.
To overcome this, I used pretrained models to estimate body landmarks and reconstruct the body, added confidence checks, and synchronised the movement with the original video timestamps.
I then tested and refined the pipeline across different activities and recordings.
Accomplishments that I'm proud of
Getting this to work on a random new video in such a short timeframe was probably my proudest achievement throughout this hackathon.
What I learned
I learned just how much can be achieved in such a short timeframe by building on existing research, experimenting with different approaches, and taking advantage of recent advances in AI. Perhaps more importantly, I learned how much there still is to explore when bringing AI beyond the digital world and into physical activities.
What's next for Optimise
To optimise further, the biggest thing is increasing the amount of data the models have seen. In this way, the product is actually built to improve itself.
Let's say it gives you a recommendation, you implement it, and then it doesn't really work, so you upload a video of it not working.
If you've consented to your data being used for this purpose, that data could then be utilised to refine the models and improve future recommendations.
A lot of outer worlds experiences are going to be built for this track, however I truly believe that the best version of a, "New World", is just a slightly better version of this one.
