boxing in mixed reality
robotic arms

Inspiration

Ever since covid-19, it has never been more clear that the physical fitness industry has grown exponentially. Multiple prominent market research and financial studies cite a noticeable shift in young people using their recreational time and income away from clubs and bars now for fitness studios, well being and mental health improvement (BBC, Forbes, Bloomberg). Due to the incredible advancements in the frontier of physical AI it's never been a better time to allow the users to 'feel' the presence of a trainer digitally and physically. This mixed reality approach introduces an interesting topic of debate towards improving livelihoods within our community, without having the social barrier of leaving a safe space. DEMO: https://youtu.be/qOlIdqNR1CQ

What it does

The primary interface for our users is the phone app which uses convex for type safe and fast authentication. We use these accounts to only collect non-personal data of the users and display anonymized health metrics only for the users themselves.

How we built it

Scoping a project so novel and technically interesting was a fun challenge. We collected our own data through teleoperation and trained a VLA model, after comparing tradeoffs and performance. We opted for a MediaPipe implementation for our V1 and modified a custom reBot arm to prioritize user safety. Our mobile app is web first to ensure it's cross compatible and we developed our VR application in a similar fashion to ensure maximum throughput towards the prototype.

Challenges we ran into

A key challenge throughout the weekend was using the new hardware. A particularly troublesome one was when we had to architect an AI reward system for an action of avoiding. This is a challenge we solved by opting for a tradeoff model due to our time frame.

Accomplishments that we're proud of

We managed to link physical AI with mixed reality. The experience is memorable to users and we effectively managed to build a useful tool with unfamiliar hardware.

What we learned

Many things about training robotics AI models. Such as the tradeoffs with throughput, inference and collecting reliable data from teleoperation. We also learned more about how to use the mixed reality mode of the Oculus, not simply an isolated sandbox.

What's next for Wax on! Wax off!

Towards our V2, we're going to improve the reward system and training for our AI model. We'll take greater care and more time to produce excellent data without the extreme time pressure of the hackathon. We also plan to see how we can make the project transferable to different robotic arms for a variety of budgets.