Inspiration

Health information is scattered across apps, devices, and conversations. A workout happens on one screen, meals are logged somewhere else, and an AI assistant often starts without knowing any of that history.

We built Rhythm to connect those pieces: a personal AI health companion that remembers your context and helps you understand your progress.

What it does

Rhythm brings together workouts, meals, sleep, and conversations with your preferred AI assistant.

Its main feature is a camera-based workout coach. MediaPipe tracks body landmarks in the browser, and an exercise engine uses those landmarks to count repetitions and check specific aspects of movement.

When it detects a supported form issue, Rhythm explains the correction and shows a visual comparison:

A replay of your tracked movement.
A reference showing the intended movement.
A 3D comparison highlighting the relevant body part.

Checks use persistence and cooldowns to avoid correcting every brief wobble. When the camera cannot see enough of your body, the coach asks you to adjust your position.

A calendar brings meals, workouts, and sleep into a daily view, with a monthly health overview. The Chrome extension adds a way to connect relevant browsing interests with the wider health companion.

How we built it

We used Expo and React Native as the shared foundation for mobile and web, alongside platform-specific work for Windows, macOS, Apple Watch, and the Chrome extension. Our demo shows the different clients we built.

Clerk makes signing in easier. SurrealDB Agent Memory provides the shared context layer, allowing Rhythm to carry relevant preferences and history between devices and connected assistants.

The workout coach combines browser-based pose tracking, exercise-specific rules, and a Python backend. Claude or ChatGPT can interpret structured workout events and answer questions through connected tools. We also developed a local voice pipeline using Parakeet for transcription and Kokoro for speech.

Reference footage is converted into skeletal motion and retargeted onto 3D characters to demonstrate movements and support comparisons.

Challenges we faced

One challenge was separating what the camera actually measured from what an AI assistant could reasonably explain. We designed the coach around structured measurements so it does not need to invent repetitions or observations.

Other challenges included camera placement, noisy pose estimates, speech latency, cross-platform integration, and keeping context consistent between clients.

The current prototype includes both hosted and local components. The live website is an entry point; some camera, voice, and native demonstrations use the local development setup.

What we learned

A useful AI coach needs more than a capable language model. It needs reliable observations, relevant memory, clear limits, and feedback delivered at the right moment.

We also learned that sharing code across platforms helps, but each platform still needs its own testing and integration work.

What’s next

We want to validate exercise checks against more labelled recordings, improve voice responsiveness, strengthen cross-device continuity, and make the complete experience easier to deploy and use.

Rhythm is a wellness prototype, not a medical diagnostic tool. Its movement checks are limited to supported exercises and visible measurements.