Inspiration
Fitness apps send notifications that are easy to ignore. We wanted an accountability coach you can't swipe away: if you haven't moved enough today, it FaceTimes you and watches you work out.
What it does
- An Apple Shortcut sends your daily step count from Apple Health to our server.
- If you're under your goal, the server uses Chert's Calls API to FaceTime you.
- On the call, an AI coach (Gemini Live) sees you through FaceTime video, gets you in frame, and coaches you through squats, push-ups and jumping jacks with real-time form feedback.
- Our code detects whether you answered or missed the call. Missing the call or skipping the workout triggers a penalty: SOL you staked goes from an escrow wallet to a charity wallet on Solana devnet.
How we built it
- Chert FaceTime API for the phone line and outbound calls
- LiveKit Agents (Node) with Gemini Live native audio for the voice coach
- A custom video bridge that samples the caller's FaceTime camera at about 1 fps and streams the frames into Gemini, because the Node plugin drops video by default
- Answered/missed detection from Chert call events (bridge_ready)
- Solana devnet escrow with @solana/web3.js (in progress)
- MediaPipe pose detection for accurate rep counting (in progress)
Challenges we ran into
- The Gemini plugin discarded video frames, so we wrote our own frame pump to make the agent actually see the caller.
- Chert's API response shapes were different from what we expected, and that silently broke answered/missed detection until we fixed the event parsing.
- Calls are capped at 5 minutes and the agent has to greet within 10 seconds, so the whole workout flow had to be tight.
Accomplishments that we're proud of
The agent calls you on FaceTime, sees you, tells you to back up into frame, and coaches your squats live.
What we learned
Real-time video agents work, but you need to understand every layer: FaceTime bridge, LiveKit rooms, and how the model receives video.
What's next
- MediaPipe rep counting with a skeleton overlay sent back to the caller
- An Anchor smart-contract escrow so stakes are trustless
- Strava and Fitbit integrations
Built With
- claude
