Tempo

Tempo turns any phone into a motion controller for browser-based boxing, bowling, and golf, with real physics, AI coaching, voice commentary, and health tracking. Just scan a QR code and play!

Comment

Tempo

Inspiration

  • https://www.youtube.com/watch?v=y-jGDvdZrgE
  • We grew up on the magic of Wii Sports, where anyone could walk into a living room, pick up a controller, and instantly feel like an athlete.
  • That experience faded with the Wii. Modern VR requires expensive headsets, and motion consoles rely on proprietary hardware that most people do not carry around.
  • We wanted to bring that feeling back and make it available to everyone, anywhere, using hardware they already own.
  • No app to install.
  • No special controller to buy.
  • Just scan a QR code with your phone, and suddenly you are in a 3D boxing ring throwing real punches.

What It Does

Boxing

  • Throw jabs, crosses, ducks, and slips using real phone motion sensors or your keyboard.
  • Fight 1v1, spar against AI, or watch two autonomous AI fighters battle it out.
  • Live LLM corner strategists coach AI fighters during matches.
  • Spectators can bet play chips on the outcome.

Bowling

  • Full 10-pin physics with realistic lane friction, ball spin, and pin deflection.
  • Aim with the D-pad.
  • Arm your throw with the phone's B button.
  • Release the ball with a real swing.

Golf

  • First-person 3D golf with wind, lie conditions, and a precision 3-stage swing meter.
  • Control your club, aim direction, power, and accuracy across multi-hole courses.

Phone as Controller

  • Scan a QR code with any iPhone or Android browser.
  • The phone streams gyroscope and accelerometer data over WebSockets.
  • Swing velocity, wrist rotation, and timing feed directly into game physics.

Webcam Head Tracking

  • A separate MediaPipe head tracker watches you through the laptop camera.
  • A fast, deliberate head snap becomes defence. Down is a duck, left and right are slips.
  • It joins the relay as a third controller beside the two phone slots.
  • So in one boxing match your hands can throw punches from the phone while your head slips shots on camera and a friend plays on the keyboard.

AI Fight Night

  • Two LLM-powered fighters have distinct personalities such as:
    • Brawler
    • Counter-puncher
    • Showboat
    • Stamina monster
  • Real-time OpenAI agents analyze match state and adapt strategies mid-fight.

Live Commentary

  • An ElevenLabs voice announcer calls every strike, knockout, eagle, and turkey in real time.
  • Web Audio ducking keeps commentary clear.
  • Comic-style captions sync with major game events.

Health Tracking

  • Tracks:
    • Calories burned
    • Active minutes
    • Punch velocity
    • Swing velocity
    • Range of motion
  • Uses metabolic energy formulas to estimate activity across every session.

Camera Vitals with Presage

  • With consent, the laptop camera reads pulse and breathing through Presage SmartSpectra.
  • Readings are taken at check in and recovery boundaries, never mid swing, because the subject has to be still.
  • A Wellness Lab page exposes source, validation, stability, confidence and freshness for every reading.
  • These are wellness readings by the vendor's own terms, not measurements for diagnosis or treatment.

Adaptive Tempo Sessions

  • Pick a goal and a length, ready up with the phone QR code and an optional camera, play, recover, and see why Tempo adjusted.
  • A deterministic adaptation engine nudges the next segment's challenge by a small bounded amount and explains the reason in plain language.
  • A language model is never in that decision path.
  • Keyboard only matches are labelled honestly as having no phone movement record.

How We Built It

Dual-Engine Architecture

  • Phaser 3 handles:
    • 2D game orchestration
    • HUD
    • Menus
    • Comic aesthetics
  • PlayCanvas handles WebGL rendering for:
    • The 3D boxing ring
    • Bowling lane
    • Golf course
  • Both engines share a unified 120 Hz physics tick.

Phone Motion Pipeline

  • HTML5 sensor data is streamed over WebSockets through a Cloudflare outbound tunnel.
  • A custom complementary filter fuses acceleration and rotation data.
  • The system detects clean swings with sub-20 ms latency.

The Controller Relay

  • A small Node service owns the phone relay, the agent, the vitals bridge and the database.
  • Three controller slots, two phones and one head tracker.
  • A hello handshake with a claim timeout, strictly increasing sequence numbers, and a ring of seen event ids per slot, so a retransmitted swing can never fire twice.
  • The relay never interprets motion and never knows what a punch is.

MediaPipe Head Tracking

  • A Python process using MediaPipe's BlazeFace detector learns a neutral head position from 25 frames.
  • A movement only counts when it is both large, about 40 percent of the detected face's own scale, and fast, above a velocity threshold in face units per second.
  • Your head then has to return near centre to re arm, with a cooldown between triggers.
  • That double gate is why leaning on the table does nothing while a real slip registers.

Deterministic Physics Simulations

  • All three sports run fixed-step simulations at 120 Hz.
  • Simulations are deterministic from a seed.
  • This makes gameplay:
    • Reproducible
    • Testable
    • Replayable

LLM Agents

  • OpenAI-backed corner coaches evaluate:
    • Round stats
    • Stamina
    • Knockdown counts
    • Fighter personality
  • Coaches issue punch sequences between exchanges without stalling game physics.
  • AI reasoning is fully decoupled from the main simulation loop.

ElevenLabs Commentary

  • 142 pre-synthesized audio lines cover major in-game calls.
  • The commentary system uses:
    • Priority queues
    • Staleness windows
    • Cooldowns
    • Web Audio ducking
  • This prevents overlapping or repetitive commentary.

Health Service

  • A local SQLite-backed Node server records every session's movement data.
  • A REST API exposes:
    • Daily goal tracking
    • Calorie summaries

Camera Vitals Through Presage

  • The Presage SmartSpectra Node SDK runs inside the service, so the key never reaches the browser.
  • The page opens the camera and pushes the same frames the player can see, so there is one permission prompt and no hidden sensor.
  • A quality gate decides what counts. A pulse needs a finite value in a supported range, the SDK's own stable flag, a confidence floor, valid framing, and a recent timestamp.
  • Advanced cardio models are deliberately not requested, and the bridge falls back to the core metric set when a face model is not provisioned on the key.
  • No frame is ever stored or forwarded over a game socket.
  • Missing key, missing camera, denied permission or bad light never blocks play. It falls back to phone motion and game performance.

The Adaptive Wellness Model

  • Phone motion is reduced on the device into one second summaries. Raw samples are never stored.
  • Each summary carries acceleration RMS, rotation RMS, active sample fraction, action count and peak action power.
  • Those become a bounded activity load with explicit per sport weights, because sustained boxing and a single deliberate golf swing are not the same kind of work.
  • Energy uses conservative Compendium bands with the resting component removed.
  • Adaptation is deterministic, bounded, and stores a reason code for every decision it makes.

Challenges We Ran Into

Phone Motion Latency

  • Mobile browsers throttle sensor events, introduce jitter, and behave differently across operating systems and hardware.
  • We built:
    • Sensor fusion
    • Peak-velocity thresholding
    • Client-side extrapolation
  • These systems make swings feel instant and fair.

Two 3D Engines on One Screen

  • PlayCanvas had to render seamlessly behind a Phaser game layer.
  • We coordinated:
    • Depth clipping
    • Resize handling
    • Shared timing
    • Viewport behavior

LLM Latency in Live Matches

  • A boxing match runs at 60 FPS.
  • AI decisions had to feel reactive without blocking the physics loop.
  • We fully decoupled AI reasoning from game physics.
  • Scripted fallbacks cover the gap when an AI response is not immediately available.

HTTPS for Sensors

  • Browser motion events only work on secure origins.
  • We built an automatic Cloudflare tunnel into:
npm run dev
  • It spins up an HTTPS endpoint with zero configuration.
  • Any phone on any network can connect in seconds.

Commentary Timing

  • Sports events are fast and bursty.
  • The announcer needs to:
    • Avoid overlapping lines
    • Skip stale events
    • Stay natural during rapid exchanges
  • We built a complete speaking-rules engine for sequences such as a knockdown followed immediately by a count.

Agents That Could Not Afford Their Own Combinations

  • Given a tool schema and a fight, the models happily scripted punches the fighter could not pay for.
  • They would gas out, then spend the rest of the round having punches refused by the simulation with no idea why.
  • Stating the budget in the prompt helped and was not enough.
  • The fix is a server side filter that drops unaffordable punches before they ever reach the ring, and swaps an emptied out script for a recovery beat so the fighter is not left standing still with his hands down.

Head Tracking That Ignores You Living in a Chair

  • A single displacement threshold fires constantly. People lean, they shift, they glance at their phone.
  • Requiring displacement and velocity together, plus a return to centre before the next trigger, is what made it usable.
  • The thresholds moved a long way from where we started.

Trusting a Physiology Signal

  • The hard part of camera vitals is not reading a number, it is deciding when a number deserves to be shown.
  • Confidence alone is not sufficient, because large motion and flickering light can produce a confident and wrong value.
  • Building the quality gate in front of the SDK, rather than showing whatever it last said, took most of a day and is the part of the health layer we would defend hardest.

Accomplishments That We're Proud Of

Zero-Friction Onboarding

  • Walk up.
  • Scan a QR code.
  • Throw a punch.
  • Under 5 seconds.
  • No install.
  • No account.

A Complete, Polished Game

  • Three fully featured 3D sports.
  • Menus and tutorials.
  • Pause screens.
  • Victory animations.
  • 2-player split view.
  • Health tracking.
  • AI commentary.
  • All shipped in one hackathon weekend.

Real Physics That Actually Feel Good

  • Bowling and golf simulations are seeded and deterministic.
  • The boxing engine models:
    • Stamina
    • Guard breaks
    • Staggers
    • Knockdown recovery
    • Round scoring

Three Input Paths in One Match

  • Keyboard, phone motion and webcam head tracking are merged every frame.
  • Any of them can be absent and the game does not notice.

Nothing in the Stack Can Ruin a Demo

  • No key, no network, no camera, no phone, no tunnel.
  • Every one of those has a path that still produces a playable, completable match.
  • The interface tells you which mode you are in instead of pretending.

A Health Layer That Says What It Does Not Know

  • Not measured, not calculated, no phone movement record.
  • We think an honest gap is worth more than a confident number.

What We Learned

  • How to fuse noisy mobile gyroscope and accelerometer data into clean, latency-tolerant gesture recognition.
  • How to architect a dual-engine game where a 2D framework and a 3D framework share:
    • State
    • Ticks
    • Viewport behavior
  • How to write deterministic, seed-reproducible physics simulations that are both fun to play and easy to test.
  • How to use LLMs as real-time autonomous decision agents operating under hard latency budgets inside a live game loop.

  • How to build a production-grade voice commentary system with:

    • Audio priority
    • Ducking
    • Variant picking
    • Caption synchronization

  • How to gate a physiology signal properly, using range, stability, confidence, framing validation and freshness together, rather than trusting any one of them on its own.

  • That a language model is a great source of intent and a poor source of truth. Everything an agent returns is validated, clamped and filtered against the simulation's own constants before it reaches the ring.

  • That browser platform rules decide your architecture whether you plan for them or not. Secure context requirements for sensors, one permission prompt per camera, and mixed content blocking on sockets shaped the design more than any feature decision did.

  • That refusing to show a number can itself be a feature.

What's Next for Tempo

Pairing Auth on the Local Service

  • This is first on the list, and the reason we describe Tempo as a demo rather than something to run on an open network.
  • The tunnel currently fronts the agent, controller, health and chip endpoints with no pairing secret, which is fine on a hackathon table and not fine anywhere else.
  • The fix is a pairing code carried in the QR code and checked on the hello handshake.

Online Multiplayer

  • Add WebRTC peer-to-peer matchmaking.
  • Let players challenge friends across the country, not just in the same room.

More Sports

  • Tennis
  • Baseball home-run derby
  • Ping pong
  • Other swing-based sports that naturally map onto the motion pipeline

Wearable Integration

  • Direct Apple Watch and Wear OS streaming.
  • Wrist-level precision tracking.
  • Real-time heart rate during bouts.

Tournament Mode

  • Bracket-style Fight Night tournaments.
  • Persistent chip economies.
  • Leaderboards.
  • Crowd betting.

Projector and TV Mode

  • Add a host mode that turns any HDMI display into an arcade cabinet.
  • Spectator links let the crowd:
    • Vote
    • React
    • Participate
Share this project:

Updates

Submission history