Inspiration

Hand and wrist rehab exercises are repetitive, and people stop doing them at home. We asked: what if the exercise were a game? In Tilt Sketch you draw by moving your phone, so every drawing is wrist and forearm movement, and an AI makes it fun by guessing what you drew.

What it does

Draw with the gyroscope: aim the phone and the sheet moves under a fixed brush.
On-device AI judge: it shows its live guess while you draw and explains its top 3 guesses with confidence. The score combines how well the AI recognizes the drawing with how much detail you drew compared with real drawings of that object.
Play with friends: Solo, Teams (relay drawing) or Impostor (one player gets a similar object, then everyone votes). It works on one phone or many; join with a QR code or a 4-letter code.
Challenges: a daily challenge with a shared leaderboard, 40 levels on a path map, and 30 guided line-art drawings with no time limit.
Wrist practice: adaptive tracing exercises that measure tremor, smoothness, range of motion and accuracy, with progress charts and a printable report. Patients can share their results with a therapist through a clinician dashboard.
English and Spanish, accessibility options, and an installable app that works offline.

How we built it

Custom AI model: a CNN with 1.1M parameters trained on 395k Quick, Draw! drawings across 78 objects plus an "other" class. We augmented the drawings to look like gyroscope input (tremor, rotation, cut-off strokes) and calibrated the model with temperature scaling. It runs in the browser with TensorFlow.js.
Wobble cleaning: every stroke is resampled, smoothed and simplified before judging. The exact same step runs in training and in the browser.
Results on held-out gyroscope-style drawings: 82.3% top-1, compared with 38.7% for DoodleNet, plus 91.9% top-3 and a calibration error of 0.004.
Real examples: before each drawing, the game shows 3 real drawings the model recognizes best, so players know the shape the AI expects.
Motion analysis: a 4–12 Hz tremor band, jerk-based smoothness and range of motion, computed from the orientation stream.
Multiplayer: rooms run through Supabase Realtime, so up to 12 phones can join on campus Wi-Fi or mobile data.
Backend: Supabase with row-level security for leaderboards, opt-in drawing donations, anonymous feedback and clinician sharing.
Quality: 16 Playwright end-to-end tests in CI with simulated gyroscopes, including 8 phones joining a live room.

Challenges we ran into

The off-the-shelf model collapsed on real phone drawings: a wobbly house dropped from 75% to 9%. Fixing that took gyro-style training data plus stroke cleaning. Training had to fit a hackathon, so we sped up rendering about 10x with OpenCV and used mixed precision on a free Colab T4. Direct phone-to-phone connections failed on campus Wi-Fi, so we moved rooms to a realtime relay. Players also showed us that "the AI recognized it" isn't the same as "it's a good drawing", which is why the score now includes detail.

Accomplishments that we're proud of

Our own model more than doubles DoodleNet's accuracy on gyroscope drawings. The game gives real-time AI feedback, and the full rehab mode with clinician sharing was built in a weekend.

What we learned

Real input is noisy. The best AI fix was matching our data and preprocessing to how people actually draw with a phone, and testing with real players early.