Inspiration

Learning an instrument usually starts with buying one, and controllers for music or CAD all cost money. We wanted to see how far a sheet of paper and a laptop webcam could go.

What it does

Draw shapes on paper and take one picture. Gemini labels every shape, and each one becomes something you can play by tapping the paper: boxes are piano keys, circles are drums, triangles are marimba. Write a note like F# inside a box and it plays exactly that note.

  • Song mode lights up the next note, so a beginner can play a real melody.
  • Write JOY, ZOOM and HOME in three shapes and the paper works like a 3D mouse for a CAD model.
  • Hands mode needs no paper: pinch and drag to spin, stretch and roll a 3D cube.

Impact

It helps people learn to play an instrument without buying one, and it lets you control CAD and other 3D software without extra hardware like a 3D mouse.

How we built it

It's React and TypeScript running in the browser. One Gemini call reads the shapes and handwriting as structured JSON, in about 2 seconds for $0.002 a scan, with the API key kept on the server. We then trace the actual pen lines so taps land exactly where you drew. MediaPipe tracks your fingers, and since one webcam can't see touch, we detect a tap from how your fingertip drops and stops. Web Audio plays real piano samples plus five instruments we synthesize ourselves.

Challenges we ran into

A laptop webcam looks down at the desk at a steep angle, so a tap barely moves in the image. Gemini's positions were sometimes off by half a key, and gray ruled lines and handwriting confused our tracing. Every fix got its own unit test.

Accomplishments that we're proud of

Going from a photo of a drawing to a playable instrument in about two seconds, using nothing but paper and a laptop.

What we learned

AI is great at figuring out what's on the page, and plain geometry is better at finding exactly where it is. Using both beat either one alone.

What's next for PaperPilot

Tune taps on more webcams, add lessons for beginners, and host it at paperpilot.us.

Built With

Share this project:

Updates

Submission history