SketchTunes

GIF
The drawing features and tools available for users on SketchTunes!
GIF
GIF
SketchTunes works by using the Gemini API to analyze drawings and composes music through text-only prompts to beatoven AI music generator
GIF
Create up to 4 storyboards and name them for easy access
Options to play, pause, retry and even adjust music by directly sending input

Inspiration

When we discovered the theme was entertainment, we wanted to explore a playful way to connect visual creativity with generative sound. Drawing is something anyone can do, either to doodle as a stress-buster, or to pursue their passion to convey something, but composing music can feel intimidating, especially without the technical knowledge or equipment. SketchTunes bridges that gap. You can turn sketches and doodles into personalized soundtracks so your art can finally sing. 🎨🎶

What it does

SketchTunes is a web whiteboard where you can draw or import images, then press Generate Music. The app analyzes your art, creates per-scene musical briefs, and composes a short track that matches the mood, style, and evolution of your drawing.

Features:

🎨 Multiple brush types, shapes, and color palettes
🖼 Import images or sketch freehand
🎶 Per-scene music generation with mood + transitions
⏯ Preview, download, and retry (communicate with the model) tracks
🗂 Multiple board options to create a background music for storyboards

How we built it

Frontend: Next.js (App Router) + React + react-konva for the drawing canvas, styled with Tailwind CSS
Backend: Next.js API routes for orchestration and persistence
AI/API:
- Gemini → converts drawings into structured notes with musical briefs (mood, genre, tempo, instruments, transitions)
- Beatoven.ai → generates full music tracks from those briefs
Persistence: MongoDB for accounts + optional board storage; localStorage for galleries
Auth: Email/password signup with JWT stored in HTTP-only cookies

Challenges we ran into

Building a smooth, low-latency canvas with multiple brush types while keeping export clean
Handling large base64 image payloads without crashing the server
Allowing draw, fill, and uploaded images to be treated the same like a regular drawing software
Prompt engineering for Gemini, then turning that into a Beatoven-friendly prompt
Polling long-running Beatoven jobs while keeping the UX responsive

Accomplishments that we're proud of

A seamless iterative UX where users can draw → generate → tweak → regenerate, all in one flow
A creative mapping pipeline: per-board analysis → refiner prompt → cohesive multi-segment music with smooth transitions
A fun, tactile drawing environment with thumbnails, undo/redo, background image editing, and multiple brushes
A working prototype and demo with all of the important features 👀

What we learned

Prompt engineering and design: creating the right prompt for the gemini API to analyze different types of drawings or images to create a structured, concise music brief for Beatoven to produce relevant music
UX features: adding options to export, retry, name boards, and download options to make the product truly usable and fun

What's next for SketchTunes

Creator tools: let musicians or anyone sketch art and compose tracks across multiple drawings and make background music for short films or movies
Games: “Guess the doodle from the music charades mode with multi-players
Therapy and Mindfulness: turn the project into a therapeutic system that can help people become more mindful through art and music, to help them feel calm

Ultimately, SketchTunes is a stage for creativity, making art and music creation playful, accessible, and social.