LEGATO

Inspiration

This story starts with Louie, one of LEGATO's creators.

Freshman year, I liked a girl who looked like an angel - but more importantly, sang like one. One night during orientation week, we ditched whatever event we were supposed to attend and ended up in a church with a grand piano sitting open in the corner.

She wanted to sing.

She wanted me on piano behind her.

I'd been playing since I was five, but only from sheet music. Put Mozart, Bach, and Beethoven in front of me all at once and I wouldn't break a sweat. But I couldn't sit at that piano, listen to her land on a note, and find the right chord to play next.

That gap stayed with me longer than the night did. I understood harmony as something I memorized for an exam, not something I could use freely. Many classically trained pianists end up in the same place: drilled on notation for a decade, but never taught how to improvise.

I wanted to build the tool I wish I'd had that night - something that would let me create a transition myself, hear it immediately, and understand why it worked. Not another flashcard quiz, and not a progression generator asking me to trust its answer.

That idea became LEGATO.

What it does

LEGATO is an AI-powered progression coach that helps musicians explore the space between chords.

Users can build a chord progression, choose how long each chord lasts, and customize the transition between any two chords. LEGATO supports techniques such as passing diminished chords, secondary dominants, tritone substitutions, ii-V-I movements, suspended passing chords, scale runs, and arpeggio bridges.

Every change is immediately compiled into playable piano audio and rendered as animated sheet music. Users can hear the result, see exactly what is happening, and continue experimenting without leaving the composition workspace.

Our AI companion, Tenutino, moves through the score alongside the playback. Users can ask it to:

Explain a chord or transition
Suggest ways to improve the progression
Answer open-ended music theory questions

Instead of replacing the musician, LEGATO keeps the user in control. The goal is not to generate a finished progression with one click - it is to help people develop the intuition to create one themselves.

How we built it

LEGATO uses a lightweight, framework-free JavaScript architecture.

The chord and transition engine compiles every musical decision into one shared event structure. That same structure drives the audio, sheet music, animation, and AI context, keeping every part of the experience musically consistent.

We used:

VexFlow to engrave responsive SVG sheet music
Tone.js and the Web Audio API for piano sampling and transport scheduling
Three.js, WebGL, and GLSL for the animated particle score
OpenAI API for grounded explanations, suggestions, and multi-turn tutoring
SortableJS for drag-and-drop progression editing
Node.js for the server and secure API-key handling
LocalStorage for browser-based projects and conversation history

Tenutino’s position is calculated from the actual rendered measures. During playback, it follows musical time, moves between systems, jumps over notes extending above the staff, and returns to the most recently edited measure when playback finishes.

Challenges we ran into

The hardest challenge was synchronizing three different systems: audio time, engraved notation, and real-time animation.

Audio is scheduled by Tone.js, sheet music is positioned by VexFlow, and particles are rendered independently through WebGL. Early versions allowed each system to estimate progress differently. This caused notes, particles, highlights, and Tenutino to drift apart or jump at measure boundaries.

We solved this by making the audio transport the authoritative clock. VexFlow now reports the exact rendered position of each note, which we associate with its musical beat. Particle reconstruction and Tenutino’s movement follow those shared musical anchors instead of approximating time from screen position.

Another challenge was preserving visual quality without introducing lag. The notation uses tens of thousands of particles, so rebuilding geometry during playback caused occasional frame drops. We added particle preloading, sampling caches, deterministic reveal timing, and geometry locking during playback.

Music theory accuracy created its own challenges. Enharmonic spelling, tied notes, incomplete measures, generated transition material, and technique eligibility all needed to remain correct across different keys and time signatures.

This is also the first Hackathon for all three of us, and time management became a real problem. However, we managed to have everything done in time through effective teamwork. Cheers!

Accomplishments that we're proud of

We are especially proud that LEGATO feels like an instrument for learning rather than a form that produces an AI answer.

Users can make a musical decision, hear it instantly, see its notation, ask why it works, and then revise it. That tight feedback loop is the heart of the project.

We are also proud of:

Building a custom progression and transition engine
Keeping audio, notation, particles, and AI explanations grounded in the same musical data
Creating a responsive particle score without sacrificing notation clarity
Turning Tenutino into a companion that reacts to editing and playback
Supporting project persistence, import, export, and multi-turn tutoring
Building 97 automated tests for music theory, rhythm, playback, animation, persistence, and UI behavior

What we learned

We learned that synchronization is more of a data-model problem than a visual problem. Once audio, notation, and animation shared the same musical timeline, many seemingly unrelated interface bugs disappeared.

We also learned that AI is most useful in creative education when it is grounded in the user's actual work. A generic explanation of a secondary dominant is less valuable than an explanation referencing the exact notes, voicing, key, and transition the learner just created.

Most importantly, we learned that educational software does not need to feel like homework. Feedback can be technically rigorous while still being warm, playful, and encouraging.