Abstract
Ok it's not actual magic... that would be blasphemy, buutttt. it's still pretty. darn. cool.
Murmuration turns every phone in the room into an AI musician with no conductor: each device communicates with the others, decides its own part and plays it from its own speaker, so the band keeps the music going even when the network or the server dies.
Everything is peer to peer. Agents are playing music themselves. All working towards the same harmonic goal. Someone said their judging criteria includes doing something you're excited about.... talkn 'bout me?? That's us right there..... yaaa
Watch our demo video, let our love (& excitement) adorn you: https://youtu.be/azHJPnBdplo
How we built it
- completely decentralized, each device plays it sounds from its own web client
- what's keeping the agents together is a seed that contains key, tempo, and chord progression
- the midi grid is calculated from a shared start time
How do agents decide what to play relative to what others are playing? For every phrase, each agent chooses how busy to play, which pattern, which register (how high or low), which instrument, and whether to echo or answer another part.
- LLM, gemini API is used to draft a decision and broadcast a general direction for everyone to follow.
- Jev (TypeSafe AI’s decision model, via OpenRouter) answers the same choices as typed questions, with a deadline. If it’s late, the rules decide alone.
- The final choice is sampled from the LLM’s weights blended with Jev’s probabilities. A Gemini “director”. Every four phrases, any device can ask Gemini for a direction (energy, which part is featured, who drops out). The swarm adopts a proposal by majority vote.
Challenges we ran into
The biggest challenge was definitely running out of claude usage; smart networking (using siblings' accounts) and intelligent pacing were our lifelines during the past 24 hours.
More related to murmuration itself, some interesting problems we had to deal with were
- how to have effectively 0 latency communication between the agents, even as the flock scales to 100s of phones?
- in the same spirit, how do you get all these (famously slow) agents to individually decide what music they are going to play before the time they are ? Music with latency simply isn't music anymore.
Intermission
congrats, you made it more than halfway through our description. We are immensely grateful for the time & attention you have bestowed upon our project.
In murmuration, agents are interlinked in working towards the Nobel ideal of harmony.
interlinked interlinked. InterlinkeD , interlinked: https://www.youtube.com/watch?v=ZRcpnM26nJM
Accomplishments that we're proud of
sounds amazing
moe is a really good UI designer
AI can play music wow
This is much more than just an LLM wrapper. Firstly, it's a cool use of modern, light networking. Secondly LLMs are only one of 3 artificial intelligence decision components in each agents live reasoning, the other 2 being a proprietary statistical model along with a decisionAPI called Jev, or JevAI.
The above decision stack was the key to being able to play good music within the constant, very short decision window.
What we learned
Listening to the same sounds for 10 hours at a time is quite irritating
Different AI tools are good at very different things
The difference between a sinewave and square wave
Ouverture
Could we use a similar architecture to model an actual murmuration? What is the true murmuration behind this all?
Built With
- care
- css3
- free-midi-chords
- gemini-api
- google-ai-studio
- groove-midi-dataset-(magenta)
- html5
- javascript
- jev-(typesafe-ai)
- love
- midi.js
- momsaffection
- musyngkite-soundfont
- node.js
- openrouter
- playwright
- pop909
- qrcode
- salamander-grand-piano
- tone.js
- tonejs/midi
- web-audio-api
- webrtc
- websockets
- ws