Inspiration
The main concept was inspired by the story of how JRR Tolkien's universe was created as described in the Silmarillion. The story describes divine spirits who create the world through their music. However, within them there is one being that means to corrupt the creation.
What it does
EÄ is Tolkien's creation myth built for real. Four AI agents, each running on its own phone, see only part of a scene: two see the photo, two read different captions. With no conductor, they agree on a mood by passing messages and play one piece of music together. A big screen paints every note as it sounds. Corrupt one agent (Melkor) and the others catch it by the wrong notes it plays, then take over its part. Cut the network and each agent carries on alone, and the music doesn't collapse.
How we built it
- Perception: four different models (DINOv2, ResNet-50, MPNet, MiniLM) on real COCO photos and captions. A learned rotation translates each model into one shared space, which gives nine musical moods.
- Agents: isolated Web Workers that can only pass messages. A hub sends one start time, then goes silent for good.
- Timing: each agent plans the next bar with exact timestamps, and each phone syncs its clock to the laptop.
- Output: Web Audio for the sound and a canvas painter that sees only the notes, never the photo.
Challenges we ran into
- Keeping four phones in time over a hotspot.
- Getting four different AI models to share one language.
- Catching Melkor without accusing honest agents. Honest agents sometimes echo its wrong notes on purpose, which first caused 60 false accusations. Excusing those notes brought it to 0.
Accomplishments that we're proud of
- 53.5% mood accuracy on 200 unseen scenes, against 36% for the best guess.
- Listening to each other adds about 15 points.
- When the network is cut, EÄ falls back to working alone and never below. A single-leader design collapses.
- Melkor is caught in about 3 bars, with 0 false accusations in 400 runs.
What we learned
- With no leader, nothing breaks when one part fails.
- What an agent does is better evidence than what it says.
- A simple, well-tested method beat fancier ones on small data.
What's next for EÄ
- Run the models fully on the phones.
- Test against real human mood ratings.
- Catch subtler traitors that lie but play correctly.
- Let the audience join in from their own phones