Mimetry

• 

  mimetry.live

• 

  Live usage demo

• 

  Demo 2

• 

  Demo 1

• 

  Demo 3

Inspiration

Mimetry was built to serve the needs of learning musicians wanting to develop their ear. For how important it is, we don’t really have a good way to teach ear training at the moment. Current apps are dull and dated, and there is a ton of conflicting advice out there on various internet wikis.

It’s something that has plagued both myself and all of the musicians I know at some point or another, so I wanted to tackle it this build week with my project Mimetry. When you’re jamming, you don’t have time to workout whether you’re hearing a 2-5-1, 1-5-6-4, a major 7th, or a tritone; your body needs to be able to feel it and rally the ball that’s been thrown at you.

What it does

In it's current iteration, it is a hands-free ear-training tool that allows musicians to deeply internalize musical knowledge via comprehensible inputs and an augmented reality interface.

How we built it

Mimetry was built using Codex from start to finish. I started by dictating stream of consciousness ideas for ear training tools that I would personally find helpful and beautiful. I had it return some options supporting libraries as well as their trade-offs. I then discussed scoping with Codex for the purpose of build week and scored the options against the judging criteria.

We then got to work on creative direction. I asked Codex what it needed from me to build a beautiful interface; it asked for an example of an interface that resonated, a music/performance visual that resonated, an AR interaction that didn’t feel gimmicky, a film moment that worked, and an anti-example of a current ear training app.

It got to work on V1, just getting the drone and it’s visual indicator to sound and show correctly. I tweaked the harmonics until it sounded soothing and not harsh, and then we started experimenting with gestures. It was important to me that we use gestures that actual musicians would use, but we needed to ensure that they weren’t competing with each other.

I had Codex implement a finite state machine to make sure that gestures weren’t exhibiting undefined behavior nor competing with each other when it didn’t make sense to. We then added in the listening cue gesture and the circle of fifths selection overlay, ensuring the new interactions which utilized a Face Mapping API, didn’t cause regressions. After some final formatting, we deployed the app.

Challenges we ran into

As we continued to add gestures, we started to play a game of whack-a-mole, so I recorded acceptance tests using photo booth and had Codex build a regression test harness to evaluate against with each new diff.

Accomplishments that we're proud of

I'm very proud of the novel ways in which Codex was leveraged throughout the course of the project. Codex was used in order to clean up messy human ideation, to creative direct, to scope and triage, to run tests, and to think through software design patterns that would be the most idiomatic to the use case. I also think that the final result is awesome and I continue to use it in my own personal practice.

What we learned

Media Pipe is very finnicky when it comes to managing sequences of many novel gestures. When you add in face mappings in order to detect things like ears, it becomes unavoidable to implement some sort of state machine to prevent weird behavior. When working with LLMs especially, test-driven design should be implemented as soon as is feasible, this way, as you continue to add new features, you can be sure you are not creating regressions. Last but not least, Codex is really awesome!

What's next for Mimetry

Futures versions of Mimetry (which are already being worked on) will support chord recognition, breaking down licks/riffs, interval active recall, and other locales.

Built With

• codex
• facemesh
• gpt-5.6
• mediapipe
• webaudioapi
• webrtc