Inspiration
The idea began during an AI development stretch after Global Build Week commutity event Tokyo.
After waiting a long time for an AI result, the output would sometimes be impressively useful. At other times it could be frighteningly dull, wildly off target, or simply not what I meant. Typing a carefully structured complaint every time felt like one more layer of work.
I looked at the MIDI keyboard beside me and had a much less serious thought:
What if, instead of typing immediately, I could hit the keys and go “BWAAANG”?
Maybe a loud chord could release a little frustration. Maybe a Major chord could mean “yes, this is good,” while a Minor chord could express doubt. Perhaps pitch could point to a rough position in the text. Even playing arbitrary chords while waiting for the next reasoning turn might make the interval feel less empty.
I had a playful personal hunch that music might become an interface beyond ordinary language. That was intuition, not a scientific claim about the brain, emotion, or productivity. The useful observation was simpler: playing felt immediate, physical, and enjoyable in a moment dominated by screens and text.
The problem was also larger than waiting. When I wanted to keep reviewing, I had to move through many changed files and keep notes elsewhere. The location and the observation easily became separated. At another moment, the right choice was to stop reviewing, make some noise, or take a different kind of break.
That became the central idea:
AI may work for minutes or hours. The human should have a way to remain human inside that loop.
What it does
Patch Notes supports two deliberately separate rhythms.
- In PLAY, notes remain music. Playing cannot move the review cursor, create feedback, or send anything to Codex.
- In CAPTURE, a single note moves to an approximate review location, and a recognized chord records an explicit review intent as an unsent Capsule.
The mappings are deterministic, not emotion recognition:
| Plain meaning | Capsule | Musical mapping |
|---|---|---|
| Looks good; keep this part | KEEP | Major chord |
| I need an answer or explanation | QUESTION | Minor chord |
| This needs to change | REWORK | Diminished chord |
No music theory is required. The interface presents the plain meaning first and shows the corresponding keys. MIDI hardware is optional: the complete Practice route also works with the onscreen Piano Ruler, mouse, and PC keys.
A chord never writes prose and never sends a prompt. It creates an unsent draft at an approximate location. I then return to the PC keyboard and type the exact comment in the Capsule detail field. I can refine the range, complete the Capsule, choose which Capsules belong in the next turn, inspect the combined preview, and explicitly send them.
If I hit the wrong known chord before typing or adjusting the range, the latest recognized chord replaces the intent of the same untouched draft. That makes one-handed exploration forgiving while my eyes stay on the document and HUD.
The interaction separates four decisions:
- pitch suggests approximately where;
- chord quality records which review stance I intend;
- typed text explains what I specifically want to say;
- explicit selection decides what actually returns to Codex.
How I built it
Patch Notes is a Windows desktop application built with Electron, React, TypeScript, Web MIDI, Web Audio, typed IPC, an authoritative Capsule state model, and a workspace-confined Git reader. Physical MIDI, the onscreen Piano Ruler, and PC piano keys all enter the same NoteEvent path. That path always feeds playable sound, while review gestures become active only in CAPTURE.
Codex is not merely credited as the tool that helped write the code. Codex CLI is part of the runtime architecture. In Live mode, Electron main starts or joins a localhost Codex app-server, discovers saved tasks, requires the human to select the exact task, and resumes it before enabling send. Observable Codex events are normalized into a restrained HUD and procedural BGM. Private reasoning and hidden chain-of-thought are not used as musical input.
The project itself was built through three stages:
- Ideation — ChatGPT with GPT-5.6 Pro. I used a long brainstorming conversation to test the intentionally loose idea: loud chords for frustration, Major for approval, Minor for doubt, and pitch for text position.
- Design — Codex GPT-5.6-sol, xhigh. Codex helped turn that idea into an architecture, MIDI grammar, Capsule lifecycle, safety invariants, Codex integration contract, acceptance criteria, and implementation plan.
- Implementation — Codex GPT-5.6-sol, medium. A separate Codex task used that plan to build the Electron shell, MIDI and audio paths, UI, state machine, app-server Adapter, tests, and Windows packaging.
The three stages show how the idea matured. The design phase challenged the emotional metaphor and converted it into deterministic mappings, an explicit PLAY/CAPTURE boundary, reversible drafts, and a rule that no musical gesture can send by itself. Implementation then turned those rules into domain state, UI behavior, tests, and a real Codex runtime path.
This was not a one-prompt generation process. I repeatedly pushed back on premature technical conclusions, protected safety requirements, questioned green tests that did not match physical behavior, and kept simplifying an interface that exposed too much of its internal machinery.
Challenges I ran into
Making a complex state model feel simple
Internally, the application has task discovery, attachment, observation, navigation, drafting, editing, completion, selection, preview, and sending. Early versions exposed too much of that lifecycle and required too many gaze shifts. I repeatedly reduced the visible experience to one Document, one vertical Piano Ruler, one Review Notes area, plain-language intents, and the PLAY/CAPTURE boundary.
The lesson was that deleting controls is not enough. The remaining interface must still make safety, current state, and the next action obvious.
Making physical MIDI actually work
Automated tests passed before the physical instrument behaved correctly. Synthetic events could verify the software path, but they could not prove that a real device was discovered, opened, subscribed to, and routed through a packaged desktop build.
Debugging exposed false-green event subscriptions, stale state, permission handling, port-opening behavior, and mapping gaps. Physical note input now works in the portable build on my development setup. Hardware coverage is still limited, so reconnection, long sessions, latency, stuck notes, and a wider range of controllers remain manual test targets.
Treating Codex connection state honestly
A saved task can remain discoverable even when a Codex CLI client is no longer actively attached. “Found a task” therefore cannot safely mean “ready to send.” I separated discovery, observation, resume, and send eligibility rather than collapsing them into one convenient but misleading connected state. The human must choose the exact task and explicitly decide what gets sent.
Keeping music safe
The most important invariant is that MIDI remains playable. A performance must not accidentally become a coding command. That requirement shaped explicit mode switching, deterministic mappings rather than emotion inference, unsent drafts, human-authored text, and explicit selective send.
What I learned
The first idea was emotional and deliberately playful: hit a chord instead of immediately typing another complaint at the machine. The finished interaction taught me that the instinct was valuable, but emotion inference was not the right product contract.
A musical gesture can be expressive for the person playing it while still having a deterministic meaning inside the application. Music can remain personal, ambiguous, and enjoyable in PLAY; review input can remain predictable, reversible, and safe in CAPTURE.
I also learned that an AI interface does not need to optimize every human moment for more output. Sometimes the useful action is to keep reviewing without losing context. Sometimes it is to make a loud sound, play a few notes, or walk away and let attention reset. A good tool can support those rhythms without judging either choice.
Green automated tests are evidence, not reality. Hardware response, listening quality, latency, and packaged visual interaction still require direct human observation.
Finally, the project reinforced the distinction that became central to the design:
AI can accelerate implementation, but the human must retain editorial agency.
Patch Notes does not infer the reviewer’s prose, silently select feedback, or automatically send what a gesture captured. The instrument helps preserve context; the human decides the meaning.
What’s next
My next priority is broader physical MIDI validation: more controllers, reconnection scenarios, longer sessions, latency measurement, and stuck-note recovery. I also want to keep simplifying first use and improve accessibility for non-musicians.
Patch Notes feat. MIDI began as a funny way to make AI waiting time musical. It became something more specific: a way for a developer to keep their place, preserve their thoughts, take a breath when needed, and decide exactly what returns to the AI coding loop.
Log in or sign up for Devpost to join the conversation.