Music Memory AI

Find the song you almost remember from incomplete lyrics, descriptions, scenes, moods, instruments, or voice clues.

Comment

Inspiration

Everyone has experienced a song sitting just beyond reach: a misheard lyric, a translated meaning, a movie scene, an instrument, or simply a mood. Traditional search works well when people already know precise words, but human memory is rarely precise.

Music Memory AI was built around one promise:

Find the song you almost remember from incomplete lyrics, descriptions, context, era, mood, or voice clues.

What it does

Users can type or record whatever they remember about a song, including:

  • Approximate or inaccurate lyrics
  • A translated meaning
  • A movie, series, advertisement, game, or social-media context
  • Era, genre, mood, tempo, or instruments
  • Singer or voice characteristics
  • Music-video or scene details

The application returns grounded, ranked musical works with specific evidence explanations, conflicts, confidence categories, available recording versions, and reliable YouTube links.

Users can confirm or reject candidates, answer a structured follow-up question, refine the same search session, and save discoveries to their account.

How we built it

Music Memory AI uses three retrieval strategies:

  1. Direct — fast deterministic retrieval for distinctive phrases or title fragments.
  2. AI-assisted — GPT-5.6 extracts structured clues and proposes diversified searches for vague descriptions.
  3. Smart — deterministically chooses between Direct and AI-assisted retrieval.

GPT-5.6 performs two bounded tasks:

  • Converting uncertain natural language into a typed SongMemory
  • Generating a small, validated set of diversified search queries

GPT-5.6 is deliberately not the ranking authority. It cannot insert unsupported songs, invent verified metadata, or assign arbitrary confidence percentages.

YouTube provides externally grounded candidates. Provider-neutral TypeScript then:

  • Normalizes titles, artists, and versions
  • Resolves uploads into canonical musical works
  • Aggregates evidence across official, live, lyric, translation, cover, and other versions
  • Ranks every musical work once
  • Selects the best representative recording for playback
  • Generates explanations from observable evidence only

The frontend uses React, TypeScript, and Vite on Cloudflare Pages. A Hono API on Cloudflare Workers protects credentials and coordinates OpenAI, transcription, and YouTube retrieval. Supabase provides authentication and account-owned Saved Songs.

Challenges

The hardest problem was distinguishing a musical work from a YouTube upload. Official audio, live performances, covers, lyric videos, shorts, and reactions may all represent the same song. Ranking them independently can fill the results with duplicates or incorrectly promote a recent performance.

We introduced deterministic candidate identity resolution. Recordings are conservatively grouped into canonical work entities, evidence is aggregated at the work level, and the best observable recording is selected only after song ranking.

Other important challenges included:

  • Preventing GPT output from becoming unsupported evidence
  • Diagnosing intermittent OpenAI and YouTube failures safely
  • Handling quota, timeout, malformed-response, and CORS failures
  • Making browser microphone recording work across Chrome and Safari
  • Preventing stale requests from overwriting newer state
  • Preserving sessions across navigation and authentication
  • Selecting useful follow-up questions without becoming an open-ended chatbot

How Codex accelerated development

Codex acted as an engineering collaborator throughout the project.

It helped with:

  • Repository analysis and architecture planning
  • React and Cloudflare Worker implementation
  • Provider-neutral domain modeling
  • Canonical song-entity resolution
  • Deterministic ranking and evidence provenance
  • Voice recording and real-browser debugging
  • Authentication and Saved Songs
  • Safe provider diagnostics and error classification
  • Benchmark traceability
  • Security review
  • Automated tests
  • Documentation and deployment

Important product and architecture decisions remained human-controlled, including the technology stack, deterministic ranking requirement, external-grounding invariant, provider boundaries, privacy rules, and release approvals.

The current release passes strict TypeScript checking, ESLint, 240 automated tests, Worker and frontend production builds, and all 24 recorded benchmark case/pipeline runs with perfect recorded Recall@1, Recall@3, Recall@5, and MRR.

Recorded fixtures provide deterministic development validation; they do not guarantee identical results for every future live provider response.

What we learned

The most important lesson was that song identification is not simply a search problem. It requires separating:

  • What the user remembers
  • What AI inferred
  • What an external source actually returned
  • Which uploads represent the same work
  • Which evidence genuinely supports the match

Keeping those boundaries explicit made the system more reliable, explainable, testable, and honest.

What’s next

Future work could evaluate one additional approved metadata provider, improve canonical metadata coverage, expand multilingual benchmarks, and further optimize AI-assisted latency.

The core principle will remain unchanged: AI interprets memory, external retrieval grounds candidates, and deterministic code decides the ranking.

Built With

Share this project:

Updates