QuackThat!

We fed your favorite songs to an AI goose. Now you have to guess what it's quacking (It doesn't feel like honking). Everyone is suffering.

Comment

Inspiration :goose:

We've all heard a song and recognized it before a single word was sung — that melody recognition is almost muscle memory. We wanted to stress-test it. What if the song was replaced entirely with goose quacks, but quacks that faithfully mirrored the original rhythm, pitch, and syllable stress? Would you still recognize it?

That's the dumb, beautiful question at the heart of QuackThat!

The name is a nod to rubber duck debugging — but our spirit animal was never a duck. It was always a goose. Specifically, Harnold: a goose who honks with purpose, who waddles with conviction, and who absolutely did not ask for this responsibility but stepped up anyway.

What it does

QuackThat! is a mobile music-guessing game where every song is re-rendered entirely in goose quacks. The quacks aren't random — they're a phonetic translation of the original vocal melody, channeled through the ancient and noble language of goose:

  • High-stress syllables → loud QUACK
  • Normal syllables → quack
  • Held notes → quaaack
  • Clipped syllables → qua

You hear Harnold's interpretation, you pick the song. Points scale with how fast you answer and how long your correct-answer streak runs.

Two modes:

  • :globe_with_meridians: Online single-player — your score goes to a global leaderboard
  • :iphone: Local pass-and-play — multiple players rotate on the same device

How we built it

The core of the project is a fully automated AI audio pipeline — designed from the ground up around one question: what would this sound like if a goose sang it?

  1. Firestore stores song metadata with iTunes 30-second preview URLs
  2. Gemini 2.0 Flash receives the actual audio clip and generates a syllable-level quack script — mapping each sung syllable to a goose variant that captures the original melody's rhythm and stress
  3. ElevenLabs Speech-to-Speech converts the original iTunes preview audio into the quacked version using a custom voice-cloned goose (meet Harnold), preserving exact pitch and timing
  4. Generated MP3s are uploaded to Firebase Storage, with the quackAudioUrl written back to each Firestore song document

The app is built on React Native + Expo with Expo Router for file-based navigation. Our design language — Unhinged Doodle Core — features wobbly borders, warm cream backgrounds, smudge shadows, and Fredoka One / Patrick Hand fonts. Harnold presides over all of it.

Game state runs through a strict state machine: Scoring formula per correct answer:

  • Base: 1000 points
  • Time bonus: +10 per 0.1s remaining on the 15-second timer
  • Streak bonus: +100 per consecutive correct answer

Challenges we ran into

The hardest part was the audio pipeline. Getting Gemini to produce quack scripts that were rhythmically faithful — not just phonetically plausible — required careful prompt engineering around stress patterns, note duration, and phrase boundaries. Early drafts were completely unrecognizable. Harnold was not pleased.

ElevenLabs Speech-to-Speech was the breakthrough. Rather than feeding in a text quack script (which loses all timing), we feed in the actual song audio and let S2S do the rhythm-preserving translation through our cloned goose voice. The output is eerily recognizable in a way plain TTS never achieved.

Cross-platform audio playback in Expo had more edge cases than expected — particularly expo-av behavior differences on Android vs. iOS. Designing pass-and-play mode also required careful Firebase write policy work: only online mode writes to Firestore; local session scores stay on-device only and never touch the global leaderboard.

What we learned

  • Multimodal AI (audio-in, structured-data-out) is powerful but brittle at the prompt boundary — small wording changes to the Gemini prompt produced dramatically different rhythmic fidelity
  • Speech-to-Speech TTS is massively underrated for creative applications
  • React Native's navigation model rewards you heavily for designing your state machine before building any UI
  • A goose is not a duck. Harnold taught us that. We will not make that mistake again. :goose:
Share this project:

Updates