EchoCrime

A detective mystery where user solves who committed a break-in.

Comment

Inspiration

Most detective games let you look for clues. We wanted one where you also have to listen. The idea started with one line: the security cameras were switched off, but one microphone kept recording. What if a faint sound on a tape could crack the case, and you could test your own theories by rearranging what you hear? We also wanted the game to be playable without a keyboard, using just your hand in front of a webcam.

What it does

The user is solving a break-in that was committed the night before in Riverside Museum. The user travels across the museum and collects clues of evidence along the way. As they collect evidence, the app builds a potential timeline of what happened, allowing the user to narrow down suspects. Once they have gathered evidence, the user is allowed to make a guess on who they think the suspect is.

How we built it

  • Game: React, TypeScript, Vite, React Three Fiber, Three.js, Rapier physics, Zustand, Tailwind and Framer Motion.
  • Art: CC0 models and PBR textures from Poly Haven, a re-textured Quaternius character with real walk, jog and idle animations, and public-domain paintings from the Met Open Access collection. Textures and meshes were compressed and decimated to keep the whole scene around 10 MB.
  • Audio: wavesurfer.js and the Web Audio API. Every sound in the case is generated by a reproducible DSP script (scripts/make-audio.mjs). "What if?" assembles the alternative tape in the browser with an OfflineAudioContext, and the same edits always give the same sound.
  • Backend: Node, Express and Zod. A rule-based parser turns sentences into structured edits, with an optional LLM parser behind an API key. The hidden solution lives only on the server: the browser can ask whether a clue links to a statement, and submit an accusation, but never receives the answer key until the case is solved or attempts run out.
  • Hand tracking: webcam landmarks from MediaPipe turn palm position, swipes and pinches into movement.
  • Quality checks: we wrote automated browser tests for movement and collisions, inspection, the timeline, the audio lab, the what-if feature, and the accusation, plus unit tests for the parser and the speed maths.

Challenges we ran into

  • A body with no clothes. The free character mesh came with only skin and underwear bands. We painted the coat, trousers and shoes onto its texture, using the mesh's bone weights to find which body part is which.
  • Honest audio. We had no real recordings and no AI sound API key, so we made the audio ourselves and said so in the UI instead of calling it AI.
  • Fair hypotheses. "What if?" must not leak a sound the player hasn't found yet. We restricted hypotheses to identified sounds and masked unidentified ones.
  • Keeping the answer secret. The solution had to stay off the client while still letting the theory board give meaningful feedback, so those checks run on the server.
  • Performance and layout. We cut the scene's draw calls and triangles, fixed doorway collisions (including a plant that blocked a doorway), and tuned the hand controls so they weren't twitchy.

Accomplishments that we're proud of

  • A complete, playable loop from first clue to verdict with a real win screen.
  • A waveform drawn from the actual audio file, with a test that checks it matches the file.
  • Hypotheses you can hear, and an interface that stays honest about what is evidence and what isn't.
  • A server-side solution that can't be read from the browser.
  • Hand-gesture control that works alongside the keyboard.

What we learned

  • Game feel comes from small details: a prompt that appears only when you're close, a camera that glides in for an inspection, a banner when the timeline updates.
  • Free assets can look good, but they need work: textures resized, meshes decimated, and materials fixed.
  • Writing tests that drive a real browser caught many bugs a quick play-through would have missed, and slow rendering taught us to test behaviour rather than wall-clock timing.
  • Being clear about limits (synthesised audio, no AI sound provider) makes the project easier to trust.

What's next for EchoCrime

  • Plug in a real AI sound-effect provider for "what if?", with the bundled clips as a fallback.
  • Real recorded audio, and more cases beyond the Midnight Theft.
  • Finish and tune the hand-gesture controls with real-webcam testing, then merge them.
  • Deploy a public build with the API server, and speed up first load.
  • Better visuals: more rooms, modelled clothing for the detective, and baked lighting.
  • More ways to question suspects, not only to test their statements.

Built With

Share this project:

Updates

Submission history