EchoTrail

Inspiration

Our hackathon isn't really about hearing loss — it's about the gap that opens up whenever no interpreter is available at all: a quick chat at a counter, a classroom aside, a moment where one person speaks and the other signs, and there's nothing bridging them. Professional interpreters can't be everywhere, and we wanted to see how much of that gap could be closed by things that already run in a browser, with no server and no API costs.

What it does

EchoTrail runs two channels side by side, live:

• Spoken → text. Your voice is transcribed continuously using the browser's built-in speech recognition, in any of 7 languages.
• Signed → speech. Your webcam feed is analyzed in real time. Seven common hand gestures (thumbs up/down, open palm, fist, peace, pointing up, "I love you") are recognized instantly for anyone — no setup. On top of that, you can teach it new signs live: hold a pose, type a label, and it's recognized from that point on.

Both sides land in one shared, scrolling transcript with timestamps and confidence scores — that's the "bridge." Everything else (session stats, exportable transcripts, saved session history) grew out of wanting the tool to feel complete, not just a tech demo.

How we built it

Everything runs client-side, in the browser, with no backend:

• Speech uses the Web Speech API — free, built into Chrome/Edge, zero setup.
• Gesture recognition uses MediaPipe's pretrained Gesture Recognizer model (WebAssembly, runs on-device), which ships with 7 gestures already trained.
• For gestures outside that set, we built a lightweight custom matcher: each hand pose gets converted into a position- and scale-invariant vector from its 21 landmarks, then compared by cosine distance against anything you've taught it. No training step, no GPU required, instant recognition — exactly what a weekend timeline needs.
• The whole thing is React + Vite, styled in a Spotify/Netflix-inspired dark UI with a persistent sidebar and a single dominant accent color.

Challenges we ran into

• Picking the right tradeoff on gesture recognition. Full ASL recognition needs a large trained model and lots of labeled data — not realistic in a hackathon window. We chose a hybrid: pretrained gestures for instant, zero-setup coverage, plus a simple nearest-neighbor matcher you can extend live. Being upfront about that scope, instead of overclaiming "ASL support," ended up being one of our better decisions.
• Camera resolution vs. reliability. Requesting a high resolution from getUserMedia caused intermittent timeouts on some webcams. We added an automatic fallback to a lower/default resolution so the camera reliably starts regardless of hardware.
• Getting priority right between "taught" and "built-in" gestures. Early on, a custom-taught pose could get overridden by a similar built-in gesture. We reordered the matching logic so your own taught signs always take priority, since you taught them on purpose.
• Keeping the UI honest. We got a lot of feedback suggesting AI-generated summaries, sentiment analysis, and fabricated "accuracy" percentages. We deliberately left those out — there's no backend LLM in this project, and shipping fake numbers or non-functional buttons felt like exactly the kind of thing that falls apart under a judge's follow-up question. Every stat shown (words captured, signs recognized, session time, FPS) is computed from real, live data.

Accomplishments that we're proud of

• A fully working, zero-backend, real-time bidirectional communication bridge that runs entirely in a browser tab.
• Instant gesture recognition that works for any user with no setup, layered with a live-teachable custom vocabulary — the best of both "works immediately" and "extensible."
• Every number on screen is real. Nothing is faked for demo purposes.

What we learned

How much is actually possible client-side now: real-time hand tracking, speech recognition, and text-to-speech, all for free, with no infrastructure. We also learned a lot about being disciplined under hackathon-feedback pressure — it's tempting to add every suggested feature, but knowing which ones would compromise the project's honesty (and which ones were genuinely worth building) turned out to be its own design skill.

What's next

• A trained classifier (instead of nearest-neighbor matching) once there's enough labeled data from real users
• Motion-based signs, not just static poses
• A real backend to support persistent, multi-user sessions and richer AI features like summarization, built properly rather than faked

Built With

• accessibility
• computer-vision
• css3
• gesture-recognition
• html5
• javascript
• localstorage
• mediapipe
• react
• speech-recognition
• text-to-speech
• vite
• web-speech-api
• webassembly
• webcam