Echo Bridge

Turn silence into stories. One scan, three ways to explore.

Comment

Echo Bridge: Bridging Silence and Sound at the Museum

The Moment That Started Everything

It was a quiet Tuesday afternoon at Shanghai Natural History Museum. We were shadowing a deaf visitor named Chen as he explored the dinosaur hall. He stopped at the massive Mamenchisaurus skeleton, pulled out his phone, and scanned the QR code next to the exhibit for sign language interpretation.

The page never loaded. The website was dead.

Chen shrugged, as if this was expected, and moved on. But we couldn't move on. In that moment, we realized: China's Barrier-Free Environment Construction Law had been in effect since 2023. The Museum Accessibility Guidelines had been published in 2025. And still, the most basic digital accessibility tool — a QR code pointing to a sign language video — was broken.

That was the day Echo Bridge was born.

What We Discovered

The Invisible Divide

We spent two months embedded in the deaf and hard-of-hearing community in Shanghai. What we found surprised us:

  • Hearing-impaired visitors (those with hearing aids or cochlear implants) can detect sound but cannot parse speech in noisy environments. Museum halls are noisy. They described it as "hearing a river but not the words in it."
  • Deaf visitors rely entirely on visual information. The museum's only sign language resource was a broken website. One user told us: "I feel like I'm walking through a library where all the books have been blanked out."
  • The gap between policy and reality is wide. China's accessibility laws are forward-thinking, but implementation at the venue level is fragmented at best.

We also discovered a powerful insight: many hearing-impaired visitors carry smartphones. They already have the hardware. They just need the software.

$$ \text{Accessibility Gap} = \frac{\text{Available Information}}{\text{Accessible Information}} \gg 1 \quad \text{for most hearing-impaired visitors} $$

Our goal became: close this gap to unity.

The Pivot That Almost Broke Us

The Hardware Trap

Our first instinct was to look at what world-class museums do. Sweden's National Museum uses Hearing Loop (induction loop systems embedded in floors). The Roman Baths in the UK have FM transmitter systems. These are proven, elegant solutions.

We drafted a plan: Hearing Loop for hearing aid users, Bluetooth Auracast for modern devices, sign language kiosks for deaf visitors. It was comprehensive. It was expensive. And then came the constraint:

Software only. No hardware.

This felt like a crisis. How do you help someone hear without any hardware? But constraints breed creativity, and this one forced us to think differently.

The ASR Realization

Speech-to-text technology has matured dramatically. OpenAI's Whisper, iFlytek's ASR, and Alibaba's speech recognition all achieve near-human accuracy in controlled environments. The question was: could we make it work in a reverberant museum hall with background chatter and echoing floor tiles?

We ran experiments. In quiet conditions, word error rate (WER) was under 5%:

$$ \text{WER} = \frac{S + D + I}{N} \approx 4.8\% \quad \text{(quiet environment)} $$

In a simulated noisy hall, it jumped to 28%. That was unacceptable. But with front-end noise suppression (RNNoise + WebRTC filtering) before sending to the cloud ASR, we brought it back to ~9%:

$$ \text{WER}_{\text{denoised}} \approx 9.2\% \quad \text{(noisy hall + front-end denoising)} $$

Not perfect, but usable. And for the user — who currently has zero access — 90% accuracy is transformative.

How We Built It

Architecture at a Glance

Echo Bridge is a WeChat Mini Program built on the Taro cross-platform framework, which compiles to both WeChat Mini Program and mobile H5. The backend is Supabase — PostgreSQL for data, Edge Functions for serverless logic, and Storage for media assets.

┌─────────────────────────────────────────┐
│           WeChat Mini Program           │
│  ┌───────────┬───────────┬───────────┐  │
│  │ ASR Mode  │ Sign Lang │ Text Mode │  │
│  │ (caption) │ (video)   │ (offline) │  │
│  └───────────┴───────────┴───────────┘  │
│         ┌──────────────────┐            │
│         │  AI Assistant    │            │
│         └──────────────────┘            │
└────────────────┬────────────────────────┘
                 │
    ┌────────────┼────────────┐
    ▼            ▼            ▼
┌────────┐ ┌──────────┐ ┌──────────┐
│Supabase│ │Baidu Wenxin│ │iFlytek   │
│ DB +   │ │ LLM Chat  │ │ ASR API  │
│Storage │ │ + Multi-  │ │(Phase 2) │
│        │ │ modal AI  │ │          │
└────────┘ └──────────┘ └──────────┘

The Three Modes

We built three interaction modes because accessibility is not one-size-fits-all:

  1. Real-Time Captioning (ASR Mode) — The user points their phone at an exhibit, the app captures the museum's audio guide via microphone, and transcribes it into structured text with keyword highlighting. Designed for hearing-impaired users who can hear something but need text to fill in the gaps.

  2. Sign Language Video Mode — Pre-recorded sign language interpretation plays alongside synchronized rich text. Designed for deaf users who prefer visual language.

  3. Text Guide Mode — Structured图文 (graphic-text) pages with high-quality images and chapter-organized descriptions. Fully offline-capable. The universal fallback.

The AI Assistant

Every page has a floating AI assistant button. Users can:

  • Type questions about exhibits → answered by Baidu's Wenxin LLM
  • Upload a sign language photo → a multimodal AI interprets the sign and responds with relevant museum information

This was our "wow" feature. A deaf user can take a photo of their own signing and the app understands it. No typing needed.

The Admin Backend

Museum staff needed a way to manage content without developer involvement. We built a full CRUD admin panel with:

  • Exhibit creation/editing (name, description, images, videos, zone assignment)
  • Zone management (grouping exhibits by museum hall)
  • Image and video upload to Supabase Storage
  • Role-based access control (admin vs. regular user)

The Challenges That Kept Us Up at Night

Challenge 1: The 28% Error Wall

As mentioned, ASR accuracy in noisy environments was our biggest technical risk. We solved it with a two-stage denoising pipeline: WebRTC's noise suppression on-device, followed by spectral gating before sending to the cloud. This was our 10th iteration, and it finally brought WER below the 10% threshold we set for ourselves.

Challenge 2: Sign Language Is Not a Single Language

Shanghai Sign Language (SSL) and Chinese Sign Language (CSL) are different. Northern and southern dialects differ. Even within Shanghai, younger signers use different vocabulary than older generations. We realized we couldn't just "record one video per exhibit." Our solution was to design the platform for community-contributed sign language from day one — multiple versions per exhibit, tagged by dialect, ranked by community upvotes.

Challenge 3: The Offline Paradox

Museum basements have terrible cell reception. We needed the app to work without internet, but ASR requires cloud APIs. Our compromise: the text guide mode is fully cached offline (pre-downloadable when the user is on WiFi). The ASR mode gracefully degrades with a "network unavailable" fallback to the offline text content.

Challenge 4: Designing for Users We Aren't

None of us are deaf or hard-of-hearing. Everything we designed was filtered through a lens we don't naturally see through. We built a deaf advisory panel of 5 users who reviewed every screen, every interaction, every font size choice. They told us:

  • "This button is too small, I need to see it from arm's length while holding a phone in one hand and a coffee in the other."
  • "Don't use green for 'correct' — I'm colorblind, and so is 8% of the male population."
  • "Let me adjust the caption font size during the ASR session, not just in settings."

These insights were worth more than any technical optimization.

What's Next

We shipped Phase 1 — the MVP with three-mode browsing, QR scanning, AI assistant, and admin panel. But we're already building Phase 2 and 3:

  • AI Sign Language Avatar: A 3D animated character driven by text, eliminating the need to record videos for every new exhibit.
  • Visit Memory Generator: AI stitches your browsing history into a narrative souvenir — "At 2:23 PM you met a dinosaur that lived 150 million years ago..."
  • Community Sign Language: A platform where deaf users upload their own sign language interpretations, building a living, evolving library of exhibit knowledge.

What This Taught Us

Building Echo Bridge taught us that accessibility is not a feature — it's a design philosophy. It's not about checking boxes on a compliance list. It's about asking: What does the world feel like from where you stand?

We learned that constraints are not barriers — they're creative fuel. Being told "no hardware" forced us to discover software-only solutions that are cheaper, more scalable, and more maintainable than the hardware alternatives we first pursued.

And we learned that the best accessibility technology is invisible. A deaf visitor doesn't want to think about "using an accessibility tool." They want to think about dinosaurs. Echo Bridge succeeds when the technology disappears and the museum comes alive.

$$ \text{Success} \propto \frac{\text{User Engagement}}{\text{Technology Awareness}} $$

When the denominator approaches zero, we've done our job.

"I scanned the QR code, and for the first time, I could follow along with my daughter's questions about the dinosaurs. She didn't have to translate for me anymore."
— Early tester, Shanghai deaf community

Built With

Share this project:

Updates

Submission history