REBOUND — Biomimetic Sonar Navigation

Every phone is a bat waiting to happen.

Comment

REBOUND — Biomimetic Sonar Navigation

Every phone is a bat waiting to happen.

https://annatchijova.github.io/vigia/rebound-demo.html

https://github.com/annatchijova/rebound

https://rebound-olga.duckdns.org/

Inspiration

Picture walking into a hospital lobby you've never set foot in. Now do it without seeing it — not the layout, not the reception desk, not the person walking straight toward you. A white cane tells you what's already close enough to touch. A guide dog tells you what's ahead — if you're lucky enough to have one, afford one, and live somewhere that trains them. For almost everyone else, the world just happens to you, one step behind knowing.

Bats solved this problem 50 million years ago, and they didn't need eyes to do it. They emit a chirp, listen to the echo, and build a picture of the world from sound alone — in real time, mid-flight, in total darkness.

We kept circling back to one thing: every blind person we could think of already carries the exact sensor that idea needs. A speaker. A microphone. We didn't need to invent new hardware. We needed to stop treating the phone as a screen — the one part of it a blind user can't use — and start treating it as a bat.

What it does

REBOUND turns a phone into a sonar, end to end:

  1. Emits a biomimetic CF-FM chirp (8 kHz constant tone + downward sweep) through the speaker.
  2. Captures the returning echo through the microphone, in a latency-aware window that survives Android's variable audio output delay.
  3. Deconvolves the echo into a Room Impulse Response using an adaptive Wiener filter.
  4. Extracts acoustic features — mel spectrogram, MFCCs, RT60, spectral centroid — and classifies the space with a lightweight CNN into one of six categories: open space, nearby wall, doorway, corner, corridor, or stairs.
  5. Runs an independent staircase detector as DSP reinforcement, because a drop-off is a different kind of danger than a wall — it doesn't get to rely on the CNN alone, not when being wrong once is the whole ballgame.
  6. Hands the classification to a Qwen-powered Memory Agent, which remembers what it has learned about this specific person across sessions — where they hesitate, where they walk straight through — and turns that into a spoken instruction personalized to them, not to a generic user.
  7. Delivers that instruction through four independent channels at once: earcons shaped by rhythm and melodic contour, live synthesized speech, distinct haptic vibration patterns, and ARIA live regions that feed VoiceOver and TalkBack directly. A navigation aid that only works when you're looking at the screen has already failed the one person it was built for.

No app store. No extra hardware. Scan a QR code, tap once, and the phone in your pocket becomes a bat.

How we built it

The signal chain is pure DSP plus a small PyTorch CNN — 296K parameters, light enough to run on a CPU-only cloud instance, no GPU required in production. The reasoning layer is where Qwen-Plus earns its place: it doesn't just caption a classification, it runs an actual memory system — episodic memory for individual events, semantic memory for consolidated patterns, and Bayesian priors per user that shift based on how that person actually moves through the world — all mediated through the DashScope API, fully async, so one person scanning doesn't make another one wait.

The frontend is an install-free PWA built on the Web Audio API, because the one thing we refused to compromise on was asking a blind user to navigate an app store account they might not have set up, or an install flow built around a screen they can't see.

The backend runs on Alibaba Cloud ECS, behind Caddy with a real Let's Encrypt certificate — self-signed certs make both iOS and Android refuse microphone access outright, a lesson that cost us most of an afternoon before we understood why nothing was working.

Challenges we ran into

Latency was the whole ballgame, and we refused to guess. REBOUND started deployed in Singapore. It worked — but every scan carried almost a full second of round-trip latency, and for a tool meant to warn someone about a wall, "noticeable" can mean "walked into it before the warning arrived." That's not an edge case we could shrug off. So instead of assuming the LLM call was the bottleneck, we curled /health — an endpoint that does nothing but return a JSON blob — and it was already over a second on its own. That ruled out Qwen and pointed straight at geography. We stood up a second Alibaba Cloud ECS instance in US-Virginia at 2am, migrated the full stack, and the latency dropped hard — not because the code got smarter, but because we stopped assuming "cloud" means "close enough" and went and checked.

A second, subtler bug was hiding behind the first. /predict — the most CPU-expensive endpoint in the system — was running its full DSP+CNN pipeline synchronously inside the async event loop. Two people scanning at once weren't just competing for CPU; they were fully blocking each other, silently. We moved the pipeline to a thread pool behind a bounded semaphore, so it now degrades gracefully with a 429 instead of freezing invisibly — which mattered more for judges testing the live demo at the same time than the region migration alone ever could have fixed.

iOS Web Audio punishes hesitation. AudioContext unlock, speechSynthesis unlock, and DeviceOrientationEvent.requestPermission() all require a live user gesture on iOS — and any await before they fire burns that gesture silently, with no error, no warning, just a phone that stays quiet. We rebuilt the entire start sequence so every permission request fires synchronously inside the tap, before a single await executes, because the alternative was a navigation aid that failed exactly the users it existed for.

We turned the audit on ourselves before anyone else could. Unauthenticated endpoints, unbounded per-user memory growth, no upper bound on payload size — this is behavioral data about people with disabilities, and we didn't want to find out about a hole the way most teams do. We found these ourselves, ranked them by severity, and fixed the ones that mattered before a judge ever touched the QR code.

Accomplishments that we're proud of

  • 95.3% validation accuracy across 6 acoustic space classes, with 0.30 m mean distance error — from a 296K-parameter model light enough to run on a CPU, not a GPU cluster we'd have to apologize for.
  • A Qwen memory agent that actually forgets — not everything is stored forever; stale or contradicted observations get their confidence reduced or dropped, decided by the LLM itself, the same way a person's trust in a place updates over time.
  • Four accessibility channels working together, not bolted on at the end: earcons, speech, haptics, and screen-reader live regions, so the app degrades gracefully to whichever channel a given user actually has available, instead of assuming they all do.
  • 88 automated tests passing, and a published adversarial security audit with every finding — fixed or disclosed — instead of quietly buried in a private doc no judge will ever read.
  • Getting the whole pipeline, cloud migration included, stable enough that a judge could scan a QR code cold and sweep an unfamiliar room in under a second per scan — and trust what it told them.

What we learned

That "cloud latency, nothing we can do" is almost always a hypothesis, not a conclusion — it takes one curl to /health to find out which one it actually is. That accessibility isn't a feature you bolt on at the end; it decides architecture decisions from the first line of code, because a UI that assumes the user can see it is a UI that has already excluded them before a single pixel is drawn. And that the most honest thing a hackathon team can do with limited time isn't to hide the gaps — it's to point at them directly. Our dataset is synthetic. Our system can't yet tell you which direction a wall is in. We say so, because those limitations aren't a footnote to be embarrassed about — they're the actual roadmap, and pretending otherwise would only slow down the people who might build the next part.

What's next

A dedicated wearable, not a phone. A stereo microphone pair would solve the one thing no single phone microphone ever can — spatial direction. An onboard low-power inference chip would cut round-trip latency further. And removing the phone entirely solves the real usability cost of asking someone who already carries a cane to also hold up a screen they can't see. Alongside that: closing the sim-to-real gap with field-recorded RIRs, ideally collected in partnership with actual blind and low-vision users — because a system built to help people navigate the world honestly should be trained on the world they actually walk through, not just the one we simulated for them.

Built With

Share this project:

Updates