Here | Devpost

Hear & There
Here dashboard
Haptic belt internals

Inspiration

Caption apps turn speech into text. They don't tell you who said it, or where that person is standing. Sound alerts say "doorbell" and leave you to figure out which way to turn.

Deaf and hard-of-hearing people miss more than words. They miss where a sound came from, who in a group is talking, how it was said, and that anything happened at all while they were looking the other way. A car behind you is a direction problem before it's a vocabulary problem.

What it does

HERE fuses a microphone array, a camera, and a haptic belt into one spatial picture of the room.

🗣️ Captions on the right person. Each line floats above whoever said it (Person 1, 2, 3) with a tone chip: whispering, laughing, loud.
🔔 Sounds with a direction. Doorbell, your name, a car, footsteps, an alarm. A card names the sound and a ring points to it.
🧭 A belt that turns you. Someone speaks behind you, the belt buzzes on that side of your waist, you turn, and their caption slides into view. Works with the phone locked in your pocket.
🧠 Attention, not noise. An AI layer decides whether a sound is ambient, notice, important, or urgent. A fan doesn't buzz you. A car behind you always does.
🗺️ Memory. HERE Now shows the room live. Hear & Now replays your days on a 3D map of campus, answers questions like "when did the doorbell ring?" with evidence, and surfaces patterns across days.
🔒 Private by design. No audio or video ever leaves a device. Only text, labels, and bearings travel.

How we built it

The rig

Part	Role
ReSpeaker XVF3800 mic array	The ears: direction of arrival + two focused beams
Laptop with RTX GPU	The hearing module: speech, sound, and attention models
iPhone on a chest mount	Stands in for AR glasses: its camera is the glasses' camera, its screen is the display
ESP32 haptic belt	The output: tells your body where to turn
Relay on Vultr	Connects everything, on any network

The belt (built from parts)

We soldered and wired the belt ourselves and tested after every stage with a multimeter.

Controller: 30-pin ESP32 in a screw-terminal breakout. It only switches; it never powers a motor.
Driver: ULN2803 Darlington array, with COM tied to +5 V so its built-in flyback diodes absorb the motors' kick.
Motors: 3 V coin motors, each bonded to a 2.5 cm plate cut from a plastic card. A bare coin motor shakes a spot you can miss through a shirt; the plate spreads it over skin you can't miss.
Power: an Anker power bank through a cut USB cable into Wago lever connectors as +5 V and ground hubs, with a 1000 µF cap so the ESP32 doesn't brown out when motors spin up. Every motor gets its own home-run wire pair with strain relief.
Motor safety: a firmware duty cap on the PWM so 5 V never cooks a 3 V motor.
Bluetooth LE: one write-without-response characteristic. A 3-byte command fires one motor; a 4-byte bitmask command fires several at once; 0xFF stops everything.
Fail-safe: the firmware owns the off-timers, so Bluetooth jitter can't smear a pattern, and a disconnect stops every motor.
Mounting: motors on Velcro inside a tactical belt, electronics in a notched project box at the back, wires taped along the top edge and never across the buckle.

The phone decides where to buzz, not the belt. The array reports where the sound is in the world, the phone knows which way you're facing, and the belt gets the direction relative to your body. Turn around and the buzz stays pinned to the sound.

The mic array is embedded work too. We mask the XVF3800's two beams to separate two people talking at once, and drive its LED ring so it glows toward the sound. Even with the wearer standing off to the side of the array, bearing error stays inside one motor's sector at normal conversation distances.

The AI stack

HERE is a stack of models, each doing the one job it's good at, on a strict latency budget.

🎯 Attention with JEV, the part we're proudest of. Detecting a sound is easy. Knowing whether to interrupt a person is hard. We turn each event into one short state sentence:

Sound: doorbell (confidence 0.71). The wearer was in a conversation. No repeats in the last minute. Focus mode off.

Then we ask JEV-9B, an open decision model from AutoTrust, a typed question with fixed options: ambient, notice, important, or urgent, plus "was this said to the wearer?" It answers with calibrated probabilities, which decide between a small cue and a full card plus a belt buzz. Quantized to 4-bit so it fits beside the speech model on one laptop GPU: 208 ms median per decision.

🛡️ A safety rule around the model. The model can raise an alert. It can never lower one. Under 0.5 confidence or over its time budget, the next tier decides: a DeBERTa zero-shot classifier, then deterministic rules. A vehicle behind you is urgent no matter what any model says, and every event records which tier made the call.

Sound classification: audio-event models label what the array hears, and the phone's on-device sound analysis acts as a second sensor that has to agree.
Speech: streaming transcription with words appearing as they're spoken, a tone estimate per line, and filters that stop the model from inventing text on noise.
Fusion: the phone matches each line's bearing to a tracked face in the camera view, so the caption lands on the right person.
Context: Gemini writes a one-line context for a caption and phrases Ask HERE answers from the evidence rows.

Sponsor technology

🎙️ ElevenLabs

Scribe v2 Realtime powers streaming speech-to-text, the core of what a hard-of-hearing user reads. It sits behind a single transcriber interface with automatic mid-sentence failover to a local model, so a network drop never blanks the captions. On the live API we measured a 0.04 word error rate with first words in about 1.2 s.

☁️ Vultr

Our relay is a WebSocket service on Vultr behind Caddy for TLS. The laptop publishes events and Vultr fans them out to the iPhone in real time, on any network, no shared Wi-Fi required. It carries only text, labels, and bearings. We also built a one-command deploy script, a live status page with message rates, and session export to Vultr Object Storage using a dependency-free SigV4 client.

🤗 Hugging Face

HERE runs on open models from the Hub, and a lot of them:

Attention: JEV-9B (4-bit via transformers, peft, bitsandbytes) and MoritzLaurer/deberta-v3-base-zeroshot-v2.0
Speech: Whisper large-v3-turbo for local fallback
Sound: a real bakeoff on 254 clips of mispeech/ced-mini, AST, and laion/larger_clap_general (zero-shot with our own per-sound prompts) against PANNs CNN14

Every backend is one config line apart, and we report results as measured: CED-mini was 2.3× faster at a third of the memory, while PANNs caught more doorbells under crowd noise. Every event names the model that produced it.

🧱 Databricks

The live system answers "what's happening now?" Databricks answers "what keeps happening?" A serverless job reads the exported event log from a Unity Catalog volume, aggregates it in Spark into Delta tables, and writes ranked insights back:

where your name was called from behind and you missed it
the crosswalk where danger comes from the rear
who you talk to most
your quietest window and the loudest place-hour

These become the pattern cards in Hear & Now on a 3D map of UCF, each stamped with the run that produced it. ~4,500 events → 13 insights in 83 seconds. A plain-Python fallback means the dashboard never blocks on the cloud.

🐘 Neon

Memory lives in Neon Postgres: sessions, every event, final utterances with full-text search, and the Databricks insights. Ask HERE is a SQL question first (full-text search plus time and bearing filters), then a phrased answer linked to the evidence. Campus Wi-Fi blocks port 5432, so we wrote a small shim that talks to Neon's SQL-over-HTTPS API on port 443.

Challenges we ran into

Two motors didn't survive. We designed for eight motors and coin motor leads are hair-thin. We ended up with six working channels, remapped the firmware, added the bitmask command so two neighbors can fire together to point between them, and re-tuned the direction mapping.
Three models, one GPU. JEV-9B is 18 GB at full precision; the laptop has 16 GB shared with speech and sound. 4-bit got it to ~6 GB. Cold, it missed our latency gate (496 ms). Warm, it passed (208 ms).
Speech models hallucinate on noise. In a loud hall the transcriber wrote sentences nobody said. Now we only trust a segment if the array heard a talker long enough, and drop segments the model itself doubts.
Campus Wi-Fi. It blocks Postgres and isolates devices from each other, which is exactly why the relay and HTTPS database driver exist.
Four ideas of "forward." The array, the compass, the camera's field of view, and the belt all disagreed. Nothing worked until everything became a bearing in one shared frame.
Overloading the user. The first version buzzed for everything. Most of the attention work is deciding what to leave out.

Accomplishments that we're proud of

The turn-around: someone speaks behind you, the belt buzzes, you turn, and their caption is waiting on them.
A hand-built belt a stranger can put on and understand in seconds.
An AI layer with a real safety rule: models can escalate, never suppress.
A full system that runs without any of its hardware, from fixtures and mocks, with one message contract shared across Python, Swift, and TypeScript.

What we learned

Direction is the missing half of accessibility for sound. A label without a bearing leaves the user searching.
Haptics should carry direction and nothing else. Short patterns, never continuous buzzing.
Commit fast, then refine. Show something immediately and let slower models fill in.
Measure before you claim. Our benchmarks changed our minds more than once.
Soldering: put the heat shrink on before you make the joint.

What's next for HERE

Real AR glasses in place of the chest-mounted phone, with the mic array on the wearer
All eight motors, then a slimmer belt on a custom PCB
Testing with deaf and hard-of-hearing users, and tuning attention to each person
A phone-only mode for walking outside, where a car behind you matters most