EchoSense

Inspiration

My great-grandmother slipped at home when nobody else was there. When the family came back and rushed her to the doctor, the doctor said: "If you had brought her 30 minutes earlier..."

Those 30 minutes are why we built EchoSense. One in four older adults falls every year. Pendants only help if they are worn and pressed, and most people do not want a camera watching them at home.

What it does

EchoSense turns ordinary phones into a camera-free room sensor, and lets AI agents decide what to do when something happens.

• Sensing: each phone plays inaudible sonar chirps (17 to 20 kHz) to see body motion, and runs Google's YAMNet sound model on the device to recognize thuds, glass, voices, TV and cries. Phones send only measurements, never raw audio.
• Three agents with opposite jobs: the Responder never lets a fall go unanswered, the Skeptic looks for a harmless explanation, and the Auditor trusts no phone and checks every report against physics and the other phones. A Cartographer agent places sounds and people on a room map.
• Disagreement is handled, not voted away: when the agents agree, the decision is made locally. When they disagree, each becomes its own Claude agent and they negotiate over a ladder of actions (do nothing, watch, ask, count down, call), with offers, challenges and concessions. If there is still no deal, the person is asked.
• The person always has the last word: the phones ask "Are you OK?" out loud. "I'm okay" closes it. "No, I'm hurt" or silence alerts the caregiver on Webex, with each agent's reasoning and buttons for "On my way" and "They are OK".

How we built it

• Cisco AGNTCY: every agent and every phone has its own identity in one SLIM group channel, end-to-end encrypted with MLS and run by a moderator. Every message is an A2A message with the incident as its context, and every agent publishes an A2A agent card. We use three SLIM patterns: group chat for the debate, a broadcast "look again" to every phone, and requests addressed to one phone. A compromised phone is removed from the MLS group. Every decision is exported as OpenTelemetry traces.
• Phones: a web app using the Web Audio API for sonar (matched-filter chirps, a separate frequency band per phone), YAMNet in a worker, and the motion sensors.
• Hub: Node.js and TypeScript. The agents, the negotiation protocol, Claude (Haiku 4.5) for negotiation only, a hard daily spend cap, and Moonshine speech-to-text running locally on the laptop, so spoken answers never go to the cloud.
• Testing: 131 automated tests, a simulator, and a red team.

Results

Simulator: 14 scenarios x 50 randomized trials, 3 phones.

• Red team: we attacked our own system with 340 adversarial incidents across 26 scenario families (TV crashes, hacked phones, hallway falls, loud phone calls), plus test cases written before we saw any results. No false call when the person can answer.
• Real room, our test setup: 30 / 30 falls acted on, and 27 / 30 everyday events left alone.

Challenges we ran into

• The phone heard itself. Its own prompt ("say I'm OK, or say help") leaked into the recording and turned "I am okay" into a call for help. We now strip the prompt from what was heard, and a bare "no" makes it ask again instead of calling.
• Loud talking looked like screaming. A phone call next to a phone triggered alarms, so a scream now needs the sound model to agree it is distress, and one phone's opinion is never proof on its own.
• Cost. Early on, every small disagreement called Claude. Now Claude is only used on a genuine split, under a hard daily cap, and the rules decide everything else for free.
• Physics. iPhone speakers are weak near 21 kHz, so the phone on the highest band was half-blind. We moved all bands below about 20 kHz.
• Trust versus observability. A phone that heard nothing is not lying, so we separated "this phone could not observe it" from "this phone cannot be trusted".

What we learned

Disagreement between agents is information, not noise. Keeping it on the record and asking the person turned out safer than forcing a vote, and giving each agent a clear, opposite goal made their reasoning far easier to audit.

What's next

• Our own small sonar sensors that join the same agent channel, so no old phones are needed.
• A pilot with a senior living community.
• Turning EchoSense into a startup with a subscription for families.

Built With

• a2a
• agntcy
• anthropic
• claude
• css3
• esbuild
• html5
• javascript
• mls
• moonshine
• node.js
• opentelemetry
• sherpa-onnx
• slim
• typescript
• web-audio-api
• webex
• websockets
• yamnet