Inspiration

Every phone scam has a moment, and it always sounds the same:

"Grandpa, I'm in trouble. Buy $500 in gift cards right now. Don't tell anyone, not even Mom."

The FBI's 2024 Elder Fraud Report puts losses by Americans over 60 at almost $5 billion in a single year, and many cases go unreported. These scams aren't technically clever. They're psychologically clever. Nearly every script uses the same three moves: create urgency, block verification, and isolate the victim. The scammer's whole strategy is to make sure no one else is in the room.

Today's defences mostly arrive at the wrong time. Spam filters act before the call, and bank alerts act after the money is gone. Few tools protect the middle: the 90 seconds when a frightened person is being talked into something.

So we asked one question: what if someone you trust could be in the room when it matters?

That became Tripwire: a pause when it matters. The wearer switches it on. It notices pressure as it happens, puts a clearly identified AI between the scammer and the wearer, and brings in family with the exact words that raised concern, all before any money moves.

What it does

  1. The wearer turns protection on or it auto triggers on unknown numbers. Nothing runs outside an enabled session, and the Vexo band's solid blue light shows whenever its microphone is on.
  2. A live call happens in our browser calling app between a caller, the wearer, and the wearer's daughter.
  3. Speech is transcribed live and checked for scam patterns: urgency, secrecy, gift-card requests, and isolation.
  4. On a high-risk result, the wearer stops hearing the caller, while the caller and an AI assistant that always says it is an AI can still talk.
  5. The daughter gets an alert showing the verbatim statements that triggered it, and she can join with one tap. When she joins, the AI stops speaking.
  6. The wearer reconnects only when family chooses.

Every status on screen comes from a real event. "Protected" appears only after takeover is confirmed, and failures show a clear alert with an End call button.

How we built it

We're five people with five clearly owned parts, built around one shared event contract that we wrote before any feature code.

The pipeline

$$\text{Vexo band / mic} \rightarrow \text{Deepgram} \rightarrow \text{Rules} + \text{Gemini} \rightarrow \text{State machine} \rightarrow \text{Daily routing} \rightarrow \text{ElevenLabs} \rightarrow \text{Family}$$

  • 🎙️ Vexo band: a bridge streams the band's microphone into our pipeline. The band's hardware light is on whenever the mic is on.
  • 📝 Deepgram Nova-3: streaming speech-to-text through our own server-side relay, so API keys never reach a client. Endpointing lets us act the moment a sentence finishes, and smart formatting turns "five hundred dollars" into "$500" for our evidence checks.
  • 🧠 Google Gemini: fixed rules catch the obvious scam script instantly, and Gemini scores each sentence with a risk score between 0 and 1. It also writes the assistant's single, tightly limited live reply. "Buy gift cards now, don't tell anyone" scores very high, "call me back tomorrow" scores near zero, and "the repair is $2,000" on its own scores low: worth a second look, never a takeover.
  • 🗣️ ElevenLabs: a calm, clear voice that speaks within about a second of takeover: "I am Tripwire, an AI assistant. This conversation has been paused for family verification." It never pretends to be the wearer.
  • 📞 Daily: takeover is not a mute button. It's per-track routing inside one live WebRTC room: the wearer stops hearing the caller while the AI and family still can. Each role gets its own short-lived meeting token.
  • ⚙️ Backend (FastAPI): a strict, event-driven state machine:

$$\texttt{monitoring} \rightarrow \texttt{protecting} \rightarrow \texttt{assistant_active} \rightarrow \texttt{family_joined} \rightarrow \texttt{ended}$$

Every event is an ordered, deduplicated envelope pushed over WebSockets. Illegal transitions are rejected, and a duplicate high-risk decision never starts a second takeover.

  • 🖥️ Frontend (Next.js, React, TypeScript): four live screens make up one product: a wearer page with huge type and calm language, a family page with verbatim evidence and a one-tap join, a caller page that discloses the AI, and a presenter page showing the live transcript, risk, state machine, and audio routing.

When does Tripwire step in? A takeover starts only when the evidence is strong, meaning a high-risk result with several supporting signals. It counts as active only when both of these are true, never because a button was pressed:

$$\text{assistant active} \iff \text{wearer client confirms caller audio is blocked} \;\wedge\; \text{assistant is ready}$$

If either condition fails within $15\,\text{s}$, the UI says "Takeover failed" and tells the wearer to hang up. We never show a fake success.

Challenges we ran into

  1. Muting is not protection. We needed the wearer unable to hear the caller while the AI and family still heard everything, in one live room. Per-track subscription control, plus the rule that takeover is real only once the wearer's client confirms it, was the hardest engineering in the project.

  2. Making the evidence trustworthy. A model that paraphrases can frighten a family over words nobody said. So every quote $q$ shown to family must appear word for word in its source utterance $u$:

$$\forall\, q \in \text{evidence}: \quad q \subseteq u_{\text{source}}$$

If any quote fails this check, the whole decision is rejected.

  1. Racing the scammer. A warning that arrives after the gift card code is read out is useless. Our latency budget is roughly:

$$t_{\text{total}} \approx \underbrace{0.3\text{–}1\,\text{s}}{\text{speech endpointing}} + \underbrace{\approx 0}{\text{rules}} + \underbrace{\sim 0.4\,\text{s}}{\text{Gemini}} + \underbrace{\sim 1\,\text{s}}{\text{voice}}$$

That puts takeover at around two seconds after the scam sentence ends.

  1. False positives. "$2,000 for a repair" is not a scam. A single signal only triggers a review; a takeover needs several independent pressure signals at once. We reasoned about the trade-off like this:

$$\text{cost(false alarm)} = \text{a 30-second pause} \;\ll\; \text{cost(miss)} = \text{someone's life savings}$$

  1. Hardware reality. The band has no vibration, screen, speaker, or heart-rate sensor, and its LED is locked solid blue while the mic is on. We redesigned around voice and the app's own warning indicators instead of faking features it doesn't have.

  2. Five builders, one day. We locked the event contract first: envelopes, sequence numbers, per-role tokens, and error codes. That let five parallel builds come together in hours, not days.

Accomplishments that we're proud of

  • A full flow from suspicious call to detection, AI takeover, family alert, and daughter joining.
  • Evidence you can trust: family sees the exact words that triggered the alert, checked against the transcript, not a vague warning.
  • An AI that is open about being an AI and steps back the moment family takes over.
  • A UI that never fakes success: replays are labelled "Demo replay," and failures are shown as failures.
  • Five people shipping five modules in parallel against one shared contract.

What we learned

  • The best safety feature is a person. AI shouldn't replace family. Its job is to buy them the 30 seconds they need to step in.
  • Trust comes from showing the words. A risk score alone doesn't convince anyone; the scammer's exact words do.
  • Honesty is a feature. The AI always says it's an AI, and the app never claims protection it hasn't confirmed.
  • Latency is a product decision. It shaped our models, architecture, and thresholds.
  • Elder-tech must ask little of the user. Anything that needs opening, tapping, or understanding under stress protects nobody.
  • Contracts before code. A shared schema let five people build in parallel without stepping on each other.

What's next for Tripwire

  • Evaluating the detector on real scam-call data. Our risk threshold is a demo policy, not a validated fraud classifier.
  • Caregiver controls and clear consent settings.
  • Supporting more call types where platforms allow. Today Tripwire works for calls through our own app, not normal cellular calls.
  • A proper privacy and security review. Audio is processed by third-party services (Deepgram, Google Gemini, and ElevenLabs), and the wearer must opt in each session.

Built With

Share this project:

Updates

Submission history