Hearally

Your words. Your allies. The people on your side, kept close.

Comment

Inspiration

Many of us, at some point, have someone we look out for: a parent living on their own, a friend who needs a little help now and then. "Everything okay?" "Fine." isn't enough, and a call pushing for a card number is easy to keep to yourself. I wanted something between hovering and not knowing, that the person themselves owns and controls, and that tells the people on their side about the things worth knowing, and nothing else.

What it does

The account holder wears a small recorder (a Plaud today; Bee from Amazon is coming). Hearally works from their own words: often only their side of a call is recorded, and that is enough to tell what the call was about.

  • They choose their allies and what each one hears. Allies need no account and no app. Each confirms on their own page, after reading what Hearally is and is not, before anything reaches them.
  • Three kinds of message, by email: a heads-up at once (a call pressing for money or details, at most one a day), worth a call today (a moved appointment, a request for help), and a daily list (shopping, tasks, good news). Allies can also opt in to an automated phone call for heads-ups, after confirming their number with a texted code (built, still being tested, and not shown in the demo).
  • A heads-up names the scam it resembles and links the publisher's page, for example "This call sounds like the gift card payment scam described by the Federal Trade Commission", so an ally can check it rather than trust it.
  • Private moments stay private. Venting, family arguments, bodily remarks and any topic the holder names are held back from every ally and listed for the holder alone, who can send one anyway. The two exceptions are always passed on: a caller pressing for money or details, and a request for help.
  • It learns from "this was nothing". When an ally taps it, the next similar thing waits for the daily list, unless something new comes up, as when a second call adds gift cards. The holder sees everything it has learned and can undo it.
  • Numbers are silenced in the audio itself. The holder can listen to the copy Hearally keeps, with card and account numbers removed; the original is deleted as soon as that copy exists. Raw transcripts are never kept.

It reports what was said. It is not an emergency service.

Tavily + RAG: how Hearally knows the common scams

A heads-up is most useful when it says what the call resembled, with a source the ally can check for themselves. Hearally keeps a library of known phone and doorstep scams, keeps it current with Tavily, and retrieves from it for every conversation.

How Hearally knows the common scams: Tavily keeps the library current, Qwen3 embeddings find the closest scams, Nemotron names one only when three checks agree

  • Tavily keeps the library current, every week. A scheduled job makes live Tavily searches with fixed queries, restricted to trusted publishers: the FTC, AARP, the IRS, the SSA, US Courts, the FBI's IC3, Medicare.gov and others. Nothing from anyone's conversations is ever sent to Tavily.
  • Nemotron reads what Tavily finds; code decides what goes in. For each new page, Nemotron 3 Super writes a scam card in its own words, as strict JSON. Code accepts it only if it describes one scam that reaches people by phone or at the door, takes the source link and publisher from the Tavily result rather than from the model, and refuses it if its embedding is too close to a card already in the library. So far Tavily has led to 81 pages read and 5 new scams added to the 22 I wrote by hand.
  • Retrieval-augmented generation (RAG) for every conversation. The transcript, with numbers masked, is embedded with Qwen3-Embedding-8B on Nebius Token Factory and matched against all 27 cards by cosine similarity, in memory. The closest cards are given to Nemotron as reference, never as evidence, and its output schema is rebuilt for every call, so it can only name a scam it was actually shown.
  • A scam is named only when three things agree: Nemotron judges the call as pressure for money or details with concrete signals, Nemotron matches it to a card, and the retrieval score clears a threshold. The ally's email then says, for example, "This call sounds like the gift card payment scam described by the Federal Trade Commission", with a link to the FTC's own page.
  • Retrieval explains; it never escalates. On 22 invented test calls, 12 of 12 scams were named correctly and 0 of 10 look-alike calls were flagged, and showing Nemotron the retrieved scams changed no grade at all.

How I built it

NVIDIA open models on Nebius do all of the AI work:

  • Transcription: NVIDIA Parakeet TDT 0.6B v3 on Nebius AI Cloud. Every 30 minutes, if recordings are waiting, the backend starts a Serverless AI job on one preemptible NVIDIA RTX PRO 6000. The container transcribes the batch with word timings and word confidence, silences card and account numbers in the audio at their word timestamps, uploads the results through signed links, and exits. It holds no cloud credentials: only one-object signed links, valid for a few hours. Token Factory has no speech model, so transcription runs as its own GPU job.
  • Judgement: NVIDIA Nemotron 3 Super (nvidia/nemotron-3-super-120b-a12b) on Nebius Token Factory, with Zero Data Retention enabled. It reads each conversation, with numbers masked first, and returns strict JSON: what was said, its category, and any concrete pressure signals. I chose Super over Nemotron 3 Nano after a direct comparison: Super was about 5× faster, used a quarter of the output tokens, and judged better.
  • Known scams: retrieval-augmented generation (RAG) with Qwen3-Embedding-8B on Nebius Token Factory, kept current every week by Tavily. How it works is in the Tavily + RAG section above.

Code, not the model, decides what reaches an ally and how fast: grades, the one-a-day budget, the dignity filter, topics, learning and named scams are ordinary, tested code. Everything else runs serverless on AWS (Lambda, API Gateway, CloudFront with WAF, S3, DynamoDB, SQS, Cognito, SES, EventBridge), all in Terraform.

Recordings arrive two ways: the holder signs in to Plaud once from the dashboard (through Plaud's hosted MCP server, a preview: Plaud has not yet published terms for services using it this way), or a small open-source collector on a computer sends exported recordings.

Challenges I ran into

  • The model describes; the code decides. Testing against live Nemotron found behaviours no prompt could enforce: one scam call split into four heads-ups, a reply to a scammer read as a request for help, "my daughter never listens to me" marked relevant (it would have gone to the daughter). Each became a tested rule in code.
  • A confidence score that wasn't a probability. Parakeet transcribed a scam call word for word, yet my "is the transcript sure?" gate held the heads-up back. I had asked for entropy-based word confidence (min over word pieces) on a 25-language model and treated it as a probability with a fixed bar. Correct, relaxed speech scored 0.71 against a bar of 0.80. Now the gate measures the whole conversation, every item records why it got its grade, and pressure for money or details with concrete signals is never held back by the score: the heads-up says the recording was unclear instead.
  • The wearer's words are enough, and safer to collect. Recording both sides of a call raises consent questions in many places. I rebuilt the judgement around the account holder's own side: people repeat what a caller asks ("You need my whole card number? Within the hour?"), and that carries the signals.
  • Retrieval always answers. Embeddings find a nearest scam even for the doctor's office. A scam is named only when Nemotron's judgement, Nemotron's match and the retrieval score all agree; judging every test call with and without retrieval changed 0 outcomes.
  • Serverless GPUs in practice. A job can take over half an hour to find a GPU and pull its image, so signed links had to outlive that wait; preemptible GPUs get reclaimed mid-batch, so the collector relaunches once; and a startup state my code did not list was once read as a failure. Only failed, error and cancelled states now count as failures.
  • A presigned upload bound nothing. Tested against S3: the checksum and size headers on a presigned PUT are unsigned. Uploads now use a presigned POST whose signed policy pins the exact size and SHA-256.

Accomplishments that I'm proud of

  • An end-to-end product that is live: sign-up, the agreement, Plaud sign-in, GPU transcription with audio redaction, judgement, allies, three kinds of email, learning, topics and the held-back list.
  • Retrieval that explains without escalating: on 22 invented test calls, 12 of 12 scams named correctly and 0 of 10 look-alike calls flagged (evaluation). A small set, and I say so.
  • Privacy that is built rather than promised: numbers silenced in the audio, transcripts deleted once read, allies who must confirm, and a filter no ally can turn off.

What I learned

Hosted models are best at describing; the decisions that affect people belong in code that can be tested. Model scores need calibrating before they gate anything. And the most trustworthy design is the one where the person being looked out for holds the account.

What's next

Bee from Amazon support; calibrating transcription confidence on clear and deliberately unclear recordings; text messages to allies once carrier approval allows; a phone app so the setup never needs a computer.

Try it

Live site: https://hearally.com · Getting started: https://hearally.com/setup/ · Video: https://youtu.be/OIYDd4QhXy8 · Repository: https://github.com/amathias/hearally (MIT)

Judges: a demo account with the same staged data as the video is described in the private testing instructions.

Built With

Share this project:

Updates

Submission history