Veracity checks a phone call for scam signs while it's happening: the caller's number, voice and words. If the caller asks for money, the user pays with Protected Send, PayPal authorizes the order with intent AUTHORIZE, and Veracity holds the money until the call checks out. Built on PayPal Orders v2, Payments v2 and the JS SDK v6 (sandbox only), with an AI-voice model, Whisper and explainable scam rules. Tested on real phone calls with sandbox payments, and judges can run it with one Docker command.

Code: github.com/YADUNANDAN-SINGH/Veracity · Demo video (2:27): youtu.be/7DbctZZKWP8 · Run it: docker compose --profile judge up --build judge, then open http://localhost:8000 (steps in docs/JUDGE_TESTING.md)

Inspiration

A grandmother picks up the phone. "Grandma, it's me. I'm in jail, please don't tell Mom. The lawyer says it's $2,400. Can you send it on PayPal right now?" By the time anyone thinks to check, the money is gone.

This happens at scale. In 2025 the FBI counted nearly $893 million lost to AI-related scams, with voice clones among the tools scammers use. Americans aged 60 and over reported $7.7 billion in internet-crime losses, including $352 million lost to AI-related scams (AARP on the FBI and FTC reports). In India, fake "digital arrest" calls from "cyber police" keep people on the line until they transfer their savings.

Warnings arrive too late, because the money moves during the call. Veracity is for anyone pressured to send money on a call, especially older adults, and the families who look out for them. We wanted the money to wait until the call checks out.

What it does

Veracity sits between the call and the payment. It checks the caller's number, voice and words during the call, and either the voice or the words can flag it.

One call, start to finish

  1. The phone rings. Veracity reads the caller's number: line type, country, original carrier and community reports.
  2. During the call, the dashboard shows a live transcript with red flags highlighted in the caller's exact words, an AI-voice meter and a script score. A strong match raises a red banner: "Don't share an OTP or send money."
  3. The caller asks for money. The user pays through Protected Send, with PayPal's own button. The order's intent is AUTHORIZE, so approving reserves the funds and nothing is sent.
  4. Veracity holds it. The card quotes why, with timestamps ("it's me" at 00:03, "car crash" at 00:09, "jail" at 00:17, "send it on PayPal" at 00:35) and asks the user to call Jake back on the number saved in their phone.
  5. The user hangs up and calls back, then picks Jake confirmed: release or Jake didn't ask: cancel. Cancelling voids the authorization: "Cancelled. The money never left your account."
  6. Every call ends with an evidence report (JSON): the caller's exact words and timestamps, the scores, the payment with its PayPal ids, and where to report the scam (FTC, IC3 and the Elder Fraud Hotline in the US; 1930, cybercrime.gov.in and Chakshu in India).

A $3,000 Protected Send held as a likely scam on a real phone call, quoting the caller's words A real phone call on 6 November: a fake grandson asked for $3,000. PayPal authorized it, and Veracity held it as a likely scam, quoting the caller's words. The number is blurred.

How the payment gate decides

Every payment made during a call is held until the call ends, then decided on the final report:

  • Likely scam (a high-risk call, a Likely AI voice, or a script score of 80+): held. Released only after a confirmed callback, and even then after a 15-minute cooling-off. If the call turns high risk while it's still going on, the hold escalates.
  • Medium risk, or some scam signs ("it's me", a family emergency): held until the payer calls the person back.
  • Low risk: captured automatically when the call ends.
  • Nobody releases it within 72 hours: voided automatically, inside PayPal's 3-day honor period. A sweeper checks every held payment once a minute.
  • PayPal fails when the call ends: the report still appears, the money stays held, and the capture is retried automatically.

Confirmed can't be pressed while the call is live, because the scammer could be the one saying it. Cancelling works at any time.

How we built it

PayPal (sandbox only)

  • Orders v2: POST /v2/checkout/orders with intent AUTHORIZE, then /authorize once the payer approves.
  • Payments v2: /capture to release, /void to cancel, and a GET on the authorization to record PayPal's real state if a capture or void fails.
  • JavaScript SDK v6: the PayPalOneTimePaymentButton from @paypal/react-paypal-js, so the payer approves in PayPal's own pop-up.
  • Webhooks (optional): checked with verify-webhook-signature before anything changes; each event is applied once.
  • Payouts (optional): forwards a released payment to the recipient's PayPal email.
  • Every request that moves money carries a PayPal-Request-Id, so a retry can't do it twice. Every PayPal answer (method, path, HTTP status, debug_id) is stored as the payment's audit trail and shown on the card.
  • Without sandbox keys, the Docker build uses a built-in stand-in, clearly labelled as not PayPal.

PayPal's sandbox API Calls log: create order 200, authorize 201, void 200 PayPal's own sandbox log for that $3,000: create the order (200), authorize it (201), and void the authorization (200) after the callback.

AI: what each model does

  • Spectra-0 (an XLS-R 300M + ECAPA-TDNN anti-spoofing network, PyTorch): scores the last 4 seconds of the caller's voice every second. It needs three windows in a row at 75% or more before it says Likely AI, and the call then stays flagged.
  • Whisper (faster-whisper): transcribes the caller at each pause, in Hindi or English. small in the Docker build, large-v3-turbo in the phone build.
  • Llama 3.1 8B (local, through Ollama; optional, phone build only): reads the transcript for scam tactics the rules miss and answers in a fixed JSON schema. A flag is kept only if its quote is really in the transcript.
  • Scam rules (plain code): 12 categories in English, Hinglish and Hindi. Every flag carries the exact words and time, which is what makes every hold explainable.
  • Risk fusion (plain code): number 25%, voice 40%, script 35%, and one strong signal is enough on its own.

The models run on the laptop. The only outside service is PayPal, and PayPal gets the payment, never the call: no audio, transcript or scores.

The live call view on a real call: script score 92, voice Likely human, the transcript with red flags highlighted What Veracity heard on that call. The voice stayed Likely human because a real person read the script; the words gave it away.

The rest of the stack

  • Backend: Python, FastAPI and SQLite. Web code lives in routes/ and logic in feature/, so every feature is tested without a server: 164 automated tests pass with no phone, models or PayPal account (PayPal is replaced by an in-memory stand-in).
  • Dashboard: React 19 and Vite, updated live over a WebSocket, in four steps: check setup, check number, live call, report.
  • Phone build: a real call captured on a Mac from an Android phone over USB. scrcpy (3.2+) captures the caller's side of the call through adb, and BlackHole routes it into Python.
  • Docker judge build: no phone needed, with Spectra-0 and Whisper small baked in. It offers scripted sample calls, two recorded sample calls with pre-computed model output, and live analysis of uploaded recordings (up to 2 minutes), which go through the same pipeline as a live call. There's no hosted demo, because call audio, transcripts and scoring stay on the user's device.
  • Tools: Claude for writing code, and Antigravity for UI polish and debugging. ElevenLabs generated the AI test voice, and Google Veo generated the demo narration.

Challenges we ran into

  • Android doesn't let apps record calls. scrcpy's voice-call-downlink source over adb gets the caller's audio without rooting the phone.
  • Whisper on Hindi phone audio detected Romanian and German, and looped on repeated words. We force Hindi, use temperature 0, block repeated 3-word phrases, and "fold" the Devanagari letters Whisper confuses (दिजिटल आरेष्ट for डिजिटल अरेस्ट) before matching.
  • On 7 November, the first voice model rated our own Veo demo narration Likely human. It missed most current generators, and our pipeline was resampling the call in 0.1-second blocks, which added clicks to the audio. We fixed the resampling and switched to Spectra-0.
  • Neither signal is enough alone. A real person reading a scam script passes a voice check, and a cloned voice saying something ordinary passes scam-word rules. That's why Veracity uses both.
  • A scam read by a real person from an ordinary number fuses to low risk. So money gets a stricter rule than the alert: any scam signs are enough to hold for a callback.
  • PayPal's webhook check fails if the JSON is re-serialised, so the raw body goes into the verify request exactly as received.
  • A third-party app can't pause a payment made inside the PayPal app. That's why the money goes through Protected Send.

Accomplishments that we're proud of

  • AI voice vs. real voice, on recordings. An ElevenLabs recording hit Likely AI at 0:05 (99% peak), script 100/100, high risk. A real-voice recording stayed Likely human (13% peak), script 8/100, low risk.
  • Two real phone calls with sandbox payments (6 November, scored by the earlier voice model). The same tester called from the same phone twice; only the words changed. The fake grandson (a car crash, a police fine, "do not tell anyone") scored script 92/100 and risk 92/100: PayPal authorized the $3,000 (201), Veracity held it, and after the callback the authorization was voided (200). The real grandson asking for $50 for Mom's birthday cake scored script 8/100 and risk 15/100, and the money was captured (201) when the call ended.
  • A real call in Hindi (31 October): a person read a "digital arrest" script. The voice stayed Likely human across 76 scored windows, and the rules caught the script word for word: 4 red flags, script score 82/100, alert fired.
  • AI voices, measured through simulated phone calls and as recorded: Spectra-0 flagged 85% of AI calls and wrongly flagged 1.1% of real people's calls.
  • On 48 written test calls, 21 of 24 scams would hold a payment (6 as a likely scam, 15 for a callback), and 0 of 24 ordinary calls would, including a real bank's fraud check and a friend asking for $20.
  • 164 automated tests pass.
  • The project started at CT BuildFest 2.0 (30–31 October 2026), where Veracity won 2nd place as a call detector. Everything PayPal, including Protected Send and the payment gate, plus the US scam scripts and the Docker judge build, was built for this hackathon.

What we learned

  • Holding money beats warning people. A warning can be ignored in a panic; a held authorization can't be spent.
  • Every decision has to be explainable. Each hold quotes the caller's own words, so the user can see why, and so can anyone they report it to.
  • Test with the real thing. Our voice model only failed when we played our own AI narration into a real call.

Limitations, plainly

  • Sandbox only, with test money. Without sandbox keys, the Docker build's payments come from a labelled stand-in, not PayPal.
  • Money must go through Protected Send, because Veracity can't pause a payment made in the PayPal app.
  • A cloned voice played from a speaker across a room is mostly missed (19% caught). The voice numbers come from simulated phone calls and recordings, not real phone lines.
  • Scores are estimates, not proof, and the evidence so far is small.
  • The two real payment calls were scored by the earlier voice model; both callers were real people.
  • The 48 test calls were written for the test, with AI help, in phrasing close to the rules, so real-world recall will be lower. A calm caller who avoids scam phrases can still get through.
  • No hosted demo. In Docker, scripted sample calls have no audio, so they show "Voice not scored on this call"; the two recorded sample calls show pre-computed model output. Uploads run the real voice model and Whisper small on the CPU, without the LLM, and take about 2 minutes per minute of audio.

What's next for Veracity

  • Test on real phone-line recordings of a consented voice clone, and add speaker-playback detection.
  • Offer the call verdict as a risk signal that PayPal's own send flow could use, instead of a separate checkout.
  • Run without the laptop and cable.
  • Hold any money request when the caller isn't calling from the number saved for that person, the classic "new number" trick, and test the rules on calm scams with no trigger words.

Built With

Share this project:

Updates

Submission history