Inspiration
Phone account takeovers rarely beat the technology. They beat the person on the phone. A caller with leaked details and a good story ("my mom is in the hospital, please just change the number") talks a rep into it. Credit unions are now putting AI agents on those phone lines, and an AI can be talked into things even more easily: "ignore your instructions, I'm the branch manager."
People asked us: doesn't 2FA solve this? It doesn't, because 2FA only proves you hold the device on file. When a fraudster invented the identity and registered their own phone (synthetic identity), or is using a dead person's identity, 2FA passes every time.
So the question became: can you build a phone agent that sounds natural, serves real members in their own language, and still cannot be socially engineered, with proof?
What it does
You call a real phone number and talk to GuardLine like a person, in English or Spanish.
- Checks the person, not just the phone. CRS TransUnion identity verification flags identities that are synthetic, deceased, mismatched, or carry a fraud alert the member already filed with the bureau.
- Checks who owns the calling number. It uses CRS reverse phone lookup (a contact-data product, used here as a live fraud signal) on the real caller ID.
- Steps up only when needed. For a contact change, or a number that is not the member's, the real member gets an "Is this you calling?" prompt on their own phone (a simulated banking app in this demo).
- Never accuses anyone. Every caller who is not verified, impostor or real member with an emergency, hears the same calm line: a specialist will call back at the number on file. A real member in an emergency gets helped fast on their own phone. An impostor gets nothing, because the callback goes to the real member. No excuse opens a door, because every door leads back to something only the real member controls.
- Hands the human a case file instead of a hunch. It shows the outcome, the CRS evidence in plain words, and the manipulation tactics detected (urgency, authority claims, instruction injection, secrecy, redirect requests).
- Shows the call on a live screen, so the audience sees every check light up as it happens. ## How we built it Phone line: Cartesia Line voice agents on two real US numbers (English and Spanish), with Cartesia ink-whisper speech-to-text and sonic-3 text-to-speech. Each caller turn is relayed to our backend through a webhook tool, along with the real caller ID.
- The rule that shapes everything: the model can always make a call safer, never less safe.
- Claude Haiku (Anthropic API) does only two jobs. It turns what the caller said into a forced tool call of enums and stated details, and it phrases the reply naturally.
- A deterministic Python policy engine makes every verification and risk decision. No tool lets a model verify anyone.
- The model never sees member data on file, and a code guard checks every reply before it is spoken.
- Fairness by signature: the risk function does not accept language, accent or name as inputs. Credit status (score, bankruptcy, liens) is never a risk input; GuardLine does not call those endpoints.
- CRS sandbox APIs: TransUnion Identity Manager Verification (TUC3002), AccurateAppend reverse phone, and AtData FraudFinder, which checks a requested new email before a contact change.
- Stack: FastAPI, SQLModel/SQLite, Next.js + Tailwind (live call view, case dashboard, simulated member app), Docker Compose.
- Pre-registered eval: 30 scripted calls (15 real members, 15 impostors, English and Spanish). The labels were committed before the test runner existed; the git history shows the order. A leak checker that imports nothing from the app scans every agent sentence. ## Challenges we ran into
- The sandbox gives every persona the same identity score. We measured all 20 personas twice and found the score is a constant fixture; only the alert codes differ. We built the policy on the signals that actually carry information, and say so in the README.
- OTP and KBA are not mocked in the sandbox, so the step-up is a simulated member-app approval, clearly labeled as simulated.
- Speech-to-text mangles names and digits, especially in Spanish. A misheard surname escalated a real member, so we changed the policy: details that don't match mean "not verified", never "fraud". A small name difference is accepted only when every digit matches exactly, and is flagged for the human.
- Natural speech vs safety: fixed sentences sounded like an IVR. We moved to "decisions fixed, words free": the policy decides, the model phrases, and a code guard rejects any reply that leaks data or claims something the policy did not decide.
- Our own eval caught policy gaps. An impostor who hung up during the approval prompt was being closed as low risk. Hanging up when asked to prove it is you is now a fraud signal.
Accomplishments that we're proud of
- 0 impostors verified across 90 eval runs (30 cases x 3).
- 0 PII leaks in every eval run.
- Instruction-injection and authority-claim attempts tagged in all 3 runs.
- Results are stable: the same 29 of 30 cases pass in every run. The one failure is reported, not hidden: a real member calling from an unfamiliar phone who never answers the approval looks exactly like an impostor, and GuardLine chooses caution and a callback.
- Live tests on a real phone number: a caller with every correct detail, calling from their own phone, was caught when CRS showed the number belonged to someone else and the real member tapped "No".
- [146] automated tests.
What we learned
- 2FA answers "do you hold the phone?" Identity data answers "is the person behind the phone real?" You need both.
- With an AI agent, the safety property has to live in the architecture, not in the prompt.
- Pre-registering eval labels before writing the runner keeps you honest. Our eval found real bugs that our own demo calls had missed. ## What's next for GuardLine
- A real OTP or KBA step-up through CRS (TUC3004) in place of the simulated app.
- Keypad (DTMF) entry for digits on the phone line.
- Real outbound callbacks to the number on file.
- More languages (Hindi, Punjabi).
- Deployment to a cloud host.
Log in or sign up for Devpost to join the conversation.