-
-
After the call: what to do next, in two steps
-
Normal call: big live captions, with risky words highlighted
-
When the caller keeps pushing, one clear step: hang up
-
Be careful: two warning signs together turn the screen amber
-
Warning: a caller claiming to be family gets the code word card
-
Evidence page: every flag tied to the caller's own words
-
The family phone gets the alert with the caller's exact words
-
From our recorded live call: the Warning on a fake power company call
Inspiration
Phone scams on older people tend to follow the same script. Someone calls pretending to be their bank or their grandson, and pushes them to pay right away with gift cards or a wire transfer before they have time to think. Voices can be faked now, so we decided to listen to what the caller says, not how they sound.
What it does
Aegis runs on the grandparent's laptop while they talk on speakerphone. It shows the call as big captions and highlights the worrying words. The screen says Normal call, turns amber for Be careful, and turns red for Warning.
When a call looks like a scam, Aegis tells them one thing to do. If the caller says they're family, it says "Ask: what's our family code word?" If they say they're from a bank or the gpay or give out any numbers, and to call back. Ifthe caller keeps pushing, it says "Hang up."
At the same time, a family member gets an alert on their phone with what the caller said and a button to call. Later they can log in and look back at past calls.
We tested it with a staged call using made-up names.aller said someone in Miami had just tried to take$900 from the account. The screen turned amber. When he asked her to "initiate a reverse transfer yourself", it turned red. When he said "there's no time", it told her to hang up.
How we built it
Aryan built the server in Python with FastAPI. It sends the laptop's microphone to Deepgram, which turns the call into live captions, and passes everything to the screens.
Mihir built the part that spots scams. It checks eacthing scammers say and points to the exact words. AnAI model on Groq, with Gemini as a backup, also reads the last minute of the call to catch things the word checks miss. It can only make a warning stronger, never weaker, and anything ing that was actually said.
Aaryan built the grandparent's screen, the family papage in React and TypeScript.
Family accounts and call history are stored in Tigerut each call, never what anyone said.
Challenges we ran into
One laptop microphone hears both people as one voice, so Aegis can't tell who said what. We check every line and let the AI model work out which lines came from the caller.
That caused another problem: people talk about scamswarning her mom about gift card scams, or a friendtelling the story of a scam she got away from, sounded like a scam to Aegis. We spent a lot of time teaching it the difference.
We also ran on free API plans, so we limited how often Aegis asks the AI model, and switch to Gemini if Groq is slow.
Accomplishments that we're proud of
We wrote 16 practice calls, acted out by us: 8 scamsound a bit like scams, such as a real bank fraudalert and a surprise party plan. We kept them aside and didn't adjust Aegis based on them. Without the AI model, Aegis warned on 5 of the 8 scams and didn't wrongly warn on any of the
What we learned
Scammers change their stories, but they keep using the same few tricks: pretending to be someone, rushing, asking for secrecy, asking for odd payments. Looking for those works betfic stories. We also learned that telling someone"this is a scam" in the middle of a call isn't enough. They need to be told exactly what to do next.
What's next for Aegis
A phone app so it works without a laptop, support foy for families to add the grandparent's real bank and relatives so Aegis can tell them apart from someone pretending.
Built With
- deepgram
- fastapi
- gemini
- groq
- python
- react
- tailwindcss
- timescale
- typescript
- vite
- websockets