Inspiration
Inspired by how email automates the prevention phishing and thinking how we could automate the prevention of scam calls.
What it does
Listens in on a call and calculates a risk factor for the likelihood whether it is a scam or not. Once it reaches a certain number, the ai steps in. It can either warn that it may be a scam call or prevent the call from happening.
How we built it
We built Scam Guardian in Python, as a real-time pipeline running on a laptop beside a phone on speakerphone. Audio from the laptop mic is streamed live to Deepgram’s Nova-3 speech-to-text, with diarization to separate the caller from the person being called. Each finished sentence goes into a rolling 60-second window that Google Gemini scores for scam risk and tactics (like fake bank calls, urgency, secrecy or gift cards) and returns as structured JSON. Plain code, not the AI, then decides when to speak up, so the agent stays calm instead of jumpy. When it does speak, our persona “Rosie”, a warm, protective companion, gives a short, kind warning through ElevenLabs text-to-speech (with an offline voice as a backup), and the mic is muted while she talks so she doesn’t hear herself. Everything shows live on a NiceGUI dashboard with a transcript, risk gauge, detected tactics and call timeline. When the risk passes 80%, a push notification goes to a trusted family member’s phone through ntfy. We split the work into audio, brain and interface, built on a shared data format, and connected it all with Python’s asyncio.
Challenges we ran into
One challenge is how we had to differentiate the voices. This was an issue due to the model we were using. This sort of problem occured a lot where we had to be careful on the exact models we used.
What we learned
As our first hackathon, we learnt how to use git collaboratively, implementing parts of a program. Especially, we learnt how to handle voices within python and how to use APIs.
What's next for Save Guardian
A real phone app, which does not need a laptop to run on speakerphone. Better speaker detection.
Built With
- asyncio
- deepgram
- elevenlabs
- github
- github-jobs
- google-gemini
- nfty
- nicegui
- numpy
- python
- python-dotenv
- pyttsx3
- requests
- sounddevice
- visual-studio-code
