Inspiration
To most people, fraud seems like a distant thing that could never happen to them or someone they know. I felt that way too, until a few years ago when my Mom fell for a password reset phishing scheme, and lost her Facebook account. This wouldn't be a bug deal for most people, but my Mom used Facebook as a backup for all of her photos, so most of the pictures of my family and my childhood over the years no longer exist. I wanted to make a program that's easy to use, that could help prevent this from happening to others in the future.
What it does
SpearPhishing pierces through phishing schemes using 3 modes. It can analyze user entered text, text in images, and live audio calls. It returns the chances of the media being a scam, along with reasons why and recommended next actions for the user.
How I built it
I built it in python, using pytesseract to read text from images, and elevenlabs to transcribe audio. All text is then sent to Google's gemini API to analyze potential risks.
Challenges I ran into
I struggled greatly to incorporate the live transcription system; it was hard to incorporate a live feed with the slow responses from gemini without long periods of code blocking. I also struggled with the frontend, and ended up breaking the audio system in the process.
Accomplishments that we're proud of
I'm proud of how much I was able to accomplish in such a short period of time, especially since I was working on my own!
What I learned
I learned how to incorporate generative AI into my own projects, which I had never previously done before, and seems like a good skill to have!
What's next for SpearPhishing
I would like to incorporate audio file support, so that voicemails and call recordings can be analyzed as well.
Log in or sign up for Devpost to join the conversation.