Inspiration
Friday night, while we were ideating, one of us was browsing the Hophacks Discord and saw that someone had posted that they were nearby and giving away free stuff due to an overseas relocation. Now, a free PS5 and TVs sound a little suspicious, but our team member thought it didn't hurt to ask and messaged the user. Slowly, the "donator's" story started to unravel as they kept changing it. First, they were nearby, and then they had had suddenly moved to Atlanta, GA. After further investigation, it was clear that this was a shipping scam, one that most of us hadn't seen before and seemed to have recently popped up. That was the spark that inspired our project. Scams nowadays don't sound like your traditional scams anymore. They are new ones every week, they're personal, and there's now AI voice cloning in the mix to imitate your loved ones. Existing tools serve to educate as a preventive measure or are reactive, typically a warning you find after the money is already gone. We wanted to catch it in the moment.
What it does
YPINR (Your Prince Is Not Real) is a two-part scam disruptor and a living database of how scams work.
- Phone (Outpost): On a suspicious call, our system scores the audio with Eleven Labs detecting AI-generated audio and flags the suspicious phrases with CLAP (audio embedding model), determining if a full conversation analysis on transcript for scam detection is necessary. Grok evaluates scam risks based on the transcript, and positive predictions fires SMS notifications and get logged to backend database of all observed scam patterns. We only store deidentified attributes of scams and discard all personal data. With a constant flow of new scams being invented, we ensure that the preventative measures will be placed as soon as possible, thanks to our AI Grok training and our ever-growing pool of user provided information that will ensure a safe internet browsing experience for those of all ages.
- Social Media (Sherpa): This is the same detection as YPINR for text-based scams across platforms like Discord and Instagram, with a privacy-first policy.
- Intel Site: Everything caught with YPINR and Sherpa feeds our continuously updating database of Scam patterns and scenarios to improve our systems and educate our audience about evolving scam tactics.
How we built it
- AI voice detection: ElevenLabs synthetic-speech classifier wrapped in a FastAPI service.
- Language Analysis: CLAP flags suspicious phrases in audio and Grok evaluates transcripts for broader, context-wise patterns such as urgency.
- Confidence Scorer: Logic inside our detector service that fuses ElevenLabs audio score and Grok's language flags into one confidence score and maps it to a warning tier.
- Intel + RAG: We collected scam data from multiple online sources, and collated themm into a 132-pattern data catalog with an in-memory local TF-IDF vector index. Semantic search ranks by meaning, and the top matches are fed into Grok's prompt so it recognizes known scams faster and more consistently.
- Frontend: React site with SOC console aesthetic. Has a survey, intel repository, and a live version of Outpost (audio scam detector)
- Grok api integrated in the backend to help determine the scam risk probability and scam type of a user inputted survey.
Challenges we ran into
- ElevenLabs' model for detecting AI-generated audio underperformed, which led us to develop alternate strategies such as phrase screening.
- Wiring everything together for the home page demo was a challenge. Getting browser audio mic with clarity to then go through ElevenLabs, CLAP, Grok, the confidence scorer, SMS warning system, and then surface on the home page without anything breaking.
- Designing a clean frontend that served a clean UX that had logical flow was also surprisingly challenging. Since our team mostly consisted of backend developers, putting ourselves into that frontend mindset took some time. We had a lot of text walls at first but realized we needed to break it up to keep engagement and create a natural map for the users to follow, as we intended to provide the most value for them.
- Building Sherpa came with its own set of challenges, and we found it particularly difficult to devise (and then write) a way to quantify the "suspicious-ness" of sentences while keeping everything completely private and local.
Accomplishments that we're proud of
- Multi-step pipeline for scam detection that streamlines several deep learning models.
- A 132-pattern intel catalog with semantic search that converts plain-English descriptions of scenarios into concrete documented scams similar to the scenario.
- We learned how to utilize grok bot and it's agentic AI capabilities in our tech stack to increase productivity. *Light, local and dictionary-based word matching with normalization, tokenization, match cues, and word-association weighting makes for a fast, effective guide that keeps you out of danger on the internet while respecting your privacy.
- Sherpa was tuned on data scraped directly from real internet scams on Reddit and Discord.
What we learned
- RAG makes a smaller model far more reliable. By grounding Grok in our own catalog, the consistency and speed became much better than a cold ask.
- Lightweight and shipping fast with reliability beats fancy. TF-IDF vector search needed no API calls or GPU and performed just as reliably while still leaving clear room to implement the neural embeddings and pgvector for a better vector database.
- In this project we learned the importance of collaborating and building on one another's strengths and abilities. Our system comprises of a React frontend, a FastAPI detection service, a Postgres pattern database, a desktop application, and a Chrome extension, so we had to establish interfaces early to deliver a quality product. We also learned to integrate continuously using git, rebasing onto our teammates' pushes instead of overwriting them, and to handle API keys safely.
What's next for YPINR
- Distribute it! We already have so many people in our vicinity who would benefit from what we built this weekend.
- License work with teleproviders like T-Mobile, AT&T, and more to provide the most integrated scam protection on the market, handling the problem in real time rather than after the fact.
- Neural embeddings + pgvector to scale the intel repository from hundreds to millions of patterns
- Officially licensed Browser Extension and Discord Bot to bring detection to more users officially.
- Expansion of Sherpa to other Social Media sites like Twitter and SMS, with user permission.
- Adaption via the public intel corpus from confirmed reports to cast our net wider and effectively protect more people.
Built With
- chrome
- css
- cursor
- fastapi
- grok
- html
- javascript
- mako
- postgresql
- python
- react
- shell
- textbelt
- tkinter
- typescript
- vite
Log in or sign up for Devpost to join the conversation.