Scams Gnome More

Home page (all sessions)
Home page (all sessions, continued)
Home page (searching/filtering sessions)
Call session (live)
Call session (ended)
Settings menu

Inspiration

We wondered: what is a problem that a lot of people face on a daily basis? That's right: spam calls, and with the rise of AI, they have gotten even more persistent and frequent! So, we wanted to see if there was a way that we could fight back without giving our time to these scammers. Eureka! What if we had a program to do that for us? And what if we could discourage scammers from continuing instead? What if there was a way to have Scams, Gnome More?

What it does

Scams Gnome More has many different features, including:

A gnome defender that takes the line from AI spam calls, with the sole intention of breaking their prompting and wasting as many of their tokens as possible
A modern, easy-to-use management console, allowing live audio streaming and transcripts of intercepted calls and adjustment of the backend
Operator call interjections, allowing you to suggest changes to the gnome mid-call
A record of all previous calls, including transcripts, telemetry, and audio recordings
Unique gnome agents with special personalities and backstories for each call, creating a fun and unique experience
Leverages call forwarding to route calls to a gnome's number, a modern phone feature

This project was centered around a key computer security principle: a defender has to defend from multiple attack vectors, some unknown, while an attacker only needs one. Therefore, by attempting various solutions, we’re bound to break them eventually.

How we built it

Our project consists of three separate components: the gnome backend, gnome frontend, and an attacker backend/frontend.

We used several technologies:

Gnome backend: Built using Python, FastAPI, Uvicorn, and NumPy; Google Gemini, ElevenLabs, and Twilio APIs for processing, thinking, audio generation, and virtual phone service
Gnome frontend: Built using Next.js, React, Tailwind CSS, and TypeScript
Attacker backend/frontend: Built using Python, FastAPI, and Streamlit; Google Gemini, ElevenLabs, and Twilio APIs for processing, thinking, audio generation, and virtual phone service

Challenges we ran into

Setting up a publicly accessible domain through cloudflared
Connecting the frontend and the backend
ElevenLabs being too slow to keep up with the live phone calls
AI agents speaking over each other, entering deadlock
Gnome agent breaking itself by following its rules laid out for the scammer

Accomplishments that we're proud of

A consistent product that does what it was designed to do more often than not
A clean, polished UI that is easy to look at and navigate
Creating something that can be interacted with using nothing more than your personal phone, without any installs required
Working as a team with a fair and structured division of labor, resulting in timely completion

What we learned

How to more efficiently leverage LLMs to accelerate software development
How to integrate outside APIs to enable more content with less work
How to overcome bottlenecks with practical solutions
How to use WebSockets for real-time data streams
How to design resilient protections for our AI to prevent it from being jailbroken by itself or another AI

What's next for Scams Gnome More

Client/Mobile app integration for end users, allowing for customization and logging of personal calls; different accounts for users
Features to detect if the caller is a person or an AI, with changes to the gnome persona accordingly
Complete call interception by directing all calls to gnome agents first, redirecting users who pass, providing further protections
Have gnomes learn from previous calls, becoming better and more effective with each session