GPTCHA (CAPTCHA for Generative AI)

Example call conversation between the GPTCHA agent and RoboCaller agent
Example internal monologue and chain of reasoning from the GPTCHA agent
Custom voice cloning
Example embedded personal info documents for verifying information via fuzzy matching & semantic search

Note: If live speech synthesis / transcription doesn't work, we ran out of API credits. Let Jason know and we will fix!

Inspiration

Robocalls make up 60% of calls in 2021

Robocalls accounted for 60% of all calls.

40% of all incoming calls were scams.

The inspiration for our hackathon project came from the growing concern around the malicious use of Language Models and other Generative AI models. As these AI models become more sophisticated, they can potentially be used to automate spamming, phishing, and other nefarious activities. We wanted to develop a GAN-inspired solution that could distinguish between human and AI-generated responses, and prevent the misuse of these models.

What it does

Our project, called "GPTCHA" (CAPTCHA for Generative AI), uses a new version of CAPTCHA that can detect AI-generated responses. We built a proof of concept for a GAN-inspired, and built two generative adversarial models, one focused on being a robo-calling fraud agent and extracting sensitive information, the other focused on counteracting and verification. If the system scores a conversation and determines if it's been generated by an AI, or involves fraudulent activities.

How we built it

We built the GPTCHA system using a combination of LLMs, semantic search and other source checking methods. We prompt engineered both agents to be as convincing and human-like as possible. We also implemented vector-based search and text embeddings to provide additional verification methods and ways for the GPTCHA model to fact-check / verify information coming from the RoboCaller model, such as account info or other info encoded in bank account statement docs, among other personal identifying information that a normal person may have access to.

We implemented two custom voice models via Eleven Labs for this demo and used OpenAI Whisper to provide speech recognition and synthesis, and are in the process of cloning our own voices for the GPTCHA agent.

Challenges we ran into

We started training additional custom voice models of our own voices, which required a lot of audio data to get natural-sounding recordings without artifacts

More to be added

Accomplishments that we're proud of

We are proud to have developed a working prototype of GPTCHA that can detect AI-generated responses with a high degree of accuracy, and deployed a proof of concept for robo-calls. We were able to demonstrate that our system can effectively distinguish between human and AI-generated responses, even when the AI model is specifically trained to mimic human behavior.

What we learned

What's next for GPTCHA (CAPTCHA for Generative AI)

RLHF to improve both models with human intervention and scoring.

We plan to release our code online to github and encourage the community to participate in key areas of robustness, speed, and overall effectiveness. We aim to develop the project further so that the GPTCHA call/text assistant can be deployed in the real world and used to protect users from malicious scam calls and messages.