ACSA
ACSA - Automated Call Shield Assistant
Let ACSA analyze the call
ACSA listens to your call
And then analyzes the call passing it through 4 different layers
Finally gives you the verdict after careful analysis based on various factors

Inspiration

Voice-cloning scams are becoming a serious and rapidly growing threat. The FBI reports that Americans lost more than $893 million to AI-related scams last year, and according to Vectra AI, AI-powered scams surged by 1,210% in 2025. McAfee also found that 35% of people cannot distinguish a cloned voice from a real one, while victims aged 65 and older are three times more likely to be targeted.

Security experts already recommend that families establish a private "safe word" to verify someone's identity over the phone. While this can help, it still relies on someone remembering to use it in a stressful situation. ACSA, the Automated Call Shield Assistant, takes that idea a step further by making caller verification automatic.

What It Does

ACSA screens incoming calls before you ever have to pick up. When a call comes in, your phone displays a custom call-screening interface asking, "Would you like ACSA to answer for you?"

If you select yes, an AI assistant answers the call and asks the caller to state their name and reason for calling. While the caller responds, ACSA runs their information through a layered detection pipeline that analyzes the call in real time.

The result is displayed through a simple color-coded risk gauge:

Green: The caller is likely human, with no major warning signs detected.
Yellow: The caller appears to be human, but the conversation contains language or behavior commonly associated with scams.
Red: The system detects signs of an AI-generated voice potentially impersonating someone the user knows.

The goal is to give users clear information about a caller before they have to decide whether that person can be trusted.

Key Features

Known scam number lookup: Checks incoming numbers against a MongoDB database sourced from FTC complaint data.
Speaker verification: Compares the caller's live voice against a pre-enrolled reference sample of the person they claim to be.
ElevenLabs AI voice detection: Provides a second-opinion authenticity check to help identify potentially synthetic speech.
LLM-based scam language reasoning: Uses Gemini to identify suspicious language patterns, including urgency, secrecy, and unusual payment requests.
Custom iOS call-screening interface: Provides a streamlined, hands-free screening experience.
Live transcript and real-time risk gauge: Allows users to follow the conversation while seeing ACSA's risk assessment update.

How We Built It

Twilio: Answers incoming calls, plays the AI-generated greeting, and records the caller's response.
ngrok: Tunnels our local FastAPI backend to a public URL, allowing Twilio's webhooks to communicate with the backend during development.
ElevenLabs Speech-to-Text: Transcribes the caller's recorded response so it can be analyzed by the rest of the detection pipeline.
MongoDB Atlas: Performs the first check by comparing the caller's phone number against a cleaned dataset of approximately 950 known scam numbers collected from FTC Do Not Call complaint data.
Resemblyzer: If the caller claims to be someone the user has already enrolled, Resemblyzer creates an embedding of the caller's live voice and compares it against the pre-enrolled reference sample.
ElevenLabs: Runs the caller's audio through ElevenLabs' AI voice detector as an additional layer, helping identify synthetic speech that speaker verification alone might miss.
Gemini API: Acts as the tie-breaker when earlier layers are inconclusive. Gemini analyzes the transcript for language commonly associated with scams, including urgency, secrecy, and suspicious payment requests.
FastAPI Backend: Orchestrates the entire detection waterfall, manages communication between each service, and exposes the final results to the mobile application.
SwiftUI iOS App: Provides the custom-built call screen, live transcript, and risk gauge. The prototype does not rely on CallKit or PushKit, allowing us to run it using a free Apple developer account.

Challenges We Ran Into

We tested several audio deepfake detectors before arriving at a working combination. This included a local Hugging Face model and two models trained on ASVspoof. All three failed the same diagnostic probe using inputs such as silence, white noise, and reversed speech. The results showed that the models were reacting heavily to noise characteristics rather than reliably determining voice authenticity.
Getting MongoDB, Resemblyzer, and ElevenLabs to work together cleanly within a single waterfall required careful coordination. Each layer needed to know when it had enough confidence to return a result and when it should defer to the next layer rather than forcing a premature verdict.
Validating speaker verification in a real environment was another major challenge. A genuine voice scored 0.85 or higher in similarity against its own enrolled sample, while an AI-generated clone scored 0.72. That clear and reproducible difference gave us confidence in the speaker-verification layer and helped determine how the detection waterfall should be ordered.
ngrok's free tier generates a new public URL every time the tunnel restarts. Throughout development, this meant repeatedly updating Twilio's webhook configuration and our own backend URL constants whenever the tunnel changed.
We also worked through several infrastructure and development issues, including a pkg_resources crash, a Twilio trial account that blocked calls from unverified numbers, a localhost backend URL that worked in the simulator but failed on a physical device, a full hard drive that prevented Xcode from building, and a GitHub push that appeared to freeze because an authentication prompt was waiting in the background.

Accomplishments We're Proud Of

A real, live-tested detection waterfall: We built a working pipeline that combines number reputation, voiceprint matching, and AI voice detection to distinguish between a real caller and an AI-generated clone.
Refusing to fake a result: When our early deepfake detectors failed the diagnostic probe, we investigated why they were failing instead of hardcoding a workaround just to make the demonstration appear successful.
A true defense-in-depth system: A scammer has to get through several independent layers, including the known-number check, voiceprint comparison, AI voice detection, and LLM-based contextual reasoning. The system does not rely on a single model to make every decision.
A complete call-screening experience: We built and deployed a fully working call-screening interface to a physical iPhone using a free Apple developer account.
Built under real time constraints: The entire system was designed, built, tested, and debugged within a single 36-hour hackathon.

What We Learned

A rigorous negative result is still valuable engineering: Proving through controlled testing that a detector does not work reliably is more valuable than shipping a detector simply because it happens to produce the expected result on one example.
Multiple independent signals are stronger than a single detector: Combining identity verification, audio authenticity analysis, and contextual language analysis gave us a much stronger system than relying on any one detection method alone.
Building the infrastructure matters just as much as building the AI: We learned how to connect telephony, biometrics, AI detection, LLM reasoning, backend infrastructure, and a mobile interface into a complete working product while operating under significant time constraints.

What's Next for ACSA

CallKit and PushKit integration: Integrate ACSA more deeply with iOS so calls can be screened even when the phone is locked or the application is running in the background.
Stable production URL: Replace ngrok with a permanent production endpoint so the backend is no longer dependent on a temporary tunnel running from a development laptop.
Expanded audio-authenticity detection: Add and evaluate additional detectors trained on newer AI voice generators to improve the system's ability to recognize modern synthetic speech.
Larger scam-number database: Expand beyond FTC complaint data by incorporating a larger, potentially crowdsourced database of reported scam numbers.
Simpler voice enrollment: Allow users to create trusted voice profiles using existing voicemails or recordings instead of requiring a separate enrollment recording.
Android support: Expand ACSA beyond iOS so the same protection can be available to Android users.