GERAS: Guided Expedited Real-time Audio Screening

Inspiration

Phone scammers frequently target older adults by conducting financial, healthcare, and other forms of fraud over phone calls. Older individuals frequently prefer phone calls over digital customer service channels, and are especially vulnerable to this type of attack. Healthcare fraud is also especially prevalent as individuals over 65 automatically qualify for Medicare. To address this specific issue, our group developed GERAS, an AI-powered real-time scam detection tool.

What it does

GERAS uses a machine learning model to determine whether a phone call is malicious. For our demo, we compared S4 (Structured State Space for Sequence Modeling) with a fine-tuned BERT model. The S4 model generates an approximate reconstruction of the conversation using a very specialized latent space representation that gets updated in linear time. One key feature is that the sequence is never stored explicitly, which circumvents the limitations of a context window and of storage. A second key feature is that GERAS is capable of running inference online using far fewer parameters than a LLM.

How we built it

We used an A100 GPU to train and fine-tune our models, then we used Streamlit to build the frontend.

Challenges we ran into

One of the primary challenges was the limited availability of data for scam calls, often containing private or sensitive information shared by victims with scammers. To overcome this, we utilized recordings from organizations such as the FCC and other government agencies focused on consumer protection, supplemented by the generation of synthetic data.

Accomplishments that we're proud of

Our structured state space approach does not require learning English or understanding the semantic meaning of a conversation. Yet, it can be employed for conversation classification, thereby reducing reliance on high-quality audio for accurate speech-to-text conversion.

What we learned

We gained insights into the complexities of addressing scam calls and the importance of innovative solutions that prioritize privacy and efficiency.

What's next for GERAS: Guided Expedited Real-time Audio Screening

Our next steps include integrating GERAS with cell phones or applications with phone call functionality, such as WhatsApp, to extend its reach and effectiveness. These apps are our strategic next steps as many older adults use them frequently to contact relatives and friends.