Inspiration
The inspiration behind Tony came from the urgent need to protect customers from fraudulent activities and scam tactics over phone calls. Traditional phone interactions leave vulnerable users exposed to sophisticated voice phishing and manipulative bad actors. We set out to build an intelligent, active safety agent that bridges the gap between raw call streams and user security, intervening during suspicious calls to evaluate risks before financial or personal harm occurs.
What it does
Tony is an AI defense agent integrated directly into a mobile application. When a user receives or initiates a suspicious phone call, they can invoke the app to join Tony into the live line. Once connected, Tony passively listens to the conversation in real time, analyzing speech patterns, key phrases, and behavioral cues to determine if the caller is a scammer. The app then provides instantaneous, actionable risk assessments and visual warnings to help the user handle the call safely.
How we built it
We built the mobile app to handle seamless call joining and background processing while routing live audio streams. The incoming audio from the phone call is ingested through Photon for fast, real-time transcription. The transcribed text is immediately passed to a Nemotron large language model hosted on a Brev cloud instance equipped with an NVIDIA L4 GPU, where it scans the dialogue for deceptive patterns and scam intent. Finally, the analysis and alert triggers are handled by a Model Context Protocol (MCP) server running on Photon 2.0 to stream real-time safety notifications straight to the app UI.
Challenges we ran into
Our primary technical challenge was maintaining ultra-low latency across the entire processing loop—from initial app audio ingestion to transcription, LLM inference, and final app notifications. Any noticeable delay in analyzing speech patterns could leave a user vulnerable during a fast-moving call. Additionally, training the agent to accurately distinguish between aggressive legitimate callers (such as debt collectors or persistent telemarketers) and actual fraudulent scams required careful prompt engineering and response structuring.
Accomplishments that we're proud of
We are proud to have built an end-to-end working system where a mobile app seamlessly bridges a live phone call to a GPU-accelerated AI analysis pipeline. Successfully linking Photon live transcription, Brev-hosted Nemotron reasoning, and Photon 2.0 MCP routing to flag scam indicators on active calls in real time was a major technical victory for our team.
What we learned
Building Tony gave us deep insights into constructing real-time voice processing architectures and managing low-latency compute resources on cloud GPUs. We also learned how critical modular integration tools like MCP are when connecting front-end mobile applications to complex multi-agent AI systems.
What's next for
Tony Looking ahead, we want to enhance the mobile app to automatically detect incoming suspicious numbers and join calls proactively without requiring manual user initiation. We also plan to train Tony to recognize synthetic voice cloning and audio deepfakes, ensuring comprehensive protection against next-generation phone scams.
Built With
- codex
- java
- livekit
- nemotron
- photon
- python
- react-native
- typescript

