Aegis AI

Aegis AI is a zero-trust voice agent that screens inbound calls to block phone scams, prevents credential leaks, and handles deliveries in real time.

Comment

## Inspiration

Phone-based financial fraud, social engineering attacks, and high-pressure impersonation tactics (such as fake digital arrests and banking emergencies) are escalating exponentially. Vulnerable individuals and busy professionals alike struggle to identify sophisticated manipulation in real time. We wanted to design an uncompromising, always-on gatekeeper: an autonomous inbound voice screener that intercepts unknown calls, adheres strictly to zero-trust security boundaries, and neutralizes social engineering before it reaches the human user.

What It Does

Aegis AI serves as an intelligent voice proxy for inbound calls. Operating on a strict zero-disclosure policy, it:

  • Blocks Social Engineering & Credential Harvesting: Firmly denies sharing one-time passwords ($OTPs$), personal identification numbers ($PINs$), or personally identifiable information ($PII$).
  • Handles Everyday Logistics: Autonomously clarifies courier and food deliveries, instructing delivery partners with concise drop-off guidelines in under two conversational turns.
  • Automated Summarization: Transcribes, classifies, and logs every incoming call, flagging high-risk interactions for user review while safely dismissing routine inquiries.

How We Built It

Aegis AI was developed using an end-to-end voice pipeline designed for minimal conversational latency:

  • Orchestration & Voice Layer: Built on the Vaani platform, integrating real-time streaming speech-to-text (STT) and low-latency text-to-speech (TTS) engines.
  • Intelligence & Guardrails: Powered by llama-3.1-8b-instant via Groq, enforcing strict system-level constraints and zero-shot intent verification to ensure responses remain succinct ($<25$ words) and security parameters are never bypassed.
  • Latency Optimization: Optimized pipeline latency such that total turnaround time satisfies: $$\Delta t_{\text{total}} = t_{\text{STT}} + t_{\text{inference}} + t_{\text{TTS}} \le 1.2\,\text{s}$$ This sub-second conversational flow prevents awkward pauses and mimics natural human turn-taking.

Challenges We Faced

  1. Prompt Injection & Emotional Coercion: Call scammers frequently employ urgency and fabricated legal threats. Enforcing the LLM to refuse all credential requests without breaking character or hallucinating compliance required rigorous prompt engineering and strict boundary testing.
  2. Turn-Taking and Conciseness: Initial model responses were overly explanatory. Scammers exploit long interactions to manipulate conversational context. We iterated heavily on system directives to limit output length to 1–2 actionable sentences.
  3. Sub-Second Latency Requirements: Managing synchronous speech synthesis while parsing caller intents in real time demanded careful model selection and efficient token streaming.

What We Learned

  • Designing voice-first user experiences requires fundamentally different prompt strategies than text-based chat—brevity, cadence, and tone matter just as much as factual accuracy.
  • Implementing zero-trust principles at the conversational AI layer is an effective, practical countermeasure against modern phone-based social engineering.
  • Streamlined inference engines combined with dedicated voice infrastructure can execute complex guardrail logic within natural conversational delays.ration

Built With

  • anti-fraud
  • conversational-ai
  • cybersecurity
  • groq
  • llama-3.1
  • llm
  • python
  • speech-to-text
  • telephony
  • text-to-speech
  • vaani
  • voice-agent
  • voice-ai
  • zero-trust
Share this project:

Updates