Aegis AI

Real-time AI-powered scam call detection to keep you safe from phone scams

Comment

💡 Inspiration

  • We started with a narrow scope to help elderly people and those vulnerable to phone scams, but this prototype now helps all users protect themselves from sophisticated scams that traditional call blocking can't catch.
  • Aegis AI is easy to use - when you receive an unknown call, choose to let our AI assistant answer instead. The AI converses naturally while analyzing in real-time, keeping you and the AI aligned on whether calls are legitimate or fraudulent. Helping everyone including non-tech-savvy users.
  • Traditional call blocking relies on static blacklists, but scammers use spoofed numbers and evolving tactics that are hard to detect. We created a proactive defense system where an AI assistant screens calls first, analyzes conversations in real-time, and only connects you to legitimate callers.
  • Aegis AI transforms your phone into a smart shield - when unknown numbers call, the AI answers and analyzes the conversation for scam indicators, automatically blocking scams or handing safe calls back to you.

⚙️ What it does

AI-powered call protection system which intercepts unknown calls to analyze, detect scams, and protect users in real-time.

API used - ElevenLabs ConvAI (voice assistant), Google Gemini 1.5 Pro (scam detection), Pattern Matching API (fast detection), Real-time Analysis API (live scoring).

🛡️ Protection Components:

  • Dashboard Interface: Monitor active calls in real-time with live transcript, risk meter, and visual indicators - all in one comprehensive view.
  • AI Voice Assistant: ElevenLabs ConvAI answers calls on your behalf, converses naturally with callers, and gathers information while you watch the analysis unfold.
  • Real-Time Analysis Engine: Google Gemini analyzes conversations as they happen, providing instant scam risk scoring and keyword detection.
  • Call History Manager: Complete call history with risk assessments, blocklist management, and statistics tracking.
  • Settings Panel: Select different AI models, voice preferences (default/female/male), enable/disable features, and configure protection levels.

✨ Key Features:

  • Hybrid AI Detection: Two-layer defense system combining fast pattern matching (70+ known scam phrases) with deep AI analysis (Google Gemini 1.5 Pro) for comprehensive protection.
  • Real-Time Monitoring: Watch conversations unfold live with real-time transcript display, dynamic risk meter, and instant scam detection.
  • Multi-Voice Support: Choose from three AI assistant voices (default/female/male) for personalized call screening experience.
  • Intelligent Blocking: Automatically blocks confirmed scammers, adds numbers to blocklist, and prevents future calls from flagged numbers.
  • Safe Call Handoff: Returns legitimate calls to you after AI verification with confirmation message.
  • Pattern Detection: Immediate flagging of high-priority scams (OTP requests, bank balance checks) with 85-100 risk scores.
  • Context-Aware Analysis: AI understands conversation flow, tone, and behavior patterns - not just keywords - to catch sophisticated scams.
  • Graceful Degradation: Falls back to pattern-only detection if AI services unavailable, ensuring protection never stops.

🎯 Where it fits best (Practical scenarios):

  • Unknown Caller Protection (Dashboard): Receive an unknown call, divert to AI protection, and watch real-time analysis to determine if it's safe or a scam.
  • Works on any device (Web Dashboard): Access protection from desktop or tablet, monitor calls, review history, and manage blocklists from anywhere.
  • Real-time monitoring (Live Analysis): Watch active calls with live transcript, see risk scores update in real-time, and get instant scam alerts.
  • Call history review: Analyze past calls, review transcripts, check risk assessments, and manage your blocklist of confirmed scammers.
  • Elderly and vulnerable users: Provides automatic protection for users who may not recognize scam tactics, with clear visual indicators and automatic blocking.
  • Business and personal use: Protect both personal and business lines from sophisticated phone scams and fraud attempts.
  • Multi-language support (planned): Extend protection to international calls and multi-language scam detection.
  • Mobile integration (in progress): React Native app for Android call interception, bringing protection directly to your phone.

🛠️ How we built it

We designed, selected the proper tech stack, and thought about features like real-time call analysis, AI voice integration, multi-layer detection, etc.

  • Modern Web Architecture with Next.js: Built using Next.js 16 App Router with React 19, TypeScript throughout for type safety and maintainability, and shadcn/ui components for reactive UI and streamlined state management. Ensuring compatibility across modern browsers.
  • Modular Component System: Implemented separate modules for each surface including dashboard interface for real-time monitoring, iPhone mockup component for visual call interface, call history manager for past call analysis, settings panel for configuration. Backend API service for mobile app integration.
  • Multi-Provider AI Integration: Leveraged Google Gemini 1.5 Pro for deep semantic analysis and ElevenLabs ConvAI for natural voice interactions, employing a hybrid approach with fast pattern matching for immediate detection and cloud AI models for sophisticated scam analysis. Plus real-time streaming with debounced analysis and timeout handling for reliable UX.
  • State Management and Context Awareness: Utilized React hooks and state management for call state, feature flags, and global settings, combined with real-time transcript extraction from active calls and conversation analysis for accurate scam detection. Plus secure storage abstraction for API keys and user preferences to maintain intelligent context throughout call sessions.
  • Privacy-First Development with Robust Build System: Emphasized privacy through hybrid AI that keeps sensitive operations on-device where possible with minimal data collection and user control. While Next.js handles server-side rendering, API routes, and production builds with TypeScript strict checking, optimized streaming with chunk processing, lazy loading for UI components, and minimal permissions for security and user trust.

🚧 Challenges we ran into

  • Real-Time Audio Streaming and Analysis Coordination: Implementing bidirectional audio streaming with ElevenLabs ConvAI while simultaneously analyzing transcripts with Google Gemini proved complex. We had to handle Web Audio API for audio capture and playback, WebSocket management for real-time communication, coordinate multiple AI services working together, parse and sanitize transcript data for analysis, and inject real-time updates into React components. Balancing performance with real-time updates as conversations unfolded required careful debouncing, efficient state management, and proper cleanup to prevent memory leaks across extension boundaries.
  • Different API Technicalities and Provider Inconsistencies: Each AI provider (ElevenLabs ConvAI, Google Gemini, Pattern Matching) had their own unique API structures, streaming formats, authentication methods, and rate limits. We tackled inconsistent response formats between ElevenLabs WebSocket messages and Gemini JSON responses, varying error handling patterns across services, provider-specific features like voice selection and agent mapping, and different streaming mechanisms. Creating a unified abstraction layer while preserving each provider's capabilities required extensive testing, fallback strategies for reliability, and graceful degradation when services were unavailable.
  • Making UI and UX Intuitive with Multiple AI Services and Features: Designing an intuitive interface that seamlessly handles real-time call monitoring, AI voice interaction, scam detection, blocklist management, and call history across the dashboard was challenging. We had to prevent feature overload while maintaining comprehensive protection, provide clear visual feedback for streaming responses and risk scoring, handle model switching gracefully between pattern matching and AI analysis. Ensure users could quickly understand scam risk levels and take action, and maintain consistent behavior across all components despite the complexity of coordinating multiple AI services working in real-time.

🏆 Accomplishments that we're proud of

  • Multiple AI API integration: Integrated multiple AI APIs along with their config so users can leverage ElevenLabs ConvAI for natural voice interactions, Google Gemini 1.5 Pro for deep scam analysis, and pattern matching for fast detection. Enabling seamless coordination between services.
  • Hybrid AI Integration: Successfully implemented a flexible system allowing users to benefit from both fast pattern matching (immediate detection of known scams) and sophisticated AI analysis (catching evolving tactics). Enabling seamless model coordination based on conversation context and use cases.
  • Real-Time Multi-Modal Functionality: Developed comprehensive real-time capabilities supporting both audio streaming from calls and text analysis of transcripts, with the ability to analyze conversations as they unfold. Provide instant scam risk scoring through live dashboard updates.
  • Universal Call Protection: Created versatile protection system that works contextually for any incoming call, providing free and accessible AI-powered screening. Without requiring users to manually block numbers or disrupt their workflow.
  • Context-Aware Detection: Built AI functionality that understands and maintains context for the current active call, delivering relevant and intelligent scam detection. Based on conversation flow, caller behavior patterns, and real-time analysis.
  • Seamless User Experience: Achieved instant, in-context interactions through dashboard interface including real-time transcript display for active calls, customizable voice options, risk scoring. And a persistent monitoring interface that integrates naturally into call protection workflow.

📚 What we learned

  • Working with AI via API: Built a multi-provider AI integration using Google Gemini 1.5 Pro for semantic analysis and ElevenLabs ConvAI for voice interactions. Implemented hybrid processing with fast pattern matching for immediate detection and cloud AI models for deeper analysis. Featuring real-time streaming with debounced analysis and robust timeout handling for reliable user experience.
  • Managing context: Implemented intelligent context awareness through React state management for call state and global settings, combined with real-time transcript extraction from active calls and conversation analysis. The system maintains conversation context across call sessions, securely stores API keys and preferences, and automatically shares current call context for relevant scam detection and accurate risk scoring throughout user sessions.
  • Using Real-Time Audio Streaming: Created a bidirectional audio streaming system using ElevenLabs ConvAI WebSocket that serves as the primary voice interaction interface with access to all features including natural conversation, real-time transcription, and audio playback. Features clean integration with Web Audio API, automatic context from the active call, supports multi-voice selection via agent mapping, enables real-time transcript updates, and provides quick access to settings for voice preference and model configuration.
  • Building Modern Web Architecture: Leveraged Next.js 16 framework with TypeScript for type-safe, production-ready web applications. Implemented modular component system with separate modules for dashboard interface, call monitoring, history management, and backend API service. Handled server-side rendering, API routes, and production builds with strict TypeScript checking for maintainability.

🚀 What's next for Aegis AI

  • We are planning to complete mobile app integration that can help users not only monitor calls from the web dashboard but also intercept calls directly on their Android phones. This will help people who need protection on-the-go and elderly users who primarily use mobile devices. This is one of the crucial things we wanted to add when we started this project. We aim to integrate React Native app with Android call interception to provide seamless protection across all devices.
  • We are also planning to add community blocklist and predictive blocking support in the next major release. This allows all users to benefit from shared scam number databases and proactive protection based on historical patterns. Making Aegis AI a comprehensive community-driven protection system.

Built With

Share this project:

Updates