WhatsPriority

Orchestrate is an AI multimodal WhatsApp router using Whisper ASR & EasyOCR to route text, voice & images to notify, digest, or mute. Features 93.3% accuracy, scam defense & live web dashboard.

Comment

💡 Inspiration- Every single day, millions of WhatsApp users suffer from severe digital information overload. Essential work dependencies, urgent building utility alerts, and time-sensitive payment reminders get buried beneath an endless stream of promotional discounts, forwarded greetings, audio voice notes, and phishing scams. Existing notification controls are primitive—muting an entire group silences critical admin announcements, while leaving notifications on leads to constant distraction. We were inspired to build WhatsPriority: an intelligent, multimodal AI Attention Firewall that understands text, voice notes, images, user schedules, and security risks to protect human attention in real time.

What it does- WhatsPriority evaluates incoming multimodal WhatsApp messages in real time across 11 message categories (personal, urgent, event, payment, business_update, promotion, greeting, forward, spam, scam, unknown) and automatically assigns: action: notify (interrupt now for time-sensitive pings), digest (batch for later reading), or mute (suppress unwanted noise). reason: Dynamic human-readable explanations citing specific brand names, order IDs, deadlines, or security flags. confidence: Routing confidence score from 0.0 to 1.0. evidence_message_ids: Topically matched historical message IDs backing each decision. It features an interactive 5-tab dark glassmorphism Web Dashboard, an AI Router Simulator, a Security Threat Interception Log, and a privacy-first live WhatsApp Web connection bridge with automatic session destruction.

🛠️ How We Built It- We engineered a 6-layer safety-first hybrid architecture:

  1. Multimodal Perception Engine: Uses OpenAI Whisper ASR to transcribe voice notes and EasyOCR (with PyTorch) to extract text embedded inside image posters and screenshots.
  2. Relational Context Graph: Aggregates 7 contextual datasets to evaluate user quiet hours (DND windows), group admin authority, business domain verification, and historical user engagement.
  3. Deterministic Safety Gate: Intercepts prompt injection attacks ("ignore previous rules...") and phishing scams (unverified domain mismatches, newly registered domains <30 days) before any LLM call.
  4. Temporal Burst Sequence Analyzer: Clusters rapid message streams into 30-minute burst windows to batch casual chatter while isolating urgent pings.
  5. Hybrid AI & Schema Validation: Combines Anthropic Claude function-calling (emit_routing_decision) for semantic reasoning with Pydantic validation to guarantee 100% contract compliance.
  6. Live Web Dashboard & Bridge: Built a glassmorphism web app hosted on GitHub Pages, plus a terminal QR code bridge for live WhatsApp testing.

🚧 Challenges We Ran Into-

  1. Multimodal Noise & Speed: Processing unscripted, noisy audio voice notes and low-contrast text posters without degrading routing latency required building an efficient local caching pipeline.
  2. Adversarial & Prompt Injection Defense: Malicious payloads attempt to trick AI routers into forcing action=notify. Building a deterministic safety gate ahead of the LLM layer guaranteed zero security false negatives.
  3. Contract Precision & Evidence Matching: Ensuring historical evidence_message_ids were topically relevant without hallucinating non-existent message IDs required designing a content-aware term-frequency overlap ranker.

Accomplishments that we're proud of-

  1. 93.33% Action Accuracy & 90.00% Joint Match verified against ground-truth benchmarks.
  2. 100% Unit Test Pass Rate: 12/12 pytest cases passed covering schema validation and non-bypassable safety gates.
  3. Claimed Advanced Differentiators: Successfully implemented multimodal perception, temporal burst batching, domain verification, and dynamic signal-composed reasons.
  4. Live Production Deployment: Fully hosted web dashboard live on GitHub Pages with zero external server dependencies.

🧠 What We Learned-

  1. Hybrid Architectures Outperform Pure LLMs: Combining deterministic code guardrails for safety and contract enforcement with LLM semantic reasoning creates a far safer, faster, and more reliable system than relying on LLMs alone.
  2. Content & Temporal Awareness: Routing decisions are exponentially more accurate when considering message burst timing, sender authority, and historical user engagement alongside raw text content.

What's next for WhatsPriority

  1. On-Device Local SLM Routing: Porting the LLM reasoning layer to run completely offline on-device via quantized Small Language Models (e.g., Llama-3-8B / Gemma-2B) for ultra-low latency and 100% offline privacy.
  2. Cross-Platform Messaging Bridge: Expanding WhatsPriority beyond WhatsApp to support Telegram, Signal, iMessage, and Slack through a unified notification triage engine.
  3. Personalized Routine Learning: Implementing active reinforcement learning from user feedback (swiping to dismiss or star messages) to continuously refine personalized quiet-hour and priority thresholds.

Built With

Share this project:

Updates