LLM Drift & Abuse Sentinel

Real-time LLM observability with semantic drift detection, multi-layer safety, and comprehensive Datadog integration

Comment

## INSPIRATION

Large Language Model applications are becoming production-critical, but traditional APM tools miss LLM-specific issues:

The Problem:

  • LLM responses drift over time (model updates, prompt changes) - latency metrics don't catch this
  • Safety issues (jailbreaks, PII, toxic content) are invisible without semantic analysis
  • Token costs spiral out of control without real-time visibility
  • When LLMs misbehave, there's no trace showing WHY

Vision: Build a comprehensive observability platform that treats LLM responses as first-class citizens. We wanted engineers to answer: "Why did my model start giving different answers?" and "Is my LLM exposing PII?" as easily as "Is my API slow?"

## WHAT IT DOES

LLM Drift & Abuse Sentinel provides production-ready observability for Vertex AI / Gemini applications:

🔍 Intelligent Drift Detection

  • Embedding-based semantic analysis detects when responses drift from baselines
  • Z-score anomaly detection flags unusual patterns
  • Automated baseline management with exponential moving averages

🛡️ Multi-Layer Safety

  • AI-powered classification (TOXIC, PII, JAILBREAK, PROMPT_INJECTION, RISKY, CLEAN)
  • Real-time risk scoring with automatic alerting
  • Compliance-ready audit trail

📊 Datadog Integration

  • 15+ custom metrics (latency, tokens, cost, drift, safety, anomalies)
  • Full APM tracing with rich tags (llm.model, llm.drift_score, llm.safety_score)
  • 5 detection rules with automated incident creation
  • Executive dashboard showing health, costs, and security

🏗️ Production Architecture

  • Async telemetry (sub-50ms overhead via Pub/Sub)
  • Cloud Run deployment with auto-scaling
  • BigQuery for historical analysis
  • Real-time cost tracking

## HOW I BUILT IT

Tech Stack:

  • TypeScript/Express.js backend, React frontend
  • Vertex AI (Gemini 2.5 Pro, text-embedding-004)
  • Cloud Run, Pub/Sub, BigQuery
  • Datadog dd-trace APM, StatsD metrics, Events API
  • Terraform IaC

Key Innovations:

  1. Semantic Drift Detection - Uses embeddings to capture meaning, not just keywords. Cosine similarity against baseline detects subtle behavior changes.

  2. Zero-Overhead Telemetry - Pub/Sub async buffer adds <50ms latency. Users get immediate responses while analysis runs in parallel.

  3. Context-Rich APM - Custom span tags (tokens, cost, drift, safety) make traces searchable by business context.

  4. Intelligent Alerting - Composite conditions reduce noise (e.g., drift alerts only when baseline_ready:true).

Datadog Implementation:

  • Request-level metrics for every LLM call
  • 5-minute aggregated trends for drift/safety
  • Tagged metrics (env, model, endpoint, safety_label)
  • Monitors with detection + context + runbook + incident creation
  • Dashboard using drill-down pattern (golden signals → business metrics → diagnostics)

## CHALLENGES

  1. Embedding Cache Performance - Initial 250ms overhead per analysis. Built LRU cache achieving 65% hit rate, dropping to 87ms average.

  2. Baseline Cold Start - First 5 requests had no baseline, causing false alerts. Added baseline_ready tag and BigQuery persistence.

  3. Metric Cardinality - Tagged metrics with request_id created millions of series. Moved request_id to APM tags only, kept aggregate tags for metrics.

  4. Safety Latency - Gemini Pro added 400ms. Switched to Flash (150ms) and made async, keeping user response <1.5s.

  5. State Management - Local Terraform state blocked collaboration. Created bootstrap module for GCS backend.

## ACCOMPLISHMENTS

Technical: ✅ Zero false positives in drift detection ✅ Sub-50ms telemetry overhead ✅ 65% cache hit rate on embeddings ✅ 99.5% availability (exceeds 99% SLO) ✅ Full distributed tracing across 3 services

Innovation: 🚀 First semantic drift detection using embeddings 🚀 Multi-layer safety (AI + keyword + anomaly) 🚀 Production-ready from day one 🚀 Excellent developer experience

Datadog: 📊 5 monitors (required 3) 📊 Rich APM tagging 📊 Actionable alerts with runbooks

## WHAT I LEARNED

LLM Observability:

  • Traditional metrics miss LLM-specific issues
  • Embeddings >> string comparison for drift
  • Async analysis is essential
  • Baselines need warmup time

Datadog:

  • Custom APM tags are incredibly powerful
  • Metric cardinality strategy matters
  • Events + Monitors = comprehensive alerting
  • Dashboard layout should follow investigation workflow

Production:

  • Health checks (liveness vs readiness) matter
  • Rate limiting protects from cost explosions
  • Documentation quality drives adoption

## WHAT'S NEXT

Short-term:

  • Slack integration with interactive alerts
  • Auto-tuned thresholds from historical data
  • Per-model cost budgets with cutoffs
  • Multi-region deployment

Medium-term:

  • Prompt versioning to isolate changes
  • A/B testing across model variants
  • RAG observability (retrieval quality)
  • Automated remediation (auto-switch models)

Long-term:

  • Open-source NPM library
  • Multi-provider support (OpenAI, Anthropic, Bedrock)
  • ML-based drift prediction
  • Compliance automation (SOC2/GDPR reports)

Built With

Share this project:

Updates

Submission history