Observyze

High-throughput LLM observability & runtime reliability platform with sub-10ms telemetry, in-memory PII scrubbing, and real-time autonomous circuit breakers.

Comment

💡 Inspiration

As AI applications evolve from simple one-shot prompts to autonomous multi-agent systems, debugging becomes extraordinarily difficult. Production LLM chains suffer from non-deterministic hallucinations, runaway recursive loops, costly token drain, and data privacy risks when transmitting raw user prompts.

Most existing LLM observability tools are passive sinks — they only log failures after bad outputs have already reached users or sensitive API keys have leaked over the wire. We built Observyze as an active runtime reliability layer that intercepts, sanitizes, traces, and protects production AI workflows in real time.

⚙️ What It Does

Observyze is an end-to-end LLM Observability & Guardrails platform for engineering teams:

  • ⚡ Sub-10ms Telemetry Pipeline: Captures complex multi-agent traces, tool calls, latency spikes, and tokens without blocking user-facing applications.
  • 🔒 Client-Side In-Memory PII Scrubbing: Vectorized pattern matchers scrub 11+ sensitive data types (emails, JWTs, API keys, SSNs, credit cards, IPs) locally in memory before TLS network dispatch.
  • 🛑 Autonomous Runtime Circuit Breakers: Synchronously intercepts runaway token loops, high-hallucination outputs (>0.80), and safety breaches with automated fail-closed/fail-open policies.
  • 💰 Multi-Model Cost Attribution: Dynamic per-token pricing tracking across 40+ foundation models (OpenAI, Anthropic, Gemini, Groq, Mistral, DeepSeek) with burn-rate forecasting.
  • 🌐 Universal Transparent AI Proxy: Drop-in reverse proxy with AES-256-GCM credential vaulting for zero-code-change onboarding.
  • 🧪 Collaborative Prompt Studio: Version control, semantic variable extraction, and side-by-side prompt regression testing.

🏗️ How We Built It

Observyze is engineered as a distributed, high-throughput microservices architecture:

  1. Ingestion Tier (Fastify + Redis):
    • High-concurrency Fastify gateway accepts telemetry batches.
    • Pushes payloads into a durable Redis queue via atomic LPUSH (<10ms SLA) before returning HTTP 201, completely decoupling client latency from database write locks.
  2. Durable Persistence Worker:
    • Decoupled worker micro-batches traces, performs hash deduplication, and persists to MongoDB with compound time-series indexes.
    • Publishes real-time events to Redis Pub/Sub for live dashboard WebSocket/SSE streaming.
  3. Guardrails & Evaluation Engine (Python + FastAPI):
    • Evaluates semantic consistency, hallucination risk, and safety scores using NLI pipelines and LiteLLM pricing tables.
  4. Client SDKs (@observyze/sdk):
    • TypeScript/Node.js and Python SDKs with automated client wrapping for OpenAI, Anthropic, and OpenTelemetry.
    • Features local FIFO ring buffers (MAX_QUEUE_SIZE = 1000) and exponential backoff retries ([1s, 2s, 4s, 8s, 16s, 30s]).
  5. Dashboard & Frontend:
    • Next.js 14 App Router with full Light/Dark mode alignment, TanStack Query, and Framer Motion.

⚡ Challenges We Ran Into

  • Zero-Latency Ingestion SLA: Database writes (MongoDB) during high trace bursts caused latency spikes in user applications. We solved this by architecting a Redis-backed asynchronous write queue that guarantees sub-10ms client response times while maintaining crash-safe durability.
  • In-Memory PII Recursion Safety: Scrubbing deeply nested agent execution graphs risked call-stack overflows. We implemented pure, non-mutating recursion bounded strictly to depth <= 16 with key-aware dictionary short-circuiting.
  • Exact Cost Calculation at Scale: Foundation model pricing changes frequently with complex sub-string version names (gpt-4o-2024-08-06, claude-3-5-sonnet-20241022). We built a dynamic LiteLLM cache with longest-prefix matching to accurately attribute fractional cents.

🏆 Accomplishments That We're Proud Of

  • Achieved sub-10ms trace ingestion with zero client-side performance degradation.
  • Built zero-trust in-memory PII sanitization that runs 100% locally before any network dispatch.
  • Successfully deployed full production infrastructure on Google Cloud Run, MongoDB Atlas, and Redis with automated CI/CD via Cloud Build.
  • Implemented active inline Circuit Breakers capable of synchronously stopping runaway autonomous agents.

🧠 What We Learned

  • In high-throughput AI telemetry, decoupling ingest acknowledgment from data persistence is critical for developer adoption.
  • Active guardrails must incorporate confidence-weighted scoring to prevent false-positive interruptions on ambiguous classifications.
  • Developers want zero-code-change integration paths (like our transparent proxy) alongside deep SDK capabilities.

🚀 What's Next for Observyze

  • Open-sourcing native integrations for LangChain, LlamaIndex, CrewAI, and AutoGen.
  • Automated prompt auto-tuning and regression test harnesses in CI/CD pipelines.
  • Enterprise cold-storage S3/GCS archival connectors for multi-year compliance audit trails.

🎁 Early Access Exclusive: We are offering 90 Days of Free Early Access Pro Tier (100k traces/mo, in-memory PII scrubbing, and real-time guardrails) for the community at https://observyze.com.

Built With

Share this project:

Updates