ObservAI - AI Trust Hub

ObservAI makes Vertex AI apps fully observable by streaming LLM metrics, logs, traces, and risk alerts into Datadog—helping teams ship global AI with confidence, cost control, and governance.

Comment

ObservAI – AI Trust Hub for Global-Scale LLM Systems

Inspiration

As Gemini and Vertex AI applications move into production, teams face a dangerous blind spot: traditional observability tools cannot explain AI behavior. Latency alone doesn’t reveal hallucinations, cost explosions, or regional inequality in AI systems.

How can we ship AI globally if we can’t observe, govern, or trust it in production?

ObservAI was built to close this gap.

What it does

ObservAI transforms Gemini and Vertex AI applications into fully observable, governed systems.

It captures and streams LLM-native metrics, logs, and traces into Datadog, enabling teams to:

  • Monitor LLM latency, tokens, cost, and quality in real time
  • Trace every prompt and response end-to-end
  • Detect hallucinations, risk spikes, and abnormal behavior
  • Trigger Datadog alerts, incidents, and actionable workflows
  • Analyze global cost fairness across regions
  • Maintain a cross-border AI audit trail for compliance and trust ObservAI turns AI telemetry into decisions engineers can act on.

How we built it

ObservAI is a production-ready observability layer built on Gemini AI, OpenTelemetry, Supabase, and Datadog.

System Architecture

Key Technologies

  • Gemini 3 Flash / Vertex AI – Core LLM inference
  • ObservAI SDK – LLM interception and AI-native metric extraction
  • Antigravity + OpenTelemetry – Vendor-neutral instrumentation
  • Supabase Edge Functions – Secure, scalable backend
  • Datadog – Metrics, logs, traces, monitors, alerts, and incidents

Challenges we ran into

  • Designing AI-specific detection rules instead of generic thresholds
  • Preventing alert fatigue while maintaining high-signal monitoring
  • Instrumenting LLMs without increasing latency
  • Converting observability data into fairness and governance insights
  • Explaining a complex system clearly within a short demo format Each challenge strengthened ObservAI’s production readiness.

Accomplishments that we're proud of

  • Implemented 40+ LLM-native detection rules in Datadog
  • Built real-time risk alerts with actionable incidents and cases
  • Created a Global Cost Fairness Dashboard (unique and memorable)
  • Delivered a Cross-Border AI Audit Trail aligned with compliance needs
  • Shipped a fully deployed, open-source system, not just a concept

What we learned

  • AI observability requires context, not just metrics
  • LLM behavior varies significantly across regions and usage patterns
  • Datadog becomes a powerful AI control plane when used end-to-end
  • Trust and governance are missing layers in today’s AI stacks

What's next for ObservAI – AI Trust Hub

ObservAI aims to become the standard trust layer for production AI systems.

Next steps include:

  • Automated remediation workflows in Datadog
  • Expanded compliance reporting (EU AI Act, SOC 2, ISO)
  • Multi-model observability beyond Gemini and Vertex AI
  • Team-level governance, approvals, and policy enforcement
  • Enterprise-ready AI risk scoring and certification

Our vision: Make shipping global AI as safe, observable, and trustworthy as shipping traditional software.

Built With

Share this project:

Updates