Phantos AI

Agent Dashboard & Chat
Agent Dashboard & Chat #2
Otel Span Trace Tree
Otel Span Trace Tree #2
Otel Span Trace Tree #3
Otel Span Trace Tree #4
Arize Phoenix Traces
Arize Phoenix Traces #2
Phoenix Traces and Graphs
Phoenix Traces and Graphs #2
Phoenix Traces and Graphs #3
Self-Improvement System
Agent Battle Arena
Agent Battle Arena #2
Memory Palace
Light Mode
Dark Mode
Demo
Tour
Reset Console Base & Notification
Collapsed Sidebar

Inspiration

It started at 2:00 AM on a Tuesday. I was watching production logs for a customer service AI agent I had deployed. The central dashboard sat on green. All systems healthy. Response times normal. No errors flagged. Then I saw a specific conversation thread that stopped me. A customer asked a simple return policy question. The agent answered confidently, highly professionally, and completely wrong. It had hallucinated a non-existent restriction, causing the buyer to abandon their shopping cart. My traditional monitoring dashboard still glowed green because no hardware crashed.

I turned to community forums like Reddit (r/AI_Agents and r/LLMDevs) to see if other engineers faced this: "Three weeks later, caught it summarizing data that didn't exist. Completely made up. Formatted neatly. Looked exactly like real outputs." (847 upvotes) "Mine reported a profit that never existed for days. When I questioned it, it DEFENDED the wrong number." (1.2K upvotes) "Agents have no way to learn from mistakes in runtime. Every session starts blind from scratch." (634 upvotes)

This was every production AI team's nightmare: AI agents are black boxes. You see inputs and outputs, but you cannot easily see the intermediate reasoning, nor can you dynamically patch behavior without redeploying code. I set out to make the invisible visible. The word "Phantos" stems from the Greek root meaning to make manifest. We built Phantos AI to give agents self-awareness: an agent capable of inspecting its own trace trees, evaluating its outcomes via an LLM judge, and runtime-adjusting its own constraints using Arize Phoenix and Google Cloud.

What It Does

Phantos AI is a closed-loop customer service agent and telemetry dashboard that leverages Arize Phoenix OpenTelemetry and Gemini 2.0 / 3.5 to evaluate conversational quality in real time and automatically self-correct policy gaps.

Instead of passive, external monitoring, Phantos AI establishes a Self-Improvement Loop: Chat & Instrumentation: A customer asks a question. The Node/Express backend orchestrates a call to Gemini, fully instrumented via @arizeai/phoenix-otel to record spans, prompts, and tool arguments. Real-Time LLM Judge: Immediately upon generating a response, an asynchronous evaluator assesses the interaction across four dimensions: Accuracy, Helpfulness, Completeness, and Honesty. Trace Ingestion: Telemetry traces and quality ratings are flushed to Arize Phoenix Cloud (project: PHANTOS_AI).

Autonomous Correction Trigger: If any dimension's evaluation score falls below 0.60, a self-healing cycle triggers.

Trace Querying: The agent queries its own execution history in Phoenix to find failure patterns, isolate the root cause, and formulate a targeted "improvement rule."

Dynamic Prompt Hot-Patching: The new rule is injected into the agent's system prompt immediately. The next time the visitor asks the question, the score matches peak quality (e.g., rising from 0.41 🔴 to 0.89 🟢).

Key Components:

LLM Judge Evaluator: Under-2-second evaluation runs asynchronously to keep user-perceived latency low.

Agent vs. Agent Battle Arena: A live playground proving the value of observability. You can watch a blind, unmonitored baseline model compete side-by-side against Phantos AI across 10 rounds. You see the unmonitored model repeat identical mistakes, while Phantos learns from its trace history after

Round 1 and maintains perfect marks. Interactive Memory Palace (Knowledge Graph): A visual network mapping the agent's knowledge nodes. Nodes glow amber when unverified and turn emerald as traces verify their correctness. Operators can click nodes to read confidence details, hit counters, and manually edit the underlying rules.

Failure Autopsy interface: Allows engineers to zoom into failed runs, displaying complete telemetry trace graphs, exact evaluation metrics, and the auto-generated system prompt patches side by side.

How We Built It

We designed Phantos AI on a fast, decoupled full-stack architecture optimized for high-throughput AI operations: Frontend Experience (React + Vite + Tailwind + Motion): A desktop-optimized, responsive, high-fidelity dark-theme dashboard. Motion layouts provide fluent transitions, and Recharts parses and renders live trace trends. Server-Side API Proxy & Orchestrator (Node.js + Express + TSX): All interactions pass through a secure backend server. Client-side code never exposes API keys. Observability Core (Arize Phoenix & OpenTelemetry): The server initializes a registered tracer provider targeting the PHANTOS_AI cloud endpoint: code

TypeScript import { register } from "@arizeai/phoenix-otel"; const provider = register({ projectName: "PHANTOS_AI" }); Adaptive Serverless Layer: We engineered a robust adaptation sub-routine for the Node/Vite backend. When deployed in highly transient, serverless contexts (like Vercel or cloud run instances running on strict time-out frames), it dynamically scales back retry budgets and limits evaluation timeouts to keep runs snappy while safely flushing OpenTelemetry traces under budget limits: code TypeScript const isVercel = !!process.env.VERCEL; const actualTimeoutMs = isVercel ? 4000:20000; // Graceful telemetry flushing if (provider) { await withTimeout(provider.forceFlush(), 1000, "Phoenix flush timed out"); } AI Reasoning Pipeline: Powered by the official @google/genai TypeScript SDK. The engine utilizes model rotation to bypass transient Google AI Studio 503 (High Demand) or 429 (Quota Resource Exhausted) bottlenecks by falling back cleanly to fast backup configurations when spikes occur.

Challenges We Ran Into

High Demand API Fluctuation and 503/429 Failures During peak developer hours, the Gemini API sometimes encountered high demand limits or transient timeouts. The Solution: We implemented an active Model Rotation and Recovery Mechanism in our backend runner. If the primary model encounters a transient 503/429 error, the runner logs [Gemini Recovery] and routes subsequent retries to robust backup models (like gemini-flash-latest) using exponential backoff.
Telemetry Overhead and Force-Flush Latency in Serverless Contexts When flushing telemetry events to Arize Phoenix Cloud, waiting for long networking calls can delay the server's API response, causing serverless environments to cut off execution before traces arrive. The Solution: We isolated the forceFlush() call of the Phoenix OpenTelemetry provider, enclosing it in a protective timeout wrapper. In Vercel and serverless mode, we set the flush timeout to exactly 1000 ms, ensuring that the user's synchronous chat loop is never held hostage by trace uploads while guaranteeing that 99% of spans are still captured.
Over-Correction (Regression) in Prompt Engineering Early tests of the autonomous prompt-patcher would sometimes create overreaching feedback rules that inadvertently broke previously correct knowledge responses. The Solution: Instead of overwriting the entire system prompt, we separated our system instructions into an immutable Core Competence Section and an additive Dynamic Failure Protection Registry. This isolates runtime corrections so they target specific topics without introducing regression.

Accomplishments That We're Proud Of

First Direct Closed-Loop Self-Aware Agent: Built a clean, running showcase of an agent retrieving its own historical evaluation data from Arize Phoenix, identifying failure patterns, and repairing its own prompt structure in real-time. 0.41 to 0.89 Automatic Improvement: Proved a highly reproducible performance boost without human manual engineering. The agent detects its policy gaps and self-heals in single-digit seconds. Production-Grade Infrastructure: Clean separation of frontend and backend layers, securing all API secrets (like GEMINI_API_KEY and PHOENIX_API_KEY) on the server side. Beautiful Visual Experience: Transformed clinical debugging logs into a beautiful, immersive dashboard featuring moving trace metrics, real-time hallucination blocker counters, and an interactive memory palace node visualization.

What We Learned

Observability is the Core, Not an Afterthought: Once we had Arize Phoenix connected, debugging the agent's behavior became straightforward. Seeing exactly where a span failed is night-and-day compared to analyzing raw terminal streams. The Power of Asynchronous Evaluation: Running evaluations as a decoupled post-response process is crucial. Users receive instantaneous responses, while the background thread computes multi-dimensional scores and registers traces.

What's Next for Phantos AI

Cross-Session Vector Memory: Persisting the auto-generated evaluation rules into a vector-backed database so the agent retains its learnings across thousands of sessions and cold starts. Streaming Token Interception: Building a middle-tier token gate that intercepts active streams from Gemini. If a span’s confidence drops below a specified threshold mid-thought, the gate stops printing, rolls back, and regenerates the response before it ever reaches the user.

Multi-Agent Cascade Tracing: Setting up telemetry to follow nested or collaborative multi-agent architectures, tracking how a failure propagates across agent handoffs and allowing the self-improvement loop to identify which specific agent in the chain needs prompt correction.