TRACE-X

Dashboard
Insights
Incident report
Report details
Simulator demo

Inspiration

As AI agents become more capable, they are also becoming more difficult to debug and maintain. Traditional observability tools show logs, traces, and metrics, but they still leave developers asking the same question: "Why did my agent fail?"

During our exploration of multi-agent systems, we noticed that when an AI workflow breaks, developers spend a significant amount of time manually tracing execution paths, identifying root causes, and proposing fixes. We wanted to build something that goes beyond monitoring and acts like an AI reliability engineer.

This inspired TRACE-X — an AI-powered observability and self-healing platform that automatically detects failures, analyzes root causes, recommends repairs, and helps developers understand agent behavior in real time.

What it does

TRACE-X provides end-to-end observability and automated diagnosis for AI agent systems. The platform:

Captures traces from AI agents through a lightweight SDK.
Monitors agent performance, failures, and reliability metrics.
Uses a multi-agent AI pipeline to analyze incidents.
Identifies probable root causes using observability signals and LLM reasoning.
Generates repair recommendations with confidence scores and impact projections.
Reconstructs agent execution through a replay engine, allowing developers to step through failures frame by frame.
Streams incident updates in real time through a live dashboard.
Supports a complete Demo Mode with mock services, allowing the platform to run even without cloud credentials.

In short, TRACE-X helps developers move from "Something failed" to "Here's exactly why it failed and how to fix it."

How we built it

We built TRACE-X as a full-stack cloud-native platform.

Backend & AI Layer

Built with FastAPI, Python 3.12, and Pydantic v2.
Created a lightweight TRACE-X SDK that enables tracing with minimal code changes.
Designed a 5-agent reasoning pipeline:
- Observer Agent
- Diagnosis Agent
- Repair Agent
- Validation Agent
- Executive Agent
Integrated Gemini 2.0 Flash through Vertex AI for structured reasoning and diagnosis generation.
Connected to Arize observability services to enrich diagnostics with performance and trace insights.

Storage & Streaming

Used Firestore for operational data and trace storage.
Used BigQuery for analytics and long-term insights.
Implemented real-time event streaming using Pub/Sub and WebSockets.

Frontend

Built with Next.js 15, React 19, TypeScript, and Tailwind CSS.
Designed a live "AI Mission Control" dashboard.
Added animated incident detection and self-healing visualizations.
Implemented GitHub-style repair recommendation cards and replay views.

Reliability & Demo Experience

Built complete mock implementations of Firestore, Gemini, and observability services.
Added automatic fallback logic so the platform works even without cloud credentials.
Seeded Demo Mode with realistic traces, incidents, diagnoses, and repair histories for judging and testing.

Challenges we ran into

Building a platform that depends on many external services

TRACE-X integrates AI models, databases, observability tools, cloud infrastructure, and streaming services. Any missing credential or unavailable dependency could break the experience.

To solve this, we created a full Demo Mode with identical mock implementations that seamlessly replace real services whenever needed.

Debugging datetime inconsistencies

We encountered difficult-to-reproduce failures caused by mixing timezone-aware and timezone-naive timestamps. The issue only surfaced after failure injection scenarios and caused dashboard aggregation errors.

We resolved this by standardizing timestamp handling throughout the system.

Latency in diagnostics

Our diagnosis workflow depended on multiple observability lookups. Sequential calls created noticeable delays.

We improved responsiveness by parallelizing requests and optimizing timeout configurations, reducing diagnosis times significantly.

Frontend integration issues

As real data started flowing through the application, several type mismatches between APIs, hooks, and UI components appeared. Solving these issues required careful alignment between backend contracts and frontend data models.

Accomplishments that we're proud of

Building a complete AI-powered observability platform rather than a simple dashboard.
Creating a 5-agent autonomous diagnosis and repair workflow.
Developing a replay engine that reconstructs agent execution step-by-step.
Delivering a fully functional Demo Mode that works even without cloud credentials.
Designing an engaging real-time dashboard that visually demonstrates incident detection and recovery.
Making agent instrumentation extremely simple through a lightweight SDK.
Successfully combining LLM reasoning with observability signals to generate actionable diagnostics.

What we learned

Designing for failure early is critical, especially when building systems that depend on many external services.
Structured outputs from Gemini significantly simplify multi-agent workflows compared to parsing free-form text.
Interface-level mocking is far more maintainable than network-level mocking.
Real-time visual feedback helps users understand complex AI workflows much more effectively than static dashboards.
Small configuration issues can sometimes create larger problems than complex engineering challenges.
Developer experience is just as important as technical capability when building observability tools.

What's next for TRACE-X

We see TRACE-X evolving beyond observability into a true self-healing platform for AI systems. Our next goals include:

Automated repair validation through sandboxed execution.
One-click deployment of approved fixes.
Predictive failure detection before incidents occur.
Support for additional AI frameworks and agent ecosystems.
Advanced root-cause analysis using historical trace patterns.
Cross-agent dependency mapping and impact analysis.
Enterprise-grade governance, auditing, and compliance features.
Autonomous reliability agents capable of continuously optimizing AI workflows.

Ultimately, we want TRACE-X to become the operating system for AI reliability—helping teams build, monitor, understand, and eventually self-heal complex AI applications.