TokenWatcher

Open-source AI observability platform that tracks LLM costs, token usage, latency, and performance with real-time analytics, forecasting, and a production-ready telemetry SDK.

Comment

Inspiration

Large Language Models are powering everything from chatbots to autonomous agents, but developers still lack visibility into what happens after an API request is made. Teams often discover high AI bills before understanding which models, endpoints, or features are responsible for the cost.

Existing observability platforms are often expensive, complex, or designed for traditional cloud infrastructure rather than AI workloads.

We built TokenWatcher to give developers a simple, production-ready platform for monitoring AI applications in real time. Our goal is to make AI systems more transparent, easier to optimize, and more cost-efficient without requiring developers to build custom monitoring infrastructure from scratch.

What it does

TokenWatcher is an open-source AI observability and cost intelligence platform for LLM-powered applications.

It helps developers:

  • 📊 Monitor token usage, API costs, latency, and request volume in real time.
  • 📈 Analyze usage by provider, model, endpoint, and workspace.
  • 🔍 Search and filter telemetry logs for debugging and auditing.
  • 🤖 Receive AI-powered insights, forecasts, and optimization recommendations.
  • 📱 Access reports and notifications directly through Telegram.
  • ⚡ Integrate applications using a lightweight TypeScript telemetry SDK with batching, retries, and graceful shutdown support.

Instead of only showing infrastructure metrics, TokenWatcher helps developers understand how their AI products behave, where resources are being consumed, and what should be optimized next.

How we built it

TokenWatcher was designed as a modular, production-ready platform.

Backend

  • Node.js
  • Express
  • TypeScript
  • PostgreSQL

Frontend

  • React
  • Vite
  • React Query
  • Tailwind CSS
  • shadcn/ui

Telemetry SDK

  • TypeScript SDK
  • Automatic request instrumentation
  • Request batching
  • Retry mechanisms
  • Graceful shutdown flushing

Platform Features

  • Workspace-based authentication
  • Server-Sent Events for real-time dashboards
  • AI-powered forecasting and recommendations
  • Secure API key management
  • Telegram integration through OpenClaw

The SDK collects telemetry directly from AI applications, the backend securely stores and analyzes it, and both the dashboard and Telegram interface present actionable insights in real time.

Challenges we ran into

One of the biggest challenges was designing a telemetry pipeline that remained lightweight while reliably handling batching, retries, and graceful shutdown without affecting application performance.

Building secure multi-workspace authentication, live analytics through Server-Sent Events, and forecasting based on historical telemetry required careful backend architecture.

Another challenge was integrating conversational access through Telegram while keeping the communication layer stateless, secure, and synchronized with the analytics platform.

Balancing scalability, performance, and developer experience throughout the system was one of the most valuable engineering lessons during development.

Accomplishments that we're proud of

  • Built a complete AI observability platform instead of a proof of concept.
  • Developed a reusable TypeScript telemetry SDK.
  • Implemented real-time dashboards with live analytics.
  • Added AI-powered forecasting and optimization recommendations.
  • Integrated Telegram for conversational access to analytics.
  • Designed a modular architecture capable of supporting additional AI providers and enterprise features.
  • Built a platform that can continue evolving beyond a single hackathon into a production-ready developer tool.

What we learned

Building TokenWatcher reinforced how important observability has become in the era of AI applications.

We gained experience designing reliable telemetry systems, scalable backend architectures, real-time data pipelines, secure multi-tenant platforms, and developer-focused SDKs.

More importantly, we learned that collecting data is only the first step. The real value comes from transforming raw telemetry into insights that help developers build faster, spend less, and make better engineering decisions.

What's next for TokenWatcher

Our vision is to make TokenWatcher the observability layer for modern AI applications.

Our roadmap includes:

  • Support for additional AI providers and inference platforms.
  • Advanced anomaly detection for unusual usage and spending patterns.
  • AI-powered root cause analysis and optimization recommendations.
  • Expanded reporting, dashboards, and export capabilities.
  • SDK support for additional programming languages.
  • Docker and cloud-native deployment improvements.
  • Team collaboration features with role-based access control.
  • Richer Telegram and conversational AI workflows.
  • Integrations with popular AI frameworks and developer platforms.

Ultimately, we want TokenWatcher to become the platform developers rely on to monitor, optimize, and understand every AI request from development to production.

Built With

Share this project:

Updates