TokenWatcher

Open-source AI observability platform that tracks LLM costs, usage, latency, and performance with real-time analytics, forecasting, and a production-ready telemetry SDK.

Comment

Inspiration

As AI applications become more powerful, they also become more expensive to operate. Developers often know how much they are spending each month but have little visibility into why. Questions like which models consume the most tokens, which endpoints generate the highest costs, or where latency is increasing are difficult to answer without building custom monitoring infrastructure.

We created TokenWatcher to solve this problem. Our goal was to provide developers with a production-ready platform that makes AI usage transparent through real-time telemetry, analytics, forecasting, and actionable insights, helping teams optimize both performance and cost.

What it does

TokenWatcher is an open-source AI observability and cost intelligence platform for LLM-powered applications.

The platform enables developers to:

  • Monitor token usage, API costs, latency, and request volume in real time.
  • Visualize usage trends through interactive dashboards.
  • Analyze spending by model, endpoint, and workspace.
  • Receive AI-powered recommendations and forecasts to optimize infrastructure costs.
  • Search and export request logs for debugging and auditing.
  • Integrate applications easily using a lightweight TypeScript SDK.
  • Access analytics and notifications directly through Telegram using OpenClaw.

Instead of simply collecting logs, TokenWatcher transforms AI telemetry into meaningful operational insights.

How we built it

We designed TokenWatcher as a modular, production-ready platform with separate components for telemetry collection, analytics, visualization, and communication.

Backend

  • Node.js
  • Express
  • TypeScript
  • PostgreSQL

Frontend

  • React
  • Vite
  • React Query
  • Tailwind CSS
  • shadcn/ui

Telemetry SDK

  • TypeScript SDK
  • Request batching
  • Retry mechanisms
  • Graceful shutdown flushing
  • Lightweight instrumentation

Infrastructure

  • Server-Sent Events for live updates
  • Workspace-based authentication
  • Secure API key management
  • AI-powered analytics engine
  • Telegram integration through OpenClaw

The SDK captures telemetry directly from AI applications, the backend processes and stores the data, and both the dashboard and Telegram interface present the same real-time insights.

Challenges we ran into

Building an observability platform required solving problems beyond typical CRUD applications.

One of our biggest challenges was designing a telemetry pipeline that remained reliable under varying workloads while minimizing overhead for developers integrating the SDK.

We also built secure multi-workspace authentication, real-time streaming with Server-Sent Events, forecasting based on historical usage data, and conversational analytics through Telegram without compromising security or scalability.

Balancing performance, usability, and maintainability across multiple services was one of the most valuable engineering challenges throughout the project.

Accomplishments that we're proud of

  • Built a complete AI observability platform rather than a proof of concept.
  • Developed a reusable TypeScript SDK for seamless telemetry collection.
  • Implemented real-time dashboards with live analytics and filtering.
  • Added forecasting and AI-generated optimization recommendations.
  • Successfully integrated Telegram for conversational access to analytics.
  • Designed a scalable architecture that supports future providers, integrations, and enterprise features.

What we learned

Developing TokenWatcher deepened our understanding of observability, distributed system design, and developer experience.

We learned how to build reliable telemetry pipelines, design scalable analytics architectures, manage secure multi-tenant systems, and transform raw operational data into insights that developers can immediately act upon.

Most importantly, we learned that great developer tools succeed not by collecting more data, but by making that data understandable and useful.

What's next for TokenWatcher

TokenWatcher is designed to continue evolving beyond this hackathon.

Our roadmap includes:

  • Support for additional AI providers and inference platforms.
  • Advanced anomaly detection for unusual usage and spending patterns.
  • AI-powered optimization recommendations tailored to application workloads.
  • Expanded reporting, dashboards, and export capabilities.
  • Multi-language SDK support beyond TypeScript.
  • Docker and cloud-native deployment improvements.
  • Richer Telegram and conversational AI workflows for infrastructure management.
  • Enterprise collaboration features, including team analytics and role-based access.

Our long-term vision is to make TokenWatcher the observability layer for modern AI applications, enabling developers to understand, optimize, and confidently scale every AI request from development to production.

Built With

Share this project:

Updates