TokenWatcher

AI observability platform that tracks LLM usage, costs, latency, and performance in real time with dashboards, forecasting, alerts, and a production-ready telemetry SDK.

Comment

Inspiration

The rapid adoption of AI has made it easier than ever to build intelligent applications, but understanding what happens behind every API call remains difficult. Developers often know their monthly AI bill long before they understand which models, endpoints, or users are responsible for those costs. Existing monitoring solutions are either too expensive, too complex, or don't provide actionable insights tailored for AI workloads.

We built TokenWatcher to give developers complete visibility into their AI applications—from every request and token consumed to latency, costs, forecasts, and optimization recommendations. Our goal was to make AI observability simple, accessible, and production-ready for teams of any size.

What it does

TokenWatcher is an open-source AI observability and cost monitoring platform for LLM-powered applications.

It provides:

  • 📊 Real-time dashboards for AI usage and costs
  • 📈 Cost forecasting and budget analytics
  • ⚡ Live request monitoring through Server-Sent Events
  • 🔍 Searchable request logs with filtering and exports
  • 📦 A lightweight TypeScript SDK for telemetry collection
  • 🔐 Multi-workspace authentication and API key management
  • 🤖 AI-powered insights and recommendations
  • 📱 Telegram integration for reports and notifications via OpenClaw

Instead of only showing infrastructure metrics, TokenWatcher helps developers understand how their AI products behave, where money is being spent, and what should be optimized next.

How we built it

TokenWatcher was designed as a modular full-stack platform.

Backend

  • Node.js
  • Express
  • TypeScript
  • PostgreSQL

Frontend

  • React
  • Vite
  • React Query
  • Tailwind CSS
  • shadcn/ui

SDK

  • Lightweight TypeScript SDK with batching, retries, graceful shutdown, and telemetry instrumentation

Infrastructure

  • Server-Sent Events for real-time updates
  • Workspace-isolated architecture
  • Secure authentication and API key management
  • Telegram integration through OpenClaw
  • AI-powered analytics and forecasting

The SDK captures telemetry directly from AI applications, the backend securely stores and processes the data, and the dashboard and Telegram bot present actionable insights in real time.

Challenges we ran into

Building a production-ready telemetry platform introduced several engineering challenges.

One of the biggest challenges was designing an SDK that could reliably batch requests, retry failed uploads, and gracefully flush pending telemetry without impacting application performance.

We also had to build secure workspace isolation, real-time analytics using Server-Sent Events, and a scalable data model capable of powering dashboards, reports, forecasts, and AI-generated recommendations from a single telemetry source.

Another challenge was integrating conversational access through Telegram while maintaining security and keeping OpenClaw stateless.

Accomplishments that we're proud of

  • Built a complete end-to-end AI observability platform instead of a prototype.
  • Designed a reusable telemetry SDK for easy integration into AI applications.
  • Implemented real-time dashboards with live analytics.
  • Added forecasting and AI-powered optimization recommendations.
  • Successfully integrated Telegram to make AI analytics accessible through natural conversations.
  • Created a modular architecture that can continue growing with new providers and integrations.

What we learned

Building TokenWatcher reinforced that observability is just as important as AI model quality.

We learned how to design reliable telemetry pipelines, build scalable analytics systems, structure production-ready TypeScript applications, and create secure multi-workspace architectures. We also gained valuable experience balancing backend performance, frontend usability, and developer experience while building an end-to-end platform.

What's next for TokenWatcher

Our vision is to make TokenWatcher the observability layer for modern AI applications.

Next, we plan to:

  • Support additional AI providers and inference platforms.
  • Expand anomaly detection and predictive analytics.
  • Introduce richer cost optimization recommendations.
  • Add deeper enterprise reporting and export capabilities.
  • Improve deployment with Docker and cloud-native tooling.
  • Enhance the developer SDK with support for more programming languages.
  • Continue expanding Telegram and conversational AI capabilities for managing AI infrastructure.

Ultimately, we want TokenWatcher to become a unified platform where developers can monitor, optimize, and understand every AI request from development to production.

Share this project:

Updates