TokenWatcher – AI Cost & Observability Platform
The Problem
As AI applications move into production, developers often know how much they're spending only after receiving their monthly invoice. They lack visibility into which models, endpoints, users, or features are driving costs, increasing latency, or generating failures.
Existing dashboards expose raw API metrics, but they rarely provide actionable insights that help engineering teams understand, optimize, and control AI infrastructure.
Our Solution
TokenWatcher is an AI observability platform that helps developers monitor, analyze, and optimize production AI applications.
It provides a unified dashboard for AI telemetry across multiple providers, allowing teams to:
- Monitor token usage and API costs
- Track latency and request failures
- Analyze model and endpoint performance
- Forecast future AI spending
- Detect anomalies
- Receive AI-powered optimization recommendations
Instead of manually combining logs, spreadsheets, and invoices, developers get real-time visibility into how their AI systems behave.
Key Features
- Real-time AI telemetry dashboard
- Multi-workspace support
- Token, cost, latency, and request analytics
- Model and endpoint performance dashboards
- Cost forecasting
- AI-powered insights and recommendations
- SDK for telemetry instrumentation
- Secure API key management
- Telegram integration through OpenClaw
- Live updates using Server-Sent Events (SSE)
Built with GPT-5.6 and Codex
For OpenAI Build Week, GPT-5.6 and Codex became an integral part of our development workflow.
We used Codex extensively to accelerate implementation by:
- Debugging backend services
- Refactoring TypeScript modules
- Improving project architecture
- Implementing new dashboard functionality
- Resolving integration issues
- Generating and refining documentation
- Speeding up testing and development iterations
GPT-5.6 also helped with architecture decisions, implementation planning, debugging strategies, documentation, and feature refinement throughout development.
Rather than simply generating code, Codex acted as a collaborative engineering assistant that significantly accelerated the development of a production-ready platform.
Technical Architecture
TokenWatcher consists of four major components:
- TypeScript SDK for telemetry collection
- Express + PostgreSQL backend
- React analytics dashboard
- OpenClaw-powered Telegram interface
Telemetry collected by the SDK is securely ingested by the backend, processed into analytics, and streamed to the dashboard in real time. The same analytics layer powers forecasting, reports, recommendations, and conversational Telegram interactions.
Challenges
Building an observability platform required balancing performance with flexibility.
Some of the biggest challenges included:
- Designing a scalable telemetry ingestion pipeline
- Handling workspace isolation securely
- Maintaining real-time dashboard updates
- Building reliable analytics and forecasting
- Integrating Telegram workflows without compromising security
What We Learned
Building TokenWatcher reinforced how quickly AI infrastructure complexity grows in production.
Observability is no longer optional—teams need visibility into token usage, costs, latency, and reliability from day one. Combining traditional analytics with AI-powered recommendations creates a much better developer experience than static dashboards alone.
What's Next
Our roadmap includes:
- Support for additional LLM providers
- Advanced anomaly detection
- Deeper forecasting models
- More conversational AI workflows
- Expanded reporting and export capabilities
- Enterprise deployment improvements
Log in or sign up for Devpost to join the conversation.