Inspiration
Running a local model with Ollama is easy. Knowing what it cost is not. Token counts, latency, time to first token, and host pressure (CPU, memory, GPU) live in different places, and most observability tools want the prompt and the completion in order to be useful. That tradeoff is wrong for an enterprise workload. The prompt is the customer's data.
OMNIX AI Analytic Tower (AAT) is a monitoring-first control tower for one organization, one observer, and one Ollama workload. It records execution metadata and host metrics, then uploads them asynchronously. Inference keeps running if the observer, the buffer, or the API is unavailable.
What I learned
A useful AI monitor is an allow-list, not a log shipper. The event that is safe to store is small: event id, agent, environment, application, provider, model, token counts, timings, status, a sanitized error code, and a host sample. Anything else, including prompt text, completion text, system instructions, credentials, and request bodies, is rejected and never stored.
I also learned to label estimates honestly. A host GPU sample that overlaps a request window is evidence of resource pressure, not proof that this request owned the GPU. The hardware view says that in the product. GPU numbers appear only when nvidia-smi succeeds.
Telemetry has to be best-effort. The observer keeps a bounded queue (default capacity 1000). When it is full, the oldest event is dropped and a droppedEvents counter is exposed. Uploads retry on network failures, HTTP 429, and HTTP 5xx with exponential backoff and jitter. Other 4xx responses are discarded. The sample workload still succeeds when Ollama succeeds and the observer is down.
Throughput is derived, not guessed:
$$ \text{tokens per second} = \frac{\text{completion tokens}}{\text{total latency in seconds}} $$
How I built it
The system is a modular monolith. One ASP.NET Core API on .NET 10 owns enrollment, heartbeats, batch ingestion, queries, and retention. Business rules live in application services. PostgreSQL is accessed through EF Core and Npgsql, with migrations applied on startup in Development. The Angular 20 dashboard, styled with Tailwind CSS, is the control surface: sign-in, overview, agents, execution history, model analytics, hardware, and an audit log.
The observer is a local .NET worker. It enrolls with a one-time token, then authenticates with an agent key that is shown once and stored only as a SHA-256 hash. It accepts metadata on loopback (127.0.0.1:47321), samples the host, heartbeats every 30 seconds, and is treated as online when the last heartbeat is within 90 seconds. Dashboard passwords use PBKDF2. Tenant checks are enforced on every tenant-owned query. Event ids make ingestion idempotent, and timestamps are stored in UTC.
A small Ollama sample records one call. It sends model, token counts, and timings. It does not forward the prompt or the completion. Docker Compose runs PostgreSQL for local development and can bring up the API and dashboard together.
Optimization, approval, and deployment are intentionally out of this release. Monitoring has to be trustworthy before any later change is proposed, signed, validated, and rolled out by a person.
Challenges
The hard part was keeping the customer's LLM request off the monitoring path. A failed upload, a full queue, or a down API must not fail inference, and a retry must not turn one execution into two. Idempotent event ids, a bounded buffer, and a privacy allow-list on ingest were the answer.
The second hard part was tenant and agent trust. An organization id in a payload is not proof of tenancy. Machine endpoints authenticate the enrolled agent. Dashboard endpoints authenticate the user and authorize the tenant. Secrets stay in environment variables or user secrets, never in the repo.
The third was saying less than the charts tempt you to say. Correlating a host sample with a request is an estimate. Shipping that label in the API, instead of implying per-request GPU ownership, took more discipline than collecting the sample.
Built With
- .net
- angular.js
- api
- asp.net
- c#
- compose
- core
- css
- docker
- entity-framework
- framework
- jwt
- npgsql
- ollama
- openapi
- postgresql
- rest
- rxjs
- tailwind
- typescript