LogTrace©

LogTrace© — AI-powered cascade failure diagnosis for microservices Plug in any log source. Get the root cause in 90 seconds.

Comment

LogTrace — Hackathon Story

NVIDIA × Nebius Global AI Hackathon · Best Apps and Agents Track

Inspiration

It's 2am. Your payment service is down. You have five services, hundreds of error lines, and somewhere in that wall of text is the one event that started it all.

I've been in that situation. The real cost of a microservice outage isn't the outage itself — it's the mean time to understanding. Engineers spend two to four hours per incident manually correlating timestamps, chasing trace IDs across log files from five different services, trying to reconstruct the failure sequence in their heads.

That's not an AI problem that needs a chatbot. It's a pipeline problem — one that needs ingestion, correlation, reasoning, and structured output, running automatically the moment a spike fires. That's what I set out to build.

What it does

LogTrace is an AI-powered log intelligence system for microservice environments. It ingests raw log streams, detects error spikes in real time, and runs a two-stage NVIDIA model pipeline to produce a structured incident report — root cause, cascade chain, confidence score, fix suggestions — with no human in the loop.

  • Ingest log events via HTTP push into Redis Streams (p99 < 50 ms)
  • Detect error spikes automatically with a stream processor using XREADGROUP
  • Correlate events by trace_id and infer the service topology (Stage 1)
  • Batch-classify every event with Nemotron Nano 8B — event_role, error_category, anomaly score $s \in [0, 1]$ (Stage 2)
  • Run multi-turn root cause analysis with Nemotron Ultra 253B, streamed token-by-token via SSE (Stage 2)
  • Fetch targeted fix suggestions from Tavily in parallel
  • Deliver the full incident report — root cause service, cascade chain, confidence score $\in [0, 100]$, summary — to the portal in real time

Two flows are supported: live auto-trigger (stream processor fires on spike) and file upload (drag-and-drop your own logs). Both run the same pipeline and share no state.

How we built it

Solo, over ~41 calendar days.

I wrote a full HLD before writing any code — 11 sections, 8 revisions, ADRs, time estimates. Every decision about Redis Streams vs Redis List, S3 vs raw file buffer, SSE vs polling, was made once and not revisited.

  • FastAPI — async API, hot path to Redis in < 50 ms
  • Redis Streams with XREADGROUP — durable, back-pressure-safe ingest
  • PostgreSQL + SQLAlchemy async — 7 ORM models, Alembic migrations
  • Nebius Object Storage (S3-compatible via boto3) — log file storage for the upload path
  • Nebius Token Factory — LLM serving via OpenAI-compatible client; one base_url change, no SDK swap
  • Docker Compose + nginx — single ./scripts/start.sh deploy on a Nebius VM
  • Vanilla JS SPA served directly by FastAPI — no separate frontend deployment
  • Server-Sent Events — Ultra streams its RCA word by word to the browser

The two-model split was a deliberate architectural choice:

Model Role Why
Nemotron Nano 8B Per-event classification Fast, cheap, parallelisable — 20 events per call, 10 concurrent
Nemotron Ultra 253B Root cause reasoning 253B parameters on the hardest part only

Ultra receives only the filtered, classified events — not the raw log dump — so it reasons over signal, not noise.

Challenges we ran into

  • SSE through nginx — nginx buffers SSE silently. Required proxy_buffering off, X-Accel-Buffering: no, proxy_http_version 1.1, and proxy_read_timeout 600s. Missing any one of these stalls the stream with no error.
  • SSE authentication — Browser EventSource doesn't support custom headers. Built a second auth dependency that accepts the API key as either X-API-Key header or ?api_key= query parameter, with the same SHA-256 hash + Redis cache logic.
  • Debounce state on demo reset — The spike detector's in-memory debounce remembered the old spike after a reset, blocking the next demo run. Fixed by publishing to a stream_processor:reset Redis key; the processor watches it and clears state — decoupled, no shared memory.
  • Ultra response parsing — Ultra sometimes truncates JSON on long responses. The parser uses two regexes: one for a well-formed <ANALYSIS_JSON>...</ANALYSIS_JSON> block, and a fallback for an open tag with no closing tag. If both fail, the raw text is stored and the incident still delivers.
  • Choosing what not to build — The original HLD included Fluentd/Logstash connectors, a topology graph, an interactive Q&A panel, and a CLI. Cut everything not on the critical path. Every cut was a logged decision, not an accident.

Accomplishments that we're proud of

  • End-to-end pipeline in < 41 days, solo — from HLD to deployed VM with a working demo
  • Sub-50 ms ingest hot path — Redis Streams with XREADGROUP handles bursts without dropping events
  • Two-model pipeline that actually works — Nano classifies noise out; Ultra reasons over signal. The incident reports are coherent, structured, and actionable
  • Live streaming UX — watching Ultra write a root cause analysis word by word turns a two-minute wait into something that feels intelligent
  • Zero-config demo — one button resets all state across Postgres, Redis, and the stream processor; the next incident fires automatically
  • Graceful degradation everywhere — Ultra parse failure, Tavily timeout, S3 error — each fails independently and the report still arrives

What we learned

  • Streaming changes everything about UX. A two-minute wait for a static result feels broken. A two-minute wait where you watch the model think, word by word, feels like watching something intelligent work.
  • Two models beat one. Running Ultra on every event would be slow and expensive. Running Nano on every event and Ultra only on the filtered summary is fast, cheap, and better — because Ultra gets context, not chaos.
  • Redis Streams are the right primitive for this. XREADGROUP gives consumer groups, message acknowledgment, and replay on failure. A Redis List would have lost messages on processor crash.
  • Prompts are code. The Nano classification prompt went through seven iterations before it stopped hallucinating event_role values outside the allowed enum. The Ultra prompt required explicit JSON schema enforcement with examples to get consistent structured output.
  • HLD before code pays off. Writing the architecture document first forced every trade-off to be made explicitly. The architecture didn't drift during implementation because the decisions were already made.

What's next for LogTrace©

  • Connector ecosystem — Fluentd, Logstash, and Kafka adapters so LogTrace plugs into existing observability stacks without changing producers
  • Service topology graph — visual cascade chain rendered from the cascade_chain[] field Ultra already produces
  • Interactive Q&A panel — follow-up questions to Ultra about an incident, grounded in the classified event timeline
  • Anomaly score trending — track $s \in [0,1]$ per service over time to surface degradation before it becomes a spike
  • Multi-tenant API keys — per-team key management with usage quotas and audit log
  • Slack / PagerDuty integration — push the incident summary the moment Ultra finishes, not just to the portal

Built With

Share this project:

Updates

Submission history