Fireline

Fireline is the first Agentic AI incident router for DevOps: it streams alerts into Redpanda, auto-triages them, and fires focused Slack alerts instead of alert storms.

Comment

posted an update —

Post-Mortem: Tracing Implementation

What Broke

During development, we initially attempted to use the Akka Console for distributed tracing visualization. The Akka SDK provides built-in OpenTelemetry support, and we configured tracing export to what we thought was the Akka Console endpoint.

Issue: The Akka Console tracing endpoint couldn't be reached or wasn't properly configured for our local development setup. Traces were being generated but weren't visible in any UI, making it difficult to debug the incident processing pipeline.

Symptoms:

  • Tracing code executed without errors
  • Spans were created (verified via logs)
  • No traces appeared in Akka Console UI
  • Connection errors when trying to export traces to Akka Console endpoint

How We Fixed It

We switched to Jaeger for trace visualization, which is a well-established, production-ready distributed tracing system.

Solution:

  1. Added Jaeger to docker-compose.yml as a service
  2. Configured OpenTelemetry export to Jaeger's OTLP endpoint (http://localhost:4317)
  3. Updated backend/run.sh to set tracing environment variables pointing to Jaeger
  4. Verified traces appear correctly in Jaeger UI at http://localhost:16686

Why This Worked:

  • Jaeger is a standard OpenTelemetry-compatible backend
  • Easy to run locally via Docker
  • Provides excellent trace visualization and querying
  • Better suited for demonstrating observability to judges

Lessons Learned

  1. Use battle-tested tools for demos: Jaeger is more reliable and well-documented than trying to configure custom tracing endpoints
  2. Test observability early: We should have verified trace visibility sooner in development
  3. Docker Compose simplifies setup: Having Jaeger in docker-compose makes it easy to start/stop and share with the team

Current State

✅ Traces are working correctly in Jaeger
✅ End-to-end traces show: process-support-event → ingest-event → triage-incident → generate-comms
✅ Span attributes include: event.id, incident.id, severity, owner
✅ Error spans are properly recorded with exception details

The tracing implementation now provides excellent observability for debugging and demonstrating the incident processing pipeline.

Log in or sign up for Devpost to join the conversation.