Web-Dashboard-TraceBack
Canvas for remembering
Alerts-Routed Automatically to right channel
Logs channel for tracing events
Lineage/Escalate/Acknowledge
Proposed Architecture

TraceBack

Inspiration

When a container is compromised, a SOC responder has seconds — not minutes. The actual workflow looks nothing like that urgency: an alert fires in one tool, the responder tabs to a dashboard to see what happened, tabs again to a terminal to check the process tree, tabs again to a wiki to see if this has happened before — and by the time the story is stitched together, the attacker has moved on. Every hop is a chance to lose context, lose time, or lose the thread entirely.

We already had the hard part solved: CyberLense, our production eBPF/Tetragon platform, captures kernel-level telemetry — process execution, file access, network connections — as it happens, not after the fact from logs. What it didn't have was a way to act on that telemetry where security teams actually live. Slack's Agent Builder Challenge was the reason to close that gap properly: not a bolt-on webhook that dumps text into a channel, but a real Slack-native agent — with a genuine MCP server underneath — that turns raw kernel truth into a conversation a team can act on without ever leaving Slack.

What it does

TraceBack is a security-observability agent that lives in Slack, grounded in live eBPF kernel telemetry. It does three things, all inside Slack:

Watches, and alerts the right people. A background monitor continuously scans live telemetry for security-relevant signals — sensitive credential file access, SSH brute-force attempts, policy failures. When something fires, TraceBack posts a Block Kit alert card — severity, affected asset, plain-language summary — to exactly the right channel, routed automatically by severity and event type, so critical incidents land with Incident Response and routine noise stays in Tier-1 triage. No channel gets buried; no critical alert gets lost in the noise.
Answers, grounded in truth, never guessing. Ask TraceBack anything — @TraceBack why did this container spawn a shell?, or through its Assistant pane with suggested prompts — and it reasons over live kernel telemetry to answer in plain language, with a confidence score, citing the actual process tree and timestamp behind its conclusion. If the data isn't there, it says so — it never fabricates an explanation. One click opens the same investigation, fully charted, in the companion web portal.
Acts, and remembers. Every alert card carries three actions, right in the message: Acknowledge (updates in place — no duplicate responses when two people respond at once), Escalate (pages on-call and auto-generates a living Incident Canvas — a post-mortem with timeline, root cause, and a remediation checklist that the whole team co-edits), and View Lineage (the real parent-to-child process tree, rendered straight from the kernel event stream). Escalated incidents are recorded to a knowledge base, so the next similar alert says "you've seen this before."

Every action TraceBack takes is mirrored to a dedicated audit channel — it operates as a governed non-human identity, not an opaque bot.

How this aligns with the track — and what we understand winning looks like

Track: New Slack Agent — deliberately, not Organizations, which requires an app already submitted to the Slack Marketplace with an App ID. New Slack Agent asks for exactly what we built: an agent that "automates workflows or surfaces insights," created fresh for this challenge.

The organizers were explicit about what they reward: judges prioritize apps that "solve one specific problem so cleanly", and native Slack UI built with Block Kit is called out directly. That's the design discipline we held to throughout — TraceBack does not try to be a general-purpose chatbot with security bolted on. It does one thing: turn live kernel telemetry into a decision a responder can act on, inside Slack, in one flow. Every screen is a Block Kit card or a Canvas, not a wall of text.

We also built directly against the four stated judging criteria, rather than treating alignment as an afterthought:

Technological Implementation. A genuine, standalone MCP server (FastMCP over Streamable-HTTP) — not a Slack-only shim, directly callable from Claude Desktop or any MCP client. Real eBPF kernel telemetry via Tetragon, not simulated data. Multi-tenant isolation enforced end-to-end, from the JWT claim down to the ClickHouse row policy. Audited against the OWASP Top 10 for LLM Applications, with real gaps found and closed.

Design. Every alert is a Block Kit card, not a wall of text — action buttons update in place so two responders never create a duplicate response. Alerts are routed by severity/type so no channel gets noise outside its scope. The Assistant framework gives a native agent pane with suggested prompts; Canvas turns an escalation into a collaboratively-edited incident record instead of a cold thread.

Potential Impact. Replaces a real, painful workflow — a responder hopping across five tools to investigate one alert — with a single Slack-native flow, provably faster (triage-to-lineage in one click). Multi-tenant by construction: one deployment already serves multiple customers, not a single-org demo dressed up to look general-purpose.

Quality of Idea. TraceBack is framed as a governed non-human identity with a full audit trail, not a generic chatbot wrapper — and its core design discipline is refusing to guess: every answer is grounded and cites its evidence, or it says the data isn't there. That discipline is enforced in code (see Challenges), not just asserted here.

And on the technology tracks specifically: TraceBack authentically hits two of the challenge's primary tracks — a real MCP server integration, and deep use of Slack's AI/Agent Builder surface (Assistant, Block Kit, Canvas) — not one track with the other bolted on for coverage. Every required deliverable is a real, working artifact: a functioning-demo video (live kernel event → live Slack response, not slides or mock data), an architecture diagram, this written description, and sandbox access granted for review.

How we built it

The Slack surface (the star of this build):

Bolt for Python, running in Socket Mode — no public inbound URL required
The Assistant framework with dynamic suggested prompts, for a native "agent pane" experience
Block Kit for every alert card, action row, and threaded reply — not a wall of text
The Canvas API for auto-generated, collaboratively-editable incident post-mortems
A /sentinel slash command and an "Investigate with TraceBack" message shortcut on any message
A conversational front-door that handles greetings and "what can you do?" naturally, without ever answering off-topic questions — it stays in its lane

The MCP layer (a real, standalone server, not Slack glue code): A FastMCP server over Streamable-HTTP — sentinelops-mcp — exposing tools like explain_event, get_process_lineage, query_recent_anomalies, and search_past_incidents. This isn't a Slack-only shim: the same server is directly usable from Claude Desktop or any MCP client. Every tool result is redacted for credentials before it ever leaves the server.

The reasoning + data layer: The MCP server wraps Arch-D, our existing FastAPI reasoning engine — 73 tools, a plan → dispatch → synthesize pipeline, JWT-based multi-tenant isolation enforced down to the ClickHouse row level. Telemetry is captured by Tetragon (eBPF), shipped through an OTel Collector into ClickHouse; incidents and tenant data live in MongoDB; Redis carries the alert pub/sub with dedup so a recurring alert threads instead of spamming the channel.

Deployment: the whole stack — ClickHouse, MongoDB, Redis, the reasoning engine, the portal, the MCP server, and the Slack app — runs as one docker-compose stack on a single AWS EC2 instance. Onboarding a new monitored host is three CLI commands: install, log in, deploy.

Challenges we ran into

Making "never guess" actually true. It's easy to say an agent is "grounded" — harder to guarantee it. We audited every response path against the OWASP Top 10 for LLM Applications and closed two real gaps: an unbounded input path (fixed with a length cap and sanitizer) and a block message that was misleadingly vague about why a request was refused. When telemetry genuinely isn't there, TraceBack says "I couldn't reach the telemetry source" — it doesn't paper over the gap with a plausible-sounding guess.

Two real kernel-policy crashes. Early in the build, Tetragon crash-looped on the monitored host — twice, for two different real bugs (a mismatched matchNamespaces value, and a kprobe policy missing a returnArg it declared). Neither was mock-data-safe; both had to be root-caused against live logs.

A dead alert pipeline, found live. During final testing, no alert was reaching Slack — not because detection failed, but because the background task subscribing to Redis was silently garbage-collected on startup (a bare asyncio.create_task() with no held reference), and its cleanup path threw on top of that. PUBLISH was reporting zero subscribers. We only found it because we insisted on testing the real pipeline end-to-end rather than trusting the code path looked right — exactly the discipline the OWASP audit taught us to apply everywhere else, too.

A quieter bug: the agent second-guessing real questions. A "conversational front-door," added so TraceBack would answer "what can you do?" warmly instead of refusing it as off-topic, turned out to have a substring-matching bug: "what are you" is literally a substring of "what are your," so a genuine question like "what are your top memory consumers?" was being swallowed into a canned capabilities reply instead of a real answer. Caught and fixed with word-boundary matching and a 21-case regression test before it could embarrass us live.

Getting a reviewer sandbox actually reachable. Even the mundane parts fought back: our Slack developer sandbox shipped pre-loaded with 7 placeholder users that silently filled the 8-user cap, and deactivating them at the workspace level didn't release the seats — Slack tracks that cap at the organization level. Small thing, but it would have quietly blocked judges from ever reaching our workspace if we hadn't chased it down.

Accomplishments that we're proud of

A demo that runs on real, live eBPF telemetry — the video shows an actual kernel event, captured by a real detection rule, triggering a real alert. Nothing is mocked.
73/73 tool parity verified between the Slack agent's reasoning path and our existing production AI chat — TraceBack isn't a stripped-down surface, it's the full capability, natively in Slack.
A genuine MCP server that stands on its own outside Slack, wrapping a production telemetry engine — not a demo-only endpoint.
Multi-tenant isolation enforced end-to-end, from the JWT claim down to the ClickHouse row policy — proven, not assumed.
A security posture we can actually defend: OWASP LLM Top 10-audited, credential-redacted by default, every agent action logged to an audit channel.

What we learned

That the gap between "the backend already does this" and "this works natively in Slack" is bigger than it looks — Block Kit's constraints, Socket Mode's lifecycle, and the Assistant framework's event model all shape how an agent should actually behave, not just how it's wired up. And that the bugs worth finding are rarely in the code you're staring at — they're in the pipeline you assumed was fine because it looked right, which is exactly why we kept testing the real, live path end-to-end instead of trusting it in isolation.

What's next for TraceBack

Self-service Slack OAuth onboarding — today, connecting a new tenant's telemetry to Slack is a guided CLI + .env step; a full OAuth install flow would make it a single click.
A knowledge-graph view of incidents and assets, surfaced directly in the portal and referenceable from Slack.
Semantic-drift incident search — moving past exact-match incident recall toward "this smells like an incident we saw three weeks ago," even if the wording is different.
The Real-Time Search API, to enrich an investigation with "has this asset or CVE been discussed anywhere recently?"
Expanding the App Home into a live posture dashboard: open incidents, last-hour anomalies, and per-squad routing configuration, all at a glance.

TraceBack · Slack Agent Builder Challenge 2026