Discord interface
Agent chat interface
Diagnostics Assistant
Incidents and Symptoms
Cloud cost autopsy

FixProof: The tool we wished we had during our internship

Inspiration

During our internship, we kept coming back to the same problem: when a workload starts failing, the hardest part is knowing what to trust. A health check, a rollout, a log line, and a metric can each tell a different story. At 3:00 AM, an on-call engineer needs evidence they can trace back to the system, not a confident answer from a chatbot.

We felt the need for an assistant that could do the time-consuming investigation without taking control of production. That became the rule behind FixProof: AI can investigate, but it cannot decide on its own to change infrastructure.

What it does

FixProof guides an incident from report to verified recovery. A worker collects bounded observations from the workload, Kubernetes, logs, probes, and Prometheus. The AI compares those observations and cites the retained evidence behind its findings.

If a restore is appropriate, FixProof prepares an exact change against a trusted baseline. An operator reviews and approves it before a scoped execution path can apply it. FixProof then checks fresh traffic and health measurements. A successful patch is recorded as a patch; recovery is claimed only when the measurements support it.

We also added a console for following the investigation, an append-only audit trail, and Discord updates that show incident status and safe evidence summaries.

How we built it

We split the system into services with different responsibilities. The MCP service gathers scoped telemetry. The AI service analyzes retained evidence and has no Kubernetes write credentials. The API owns incident state, approvals, execution records, and the audit trail. The React console gives operators a place to inspect the evidence and make decisions.

That separation mattered to us. It meant an AI finding could inform a human decision, but could not silently become a cluster mutation.

Challenges we ran into

Our first challenge was making citations trustworthy. An AI model can produce a plausible explanation with an invented evidence ID, so FixProof checks citations against the evidence actually retained for that incident.

Logs were another concern. They are useful during an outage, but they can also contain misleading instructions or sensitive text. We treat them as bounded, untrusted observations rather than instructions for the AI or content to post into Discord.

We also learned to distinguish a completed action from a recovered service. A rollout can succeed while users still receive errors. That is why recovery has its own verification step using fresh probes and traffic measurements.

What we’re proud of

We’re proud that an operator can move from a finding to its supporting observation, see what was approved, and inspect what happened afterward. The AI helps connect the evidence; the audit trail preserves the decisions.

The Discord integration follows the same approach. Its channel cards show state and safe evidence summaries, while the slash commands are read-only. Approvals and fixes stay in the guarded operator workflow.

What we learned

Building FixProof changed how we thought about “autonomous” operations. The useful part of autonomy is collecting, organizing, and explaining evidence quickly. Trust comes from showing the limits of that evidence, requiring approval for a change, and measuring whether the change helped.

What’s next

We built FixProof around a Kubernetes workload, but the investigation pattern applies elsewhere. We want to add evidence providers for managed cloud services, infrastructure drift, databases, and distributed traces. We also want alerts to start a bounded investigation automatically, so an engineer opens the console with a clear record of what was observed and what still needs checking.