ResQ

• 

  ResQ Architecture

Inspiration

When a production service goes down, engineers look at logs, metrics, and code to figure out what happened. This projects aims to see if a team of AI agents could do the same thing, each one focusing on a different data source, and then working together to reach a diagnosis.

What it does

ResQ has five AI agents that investigate production incidents:

1. Log Analyzer - reads error logs, extracts stack traces, looks at source code around the error
2. Metric Monitor - analyzes CPU, memory, latency, error rates, cache hit rates
3. Coordinator - takes the hypotheses from both agents and determines the root cause
4. Runbook Executor - generates a remediation plan
5. Post-Mortem Writer - writes up the incident report

When the Log Analyzer and Metric Monitor disagree on the root cause, they go through a negotiation round. Each agent sees the other’s evidence and reconsider whether they’re looking at the actual cause or just a symptom. After they revise their positions, the Coordinator arbitrates.

There’s a live terminal UI (built with Textual) so you can watch the investigation happen in real time.

How we built it

All agent reasoning runs on Qwen Cloud (qwen-plus) through the DashScope API. Agents return structured JSON with hypotheses, confidence scores, and evidence.

Inter-agent communication goes through an async MessageBus. The negotiation round works by appending a prompt that asks each agent to re-examine the incident with the peer’s evidence in mind.

The ConsensusEngine scores hypotheses based on confidence, evidence strength, and a bonus when both agents agree.

The target service runs on Alibaba Cloud ECS and integrates with SLS (log shipping/query), OSS (report storage), ECS (infrastructure context), and CMS (host metrics) using the official SDKs. Infrastructure is provisioned with Terraform.

Challenges we ran into

My first multi-agent version actually did worse than a single agent. On a tricky incident, the agents disagreed and the coordinator just picked the most confident one. which was wrong. I fixed it by adding a negotiation step where each agent sees the other’s evidence and reconsiders.

To decide if the agents disagreed, the code compared their answers by checking if they shared keywords. But words like “requests” and “query” show up in almost every diagnosis. So when one agent said “memory leak” and the other said “cache failure,” the system saw they both mentioned “requests” and assumed they agreed, even though they were completely different diagnoses.

The fix was to just ask the AI directly: “Here are two hypotheses, do they describe the same root cause or different ones?” The model understands that two answers can mention the same symptoms but point at completely different causes.

Accomplishments that we're proud of

The live terminal UI. Being able to watch five agents investigate an incident in real time, see the negotiation happen, see them revise their positions and makes the whole system tangible in a way that a backend API call wouldn’t.

What we learned

Specialization without collaboration can actually hurt accuracy. During the negotiation round, letting agents challenge each other’s conclusions is what turns multiple agents into a measurable improvement over a single one.

What's next for ResQ

1. Having agents actually execute remediation on the target service instead of just recommending steps.
2. A web-based dashboard for teams who want a visual interface but the terminal UI will stay as the primary mode.
3. Giving agents memory across incidents. If the system investigated a similar outage last week, it should remember what worked and use that to speed up the next one.

Built With

• alibaba
• docker
• flask
• python
• qwen
• sqlite
• terraform
• textual