AG-QREW on Qwen

Five Qwen agents run a full QA sprint on any app you point them at (URL, requirements, API spec): they split the work, dispute each other's bugs, and a lead adjudicates before signing off.

Comment

About the project

Inspiration

We were browsing the Global AI Hackathon Series with Qwen Cloud when we hit Track 3: Agent Society. The brief asked for multiple agents with distinct capabilities that divide work, negotiate, and resolve disagreements, and it asked us to prove the society beats a single agent. That last line is the hard one. It is easy to make one model look busy. It is much harder to show a team of them is genuinely better.

So we asked: what job is a real team sport, full of judgment calls, where people argue about severity and root cause every day? QA. So we built an autonomous QA team.

What it does

AG-QREW takes a requirements document and a running app, asks for one human approval on its test plan, and then a society of five Qwen agents runs the rest of the QA sprint on its own: test cases, executed Playwright scripts, an API contract battery, filed bugs, and a signed verdict.

The five agents each have a distinct role, model tier, and tool set:

  • qa-lead (qwen-max): writes the test plan, adjudicates disputes, and writes the final sign-off.
  • qa-tc-writer (qwen-plus): turns the approved plan into structured test cases stored in SQLite.
  • qa-api-tester (qwen-plus): reads the OpenAPI spec, exercises every endpoint, and files contract bugs. It reads the response body, not just the status code.
  • qa-script-writer (qwen-plus): converts cases into runnable Playwright scripts and executes them.
  • qa-hawk (qwen-plus, plus qwen-vl-max for vision): does the environment check, smoke, and exploratory testing, and calls a vision model to read the screen.

The whole run streams live to a React dashboard: a nine-phase progress bar, a live signal feed, a filterable test-case browser, a bug list with dispute badges, and the final sign-off report. A Configure tab lets you point it at your own target. Give it a URL, a requirements document, and an OpenAPI spec, and it detects what those inputs unlock and runs only the phases they honestly support.

How we built it

We rebuilt a multi-agent runtime from scratch on the Qwen function-calling API instead of using a framework. The core pieces:

  • AgentLoop: one reusable engine (chat, tool calls, results, repeat) with per-agent iteration and token budgets, a loop guard against repeated calls, and a BLOCKED path instead of crashing.
  • A deterministic orchestrator: it runs the agents in a fixed, dependency-ordered pipeline of nine phases (env, plan, approval, cases, scripts, explore, api, adjudicate, sign-off). Agents never call each other. They coordinate through a shared signal bus and a shared SQLite store.
  • Conflict resolution: three agents can formally dispute another agent's bug. The lead runs a one-round debate and rules with one of four verdicts: upheld, downgraded, reclassified, or rejected.
  • A deterministic verdict: the final pass, conditional pass, or fail is computed in code from the bug set and open disputes, not decided by a model.
  • Qwen on Alibaba Cloud Model Studio: all calls go through the DashScope International OpenAI-compatible endpoint with retry and backoff tuned for the free-tier rate window.

The moment that mattered

On one run, qa-hawk deleted a task, saw it still on the page, and filed a data-integrity bug. qa-api-tester challenged it with evidence: DELETE returned 200 and a fresh GET returned the correct list, so the data layer was fine and the defect was the UI not refreshing. The lead gave qa-hawk one rebuttal, weighed both sides, and reclassified the finding from a data bug to a UI-refresh bug. Two agents disagreed with real evidence, and a third settled it correctly. That is the entire point of the track, and it emerged on its own.

Accomplishments we are proud of

On the demo app, which has four bugs planted on purpose, the society found all four in a single run, each caught by the agent designed for it, plus one extra finding. It raised and adjudicated the designed dispute live, and then signed off with a FAIL. The FAIL is the correct answer. A QA team that stamps a broken app as PASS is useless.

Efficiency vs a single agent

We ran the identical job with one monolithic agent as our baseline. Both modes write metrics to the same file for a head-to-head table. The solo agent is cheaper and faster, but it misses the bugs that need cross-layer evidence, and it structurally cannot dispute anything: it has no second agent to disagree with and no dispute tool. Conflict resolution is a property of the society, not of any one model.

Challenges we ran into

Running two agents that share one model bucket in parallel tripped the free-tier per-minute rate window, so we made the pipeline serial and let the signal bus keep coordination order-independent. We tuned retry and backoff specifically for 429s. We added context compaction so no agent blows its token budget on the re-sent conversation. And we ground the script-writer's selectors in a real DOM probe so it does not hallucinate them.

What we learned

Autonomy is not one model doing everything. Sometimes it is a few narrow specialists, one honest referee, and the discipline to say "not yet." The disagreement between agents made the final report more correct, not less.

What's next

Pluggable adapters for TestRail, Jira, and Postman, richer oracles for arbitrary production apps, and better selector generation for complex single-page apps.

Built by Mamdud Hasan and me for the Global AI Hackathon Series with Qwen Cloud, on Alibaba Cloud Model Studio.

Built With

Share this project:

Updates

Submission history