open "/Users/cindy_tong/Desktop/项目/Qwen Hackathon/open-world-game-analysis-agent/docs/PROJECT_STORY.md"

How We Built the Open-World Game Analysis Agent

The idea

We wanted to answer a simple question: if you're a game publisher deciding whether to invest in an open-world title, can you reverse-engineer what players actually care about from public Steam reviews?

Not vibes. Not "sentiment analysis says 73% positive." We wanted the chain:

design choice → player reaction → churn risk

Traceable. Quantifiable. Comparable across competitors.

That's it. That's the pitch.

The naive version (and why it failed)

First instinct: grab a few thousand Steam reviews, throw them at Qwen, ask for a summary.

This doesn't work, for two reasons:

Context window blows up. A popular open-world game easily has 10k+ reviews. Even with a million-token window, you're stuffing in noise — memes, "10/10 would crash again", copy-paste refund rants. The model can't meaningfully process all of it at once.
You can't trace the conclusion. If the model says "players dislike the driving mechanics," which reviews said that? How many? Were they 500-hour veterans or 2-hour refunders? You don't know. The publisher can't act on it.

So we needed a different architecture.

The methodology: atomic claims

We landed on a three-layer separation:

Layer	Who	Does what
Extraction	Qwen	Splits each review into structured "atomic claims"
Aggregation	Deterministic code	Counts, scores, cross-tabulates the full dataset
Interpretation	Qwen (agent)	Reads condensed metrics, writes the publishing report

An atomic claim is the smallest unit of player opinion:

one game experience object + one clear judgement + one sentiment direction

"The city looks amazing but the driving is floaty and missions get repetitive" becomes three claims, not one. Each tagged with aspect, topic, polarity, severity, confidence, and player role.

Qwen extracts evidence. Code does math. The agent explains. Nobody skips a layer.

We wrote a full extraction SOP and scoring SOP to keep things reproducible — these aren't just prompts, they're the rules the entire pipeline enforces.

The pipeline: 8 steps

fetch reviews → extract claims → quality check → aggregate → 
dimension analysis (×17, concurrent) → review → report → render

We used LangGraph for orchestration. Step 5 fans out into 17 parallel dimension analyses (combat, open_world, narrative, driving, etc.) and gathers results — this was the only way to keep wall-clock time reasonable when each dimension needs its own Qwen call.

Quality gates run at two levels:

Code gates: schema validation, enum checks, evidence quote verification
Statistical gates: if one aspect dominates > 65% of all claims, something's wrong upstream — the extraction model probably collapsed its taxonomy

When a gate fires, we don't just log it. We re-run the extraction on flagged reviews and re-check.

The bugs (there were many)

Qwen wrapping arrays in strings

The extraction prompt asked for a JSON array of claims. Qwen would sometimes return:

{"claims": "[{\"aspect\": \"combat\", ...}]"}

...a string containing JSON, instead of actual JSON. Took a few rounds of prompt iteration to kill this reliably.

The million-character wall

Qwen's input limit is 1,000,000 characters per call. Our topic_graph had 1,442 rows for a single game — mostly long-tail topics with sample_size = 1. The competitive matrix prompt concatenates two games' worth. Boom, 400 error.

Fix: filter topics where $\text{sample_size} < 2$ before they enter the graph data. These are insufficient_sample status anyway — the frontend doesn't show them, but we were still paying the token cost.

One bad game kills everything

Early version: if Qwen fails on one game's extraction, the entire multi-game run crashes. A publisher submitting 3 competitors loses all results because game #2 had a weird review.

Fix: fault isolation. Each game's pipeline runs independently; failures are contained and reported, not propagated.

API rate limits

Qwen's DashScope API has concurrency limits. 17 parallel dimension analyses + claim extraction = 429s everywhere. We added a global concurrency cap and retry backoff.

Storage layer chaos

SQLite deadlocks on concurrent writes. Corrupt JSON files from interrupted processes. FileResultStore paths that were strings instead of Path objects. We spent an entire day just hardening the storage layer — atomic writes, once-per-process schema init, deadlock retries, corrupt file tolerance.

Branch surgery

Multiple feature branches developed in parallel, some with force-push incidents that broke shared history. At one point, two major branches had no common ancestor. We recovered via targeted cherry-picks — not glamorous, but it worked.

From prototype to product

Result analysis v2.1

Yifei designed the conclusion-first report format: executive summary up top, then single-game diagnosis, competitive matrix, integrated publishing judgement. Each section backed by traceable evidence from the aggregation layer.

The tricky part: this report layer lived on a separate branch that had diverged 20+ commits from main. Manual integration, zero conflicts on the cherry-pick (we got lucky).

Going live

Alibaba Cloud ECS for compute
Neon Postgres for persistence
Cloudflare for HTTPS and CDN
DashScope (Qwen) for all LLM calls

Deployment had its own set of fun problems — Cloudflare Flexible SSL causing redirect loops, sudo in the autodeploy script breaking permissions, frontend builds not regenerating after code pushes. We ended up with a one-minute cron that pulls main, installs deps, builds, and restarts services.

The last sprint

July 7-8 was all about polish: translating the entire frontend and all LLM outputs to English (the analysis prompts were originally in Chinese), adding game suggestions for better UX, removing debug artifacts, switching the license to Apache-2.0.

By the numbers

Metric	Value
Total commits	337 (across all branches)
Backend Python	~6,300 lines
Experience dimensions	17
Pipeline steps	8
Quality gate layers	3 (schema + statistical + LLM review)
Development window	June 20 – July 20
Team size	2

What we'd do differently

Start with the storage layer. We bolted on persistence late and paid for it in debugging time.
Prompt versioning from day one. Tracking which prompt version produced which results matters when you're iterating on extraction quality.
Fewer branches. Parallel development is great until you have to merge five divergent lines at 3 AM.

What we're proud of

The core insight held up: "LLM extracts, code aggregates, agent explains" is a genuinely better architecture than "throw everything at the model and hope." Every number in the final report traces back to specific reviews. Every recommendation has evidence. A publisher can challenge any conclusion and follow the chain back to what players actually wrote.

That's what we built. It works.

Try it live →