ProfessorVGC

Inspiration

With the Pokémon VGC World Championship happening this very week, and as a competitive player myself planning to qualify for LAIC (Latin America International Championship), I wanted a tool that could actually help me improve — not another chatbot that talks about Pokémon, but something that could analyze my real ranked games with the same rigor I'd expect from a human coach. That gap between "AI that sounds knowledgeable" and "AI that's actually right" is what ProfessorVGC set out to close.

What it does

ProfessorVGC is an AI orchestration built on Google's Agent Development Kit (ADK) that analyzes real Pokémon VGC replays and significantly reduces hallucination across four areas: damage calculations, game-state/turn-order reconstruction, metagame analysis, and inference of hidden information (EVs, IVs, abilities, and movesets) based on real usage frequency data from Pokémon Showdown.

The current version is scoped to VGC 2026 Regulation M-B (Champions). To use it, a player saves their replay from the Showdown server and pastes the link (or the raw log) when prompted.

How we built it

The system is split into two AI stages with distinct responsibilities, wrapped around a deterministic core:

The 1st AI (Selection) parses the replay into a structured GameState (Pydantic models) and decides which Pokémon and matchups the user's question is actually about.
A deterministic engine — a real damage calculator (@smogon/calc, run as a Node.js subprocess) — is then re-consulted turn by turn against the actual battle log, creating a feedback loop that lets the pipeline verify projected damage against what really happened and reconstruct the causality between turns, instead of ever letting the model guess it.
The 2nd AI (Explanation), powered by Gemini 3.5 Flash, aggregates everything — the verified turn-by-turn data, live metagame statistics pulled from Google Cloud Firestore, and strategic context fetched from the official Smogon strategy wiki — into a single, grounded natural-language answer.

The whole pipeline is deployed on Google Cloud Run.

Challenges we ran into

Catching a hallucination our own architecture was supposed to prevent. While testing the app on my own ranked games, it once suggested using Fake Out — a Normal-type move — against Gholdengo, a Ghost/Steel-type Pokémon that's completely immune to Normal-type moves. The root cause turned out to be subtle: a list of "non-damaging status moves" meant to be excluded from the damage calculator had mistakenly included Fake Out (which is actually a real 40-power attack), so the engine never computed a damage figure for it at all — leaving the model with no grounded fact to catch its own bad assumption. The fix went into the deterministic engine itself, not the prompt, and we added a regression test so this exact failure mode can never silently come back.

Making Google Cloud Firestore the only source of metagame data, with no local fallback, meant every read had to be cheap and correct from day one — we designed a per-species document schema instead of one document per file, and verified it against the local dataset byte-for-byte before trusting it in production.

Corporate network TLS interception (an antivirus intercepting outbound HTTPS with its own root certificate) broke gRPC's connection to Firestore in a way that didn't affect our other HTTP-based integrations at all, requiring a custom CA bundle just for that one connection.

Accomplishments that we're proud of

We didn't just claim that grounding an LLM in deterministic data reduces hallucination — we measured it. Using an atomic-claim-verification benchmark (extract → verify → rate), we compared our grounded pipeline (Condition A) against the same LLM given only the raw battle log with no deterministic support (Condition B), using Fisher's exact test on the resulting 2×2 contingency table:

$$OR = \frac{a \times d}{b \times c}$$

For our competition-default configuration (Gemini 3.5 Flash + Google ADK), the grounded condition was correct on damage-range claims 73.3% of the time versus 11.0% for the naive baseline — an odds ratio of $OR = 22.27$ at $p = 2.70 \times 10^{-15}$. Across every LLM vendor and orchestration framework we tested, the effect held with odds ratios between 15x and 80x, always statistically significant well beyond the $p < 0.05$ threshold.

We're also proud that this was built solo, over weekends around a full-time job, with a Clean Architecture disciplined enough to swap LLM vendor, orchestration framework, or data backend without touching a single line of business logic — and that it's genuinely deployed and running on Google Cloud, not a local-only demo.

What we learned

The most useful debugging instinct we picked up: when an LLM states something factually wrong in a domain that has a checkable, computable answer, the fix usually isn't a better prompt — it's making sure the deterministic layer underneath actually computed that fact in the first place. A model can't get grounded in data that was never generated for it to see. We also learned, very concretely, that domain expertise is irreplaceable for QA: a subtle type-immunity bug that a generic test suite would never think to check was caught because the person testing the app actually understood competitive Pokémon at a high level.

What's next for ProfessorVGC

We plan to open-source the project and share it in Pokémon community forums, so competitive players can actually use it to improve — not just read about it. We also want to document the architecture itself through follow-up articles and post-mortems, since the whole system is deliberately decoupled and generic enough to be repurposed for other domains that need an LLM grounded in a real, checkable source of truth, far beyond Pokémon.