ScopeCouncil

Two AI models propose your ERP scope, independently. Python - never an LLM - decides what actually gets built, with a citation for every call.

Comment

Inspiration

"Don't ask an LLM to build your ERP until you know what your ERP should be." Growing businesses - a retail chain, a restaurant, a logistics operation - already know something is wrong: "the inventory never matches," "we have no idea what to build for the restaurant." Every AI tool that promises to help jumps straight to generating a blueprint. Asking an LLM to sketch an ERP is trivial, and that's exactly the problem: its opinion on scope isn't verifiable, isn't consistent, and isn't accountable to anything. Ask it twice, get two different answers, with no way to tell which one is right. We wanted to build the missing layer that comes before the blueprint: a system that decides what's actually worth building first, and can prove why.

What it does

ScopeCouncil takes one free-text business symptom - "our inventory never balances," in informal Spanish or English - and decides the minimum viable ERP scope the business actually needs today:

  • A DiscoveryAgent (Gemini) reads the symptom and extracts business facts: affected processes, friction points, and whether this is really a process problem or a software gap.
  • Two independent proposers - ScopeProposalAgent (Gemini) and GemmaScopeAgent (Gemma) - read the same facts and propose candidate modules in parallel, unfiltered. A module only one model flags is never silently dropped or silently approved.
  • Python (agents/scoring.py), never an LLM, reconciles both proposals against versioned rules in rules/domain_rules.yaml and decides BUILD_NOW, DO_NOT_BUILD_YET, or HUMAN_REVIEW for every module - with the exact rule citation that produced the call.
  • Python also structurally validates the proposed database schema (circular references, missing primary keys, untyped columns, dangling foreign keys pointing at a table not in the draft).
  • A BuildPackAgent (Gemini) writes the prose around the already-final verdict - it cannot change a decision or invent a citation - and the result is rendered into a downloadable AI Build Pack: 10 fixed markdown documents (project context, architecture, database schema, module verdicts, workflows, roles/permissions, business rules, implementation roadmap, a master prompt, test scenarios) plus one focused build prompt per BUILD_NOW module - ready to hand to a human team or another AI coding agent.

Features

  • Two model families (Gemini + Gemma) propose independently, in parallel, via real Google ADK LlmAgents - disagreement between them is shown, never hidden.
  • Every BUILD_NOW / DO_NOT_BUILD_YET decision cites the exact domain rule that produced it - never "the model decided."
  • A built-in honesty test: some modules clear the build threshold despite triggering a penalty rule, and the system says so explicitly instead of hiding the tension.
  • A global budget cap: past 8 real BUILD_NOW approvals in one request, a Python rule ranks even the survivors instead of silently approving everything.
  • Live agent deliberation streamed to the browser over SSE - watch DiscoveryAgent, both proposers, the scoring engine, the schema validator, and the Build Pack agent hand off state to each other in real time.
  • A downloadable .zip Build Pack: 10 fixed markdown documents plus one focused build prompt per BUILD_NOW module, ready to build from.
  • Deployed live on Google Cloud Run.

Data sources

  • rules/domain_rules.yaml: a seed set of 12 versioned business rules, written from real SME ERP implementation patterns (tax/treasury, inventory management, multi-branch operations, shift/incident coordination) - not generic ERP boilerplate, and not exhaustive industry coverage on purpose.
  • No external data API - the domain knowledge lives in versioned, auditable rules a human can read, not in a prompt.

Challenges we ran into

  • Gemma (gemma-4-26b-a4b-it) genuinely takes 25-55 seconds to reason through this task - verified live with the raw google-genai client, not an ADK artifact. Forcing schema-constrained JSON decoding on it for this task consistently hung past 40 seconds even with a minimal schema, so GemmaScopeAgent runs without output_schema on purpose, with the JSON shape spelled out in the prompt and a markdown-fence-tolerant parser as a fallback.
  • If the parallel Gemini call fails, the Gemma call (which can run up to 70s) must not keep running orphaned in the background - a real bug we found and fixed by using explicit asyncio.create_task instead of a bare gather, so a failed Gemini call cancels the orphaned Gemma task instead of leaking a wasted, unread API call.
  • Deciding how a failed/slow Gemma call should degrade: the system falls back to "continuing with Gemini's proposal alone" and says so plainly in the live log, rather than pretending both models agreed.

Accomplishments that we're proud of

  • The final BUILD_NOW / DO_NOT_BUILD_YET decision is never an LLM's opinion - it's a Python function, against a versioned rules file, with a citation for every call.
  • The honesty test: modules that trigger a real penalty rule but still clear the build threshold say so out loud, instead of a system that only shows you the answers that make it look good.
  • 69 tests, zero network calls in the suite (every LLM call monkeypatched) - the pipeline's control flow is verified without spending a single real API call per test run.

What we learned

  • Two model families genuinely disagree more often than you'd expect on the same free-text prompt - and that disagreement is signal, not noise. Showing it beats averaging it away.
  • A model that runs slowly under real reasoning load (Gemma on a multi-module request) needs a plan for degrading honestly, not just a longer timeout.

What's next for ScopeCouncil

  • Grow rules/domain_rules.yaml beyond its current 12-rule seed set toward real industry-specific coverage (retail, restaurants, logistics, professional services).
  • A feedback loop: let a human's actual build decision (did the team really build what ScopeCouncil said BUILD_NOW?) retrain which rules fire with more confidence.
  • Multi-turn scoping - right now ScopeCouncil takes one symptom per run; a longer conversation could sharpen employee_count_band and industry_hint instead of leaving them optional and often unknown.

One of four

One of four projects on the same spine — an LLM proposes structure, deterministic code decides the outcome, and the verdict is grounded in an independent source you can inspect. Here it's a versioned YAML rules file; a point-in-time Wayback capture in The Promise Ledger, Mexico's SAT shell-company blacklist in Trusted Hire México, a cited linguistic archive in Language Recovery OS.

Built With

Share this project:

Updates