Inspiration

The project originated from a QA sweep identifying eight distinct bug groups in a multi-agent clearing house system: ledger divergence with false-ok invariants, dispute amounts posted at gross instead of net, dead worker URLs going undetected, 15-second dashboard request timeouts, ROUNDS columns showing "attempt" instead of settled rounds, incorrect judge scenario labels, UI overflow and labelling nits, and unreachable approval/operator features. The inspiration was to fix every single one of them, deploy the fixes to a public-facing Render stack, and verify the entire end-to-end flow — trade creation → LLM negotiation → worker execution → settlement → invariant check — works without crashing, without zombie consoles, and without hiding external PayPal country restrictions behind generic 500 errors.

What it does

AEON is a web console (https://aeon-lqc9.onrender.com/) for a multi-agent clearing house where:

Operators can log in, run trades, approve/reject settlements, pause/resume agents, and run four judge scenarios (S1–S4) Judges see a quota badge (JUDGE RUNS LEFT · {count}), with operators getting -1 (unlimited) and judges getting real counts Trades enter 2-round live LLM negotiation via Cloudflare AI (@cf/meta/llama-3.1-8b-instruct), execute on distributed workers, and settle with price/cpu_s/rounds displayed in a grid Ledger invariant checks pass (ok:true, wallet_mismatches:[]) with dispute holds recorded at gross - fee - penalty (net proceeds), not gross price PayPal sandbox integration surfaces 403 PAYOUT_NOT_AVAILABLE with PayPal's exact message You live in a country that is not allowed to send this payout as structured 502, instead of opaque 500 Kill switch toggles system on/off, disabling all actions and showing agent down status TopBar banner shows ROLE · OPERATOR | PAYPAL SANDBOX | LLM · LIVE | DEMO READY | INVARIANT 0 USD ✓

How we built it

The system is built as a Full-Stack TypeScript/React + FastAPI project with:

Frontend (aeon/dashboard/): React dashboard with DataConsole.tsx (agent grid), ApprovalPanel.tsx, JudgeSandboxController.tsx, TopBar.tsx, styles.css (header wrap, grid rows auto 1fr, ROUNDS column fix), formatMoney in toasts Backend (aeon/backend/): FastAPI with app/services/settlement.py (propose_settlement: needs_approval before guardrail), app/guardrails/policy.py (check_settlement), app/paypal/client.py (PayPal SDK with proper 403→502 error surfacing), app/ledger/service.py (atomic post() with rollback + invariant reconciliation), app/api/control.py (exposes approval_threshold_micro), app/api/readiness.py (next_actions listing unhealthy workers) Deployment: Three Render hosts (aeon-lqc9.onrender.com, aeon-worker-1.onrender.com, aeon-1.onrender.com) with render.yaml configuring worker service names, APPROVAL_THRESHOLD_MICRO=50000, and PayPal credentials Git: Repo root E:\AEON, branch master, remote origin → Puneet04-tech/AEON; two commits (662526d, 418d58c) + one fix commit (8110ee8), pushed to origin/master with no Co-Authored-By footer AI Tool: Cloudflare LLM (@cf/meta/llama-3.1-8b-instruct) for 2-round negotiation, with verification via independent_deterministic checks (output checksum, envelope checksum, result shape, independent worker agreement) PayPal Tool: PayPal REST SDK (paypal-rest-sdk via app/paypal/client.py) in sandbox mode; micro_to_decimal/to_micro conversion; PayPalError caught and surfaced as 502 with code, message, status, details

Challenges we ran into

Resolution Log

Accomplishments that we're proud of

✅ All 8 bug groups fixed with specific code changes, verified behavior, and pushed to origin/master with proper commit messages and no Co-Authored-By footer ✅ Live deployment on Render: /healthz → ok, /api/v1/hackathon/readiness → ready:true with both workers healthy, trades SETTLED with invariant ok:true ✅ Zombie-session boot fix: stale/invalid token now lands on login form instead of console showing ROLE · JUDGE / PAYPAL NOT CONFIGURED / LLM OFF with all requests 401 ✅ Dispute hold fix: hold = net proceeds (gross - fee - penalty), not gross price; live invariant verification confirms wallet_mismatches:[] ✅ PayPal error surfacing: structured 502 PAYOUT_NOT_AVAILABLE with PayPal's exact country-restriction message instead of opaque 500 ✅ Operator/judge dual mode: -1 quota correctly reads as unlimited for operators; judge sees real count with buttons disabling at 0 ✅ Scenario controller: 4 predefined scenarios (S1–S4) run in a single step; toast S1: COMPLETED; quota tracking in TopBar badge ✅ Agent grid with Pause/Resume: toggle reflects ACTIVE/PAUSED; toast confirms every action; columns: Bond, Uptime, Late, Action, Escrow ✅ Full end-to-end verified flow: login → run trade → 2-round LLM negotiation → worker execution → trade SETTLED → grid update → event stream → invariant check → approvals (if above threshold) → scenario controller → kill switch

What we learned

QA-to-production pipeline: Explicit bug-group identification → minimal code fix → verification → commit → push → deploy is viable and produces measurable quality improvements Render deployment workflow: render.yaml + package.json scripts → npx tsc --noEmit → npm run build → automatic deployment; bundle hash must match for auto-deploy PayPal sandbox limitations: The country restriction is IP-based and external to AEON code; the correct fix is proper error surfacing, not bypassing Negative quota as unlimited: The pattern scenarios_left: -1 → "no quota" is counterintuitive but user-experience-friendly once the >= 0 guard is in place Bootstrap token validation is security-critical: Without api.me() validation on login, stale tokens leave zombies that 401 everything; adding the check is a 3-line change with outsized impact Invariant reconciliation requires wallet cache rollback: The post() method must roll wallet caches back on failure; otherwise wallet_mismatches accumulate and the invariant fails AG Grid + React: colId: "toggle_agent" with valueGetter returning "Pause"/"Resume" based on status works well; cellClass: "cell-action" + action column needs strict locator strategy to avoid "4 elements found" errors

What's next for AEON · Clearing House Console

Resolve PayPal country restriction: Deploy to a different IP region or use live PayPal with proper licensing (code already handles the error; the block is external) Add live PayPal integration: Requires OPENAI_API_KEY + production PayPal credentials; PAYPAL_ENV=live is blocked in production by settings.assert_production_safe() Extend scenario set: S5–S8 with more complex agent failure patterns, bond liquidation, and reroute logic Add audit trail export: CSV export of ledger entries, invariant checks, and approval decisions for compliance Multi-tenant Render instances: Per-user Render deployments with individual PayPal credentials and approval thresholds Agent chaos corruption mode: setWorkerChaos(workerId, workerUrl, fail, corrupt) already in store — expose UI control Formal QA pipeline: Automated regression tests for the 8 bug groups + new scenarios on every commit

Built With

• ag-grid
• cloudflare
• fastapi
• mongodb
• paypal
• react
• render
• tailwind