AgentEval Forge is a spec-driven template for building and evaluating AI agents with Kiro — plus two demo apps built from the same foundation.
The core is a small deterministic eval harness (forge/). A task is one JSONL line with id, kind, params, and expected. The runner dispatches each task to the executor registered for its kind, times it, and scores it with a judge that is a pure function: no randomness, no network, no LLM at runtime. Chess duels are scored on whether the actual winner matches the expected winner (a draw against a decisive expectation scores 0.5); trading backtests pass when they make at least the expected number of trades, stay at or above the expected PnL floor, and stay within the expected maximum drawdown. Results are stored in SQLite, and forge leaderboard aggregates them per agent — games, wins, draws, losses, average score, pass rate — combined across both demo apps. Re-running the same task file always produces the same scores.
The two demo apps prove the template with real work, not placeholders. Chess Duel Arena pits two instances of a mini chess engine (0x88 board, alpha-beta search with quiescence) against each other at different search depths, from the standard start position to checkmate, stalemate, draw, or the 200-ply limit. The engine is adapted from the author's tiny-chess engine that was a finalist in the make.fun tiny-chess competition. The BTC Paper-Trading Bot is a Python port of the author's make.fun btc-race-24h entry (8th place, $15 USDC): a dip-buy mean-reversion strategy backtested offline on 1,500 real BTCUSDT 1-minute closes from Binance public market data (18-19 November 2026), plus two small synthetic fixtures, clearly labelled, that exercise a take-profit win (+1.84% PnL) and a bounded-loss exit (-0.97% PnL). Both apps share a FastAPI web UI with a combined leaderboard and one-click runs in the browser.
How to run (for judges): from a fresh clone, in the repo root: python3 -m venv .venv; source .venv/bin/activate (on Windows: .venv\Scripts\activate); pip install -r requirements.txt. Then: python -m forge run --tasks tasks/chess-duels.jsonl --db data/results.db --agent forge-arena (takes ~4 minutes); python -m forge run --tasks tasks/trading-backtests.jsonl --db data/results.db --agent forge-trading (fast, offline); python -m forge leaderboard --db data/results.db; uvicorn apps.chess_arena.server:app --reload --port 8000, then open http://localhost:8000 (Chess Duel Arena) and http://localhost:8000/trading (BTC Paper-Trading Bot). Run the tests with pytest -q (79 tests).
Notes: No API keys. No paid services. No network at runtime. The harness, the chess engine, and the trading bot run fully offline on bundled static data. Deterministic: no randomness, no LLM calls, and no network in the scoring path. Paper trading only: the trading bot simulates trades against historical data, never holds real funds, keys, or exchange credentials, and is not live trading. Honest fixtures: the two synthetic price files (btc-dip-bounce.csv, btc-crash.csv) are small, hand-designed, and labelled as synthetic in the README.
