Reco: Autonomous Agent Engineering System

An autonomous agent engineering system that automatically designs, executes, benchmarks, diagnoses, and improves specialized AI agents through closed-loop hill climbing.

Comment

What's next for Reco: Autonomous Agent Engineering System

Inspiration

Building production-grade AI agents today is largely an unscientific, manual chore. Developers spend hours tweaking prompts without regression suites, hand-wiring multi-agent topologies based on vibes, and debugging tool schema errors through trial and error. Crucially, fixing an agent for one failure case often silently breaks another.

We asked: Can an AI system engineer other AI agents automatically? What if you could provide a high-level natural language goal, an active tool catalog, and an evaluation benchmark—and an autonomous meta-agent would design the architecture ($V_0$), benchmark it against real data, diagnose root causes of failure, synthesize targeted mutations ($V_1, V_2$), and validate them against an air-gapped held-out split to prove empirical Pareto dominance?

That vision is Reco.

What Reco Does

Reco automates the entire end-to-end AI agent engineering lifecycle through a closed-loop evolutionary harness:

  1. Goal Analysis & Task DAG Synthesis: Decomposes natural language goals and typed tool catalogs into formal task specifications.
  2. Architecture Generation ($V_0$): Synthesizes verified, acyclic agent topologies (GraphDefinition), defining node roles, system prompts, routing logic, and authorized tool bindings.
  3. Deterministic Execution Runtime: Executes agents with structured multi-round tool calling powered by TensorMux (GLM-4.7-Flash).
  4. 4-Axis Pareto Evaluation: Continuously benchmarks agent performance across Accuracy, Reliability, Cost (USD), and Latency (ms).
  5. 12-Class Failure Diagnosis: Classifies execution failures into actionable root causes (e.g., parameter mismatches, missing verifiers, hallucinated matches).
  6. Multi-Candidate Mutation Engine: Synthesizes alternative architectures (prompt tightening, tool rebinding, verifier guardrail insertion).
  7. Zero-Leakage Held-Out Promotion Gating: Tests top candidates against an unseen, held-out evaluation split to issue rigorous decisions: PROMOTE, REVIEW, or REJECT.
  8. Epistemic Memory Ledger: Extracts postmortem rules from earlier generations into permanent architectural invariants, preventing future regressions.

How We Built It

Reco was built end-to-end inside Agent Orchestrator (AO) and the agy CLI across more than 30 iterative milestones:

  • Core Engine & Runtime: Built with Python 3.11, FastAPI, Pydantic v2, and NetworkX for topological validation.
  • Model Inference Gateway: Deeply integrated with TensorMux (https://api.tensormux.com/v1) using glm-4-7-flash with native reasoning token extraction, schema translation, and token-level cost accounting.
  • Distributed Observability: Integrated with Neatlogs to stream hierarchical 5-tier execution traces (optimization_run → generation → benchmark_case → node_execution → tool_invocation) with direct inspection links and fault-contained async dispatch.
  • Cloud Ledger & Auth: Built on Supabase PostgreSQL with Row-Level Security (RLS) and GoTrue authentication to persist experiments, version DAGs, candidate scorecards, and failure diagnoses.
  • Monetization & Entitlements: Integrated Dodo Payments in test_mode for recurring Pro subscriptions ($29/mo, pdt_0Nmvzbo4wJETkRyCMAEPt), supporting hosted checkout, customer portal, and cryptographically signed HMAC webhooks with anti-replay protection.
  • Visual Console: Single-page application built with React 19, TypeScript, and Vite, featuring an interactive DAG graph visualizer, candidate tournament viewer, live trace inspector, and a 1-click Judge Demo Mode.
  • Deployment: Unified full-stack web service running on Render.

Verified Multi-Domain Empirical Results

We proved Reco's autonomous self-improvement across three distinct engineering domains with zero code changes to the core engine:

Domain Benchmark Split V0 Accuracy V1 Accuracy Held-Out Accuracy Cost Delta Latency Delta Decision
Transaction Reconciliation 12 opt / 8 held-out 75.00% 80.00% 82.50% -10.38% -20.54% PROMOTE
Dataset Anomaly Detection 5 opt / 3 held-out 80.00% 100.00% 66.67% +30.56% 0.00% REVIEW
Research Comparison 5 opt / 3 held-out 100.00% 100.00% 100.00% -9.83% -14.75% PROMOTE

In Anomaly Detection, Reco demonstrated genuine scientific honesty: despite achieving 100% on the optimization set, the held-out split hit an ambiguous tradeoff (2/3 passed), triggering a REVIEW decision instead of a naive false promotion.

Challenges We Ran Into

  1. Preventing Overfitting & Benchmark Leakage: Ensuring mutations don't memorize training scenarios required an absolute air-gap between optimization splits and held-out test cases.
  2. Deterministic Tool Calling over Real APIs: Handling real LLM tool-calling quirks (such as timestamp format mismatches between Unix epochs and ISO-8601 strings) prompted us to build the Epistemic Memory Ledger, where the system synthesizes type-coercion invariants dynamically.
  3. Multi-Objective Tradeoffs: Balancing speed and cost against accuracy required strict Pareto dominance criteria so the agent doesn't balloon tokens just to gain marginal accuracy.

Accomplishments We're Proud Of

  • Complete Closed-Loop Autonomy: Generating, executing, diagnosing, and evolving agent graphs without human intervention.
  • Rigorous Engineering Quality: 571 passing backend tests and 51 passing frontend tests.
  • Real Pareto Improvements: Up to +25% cumulative accuracy gain, with 10.38% lower inference costs and 20.54% lower latency.
  • 100% Sponsor Alignment: Native integrations with TensorMux (GLM-4.7-Flash), Neatlogs, Supabase, Dodo Payments, and built entirely within Agent Orchestrator (AO).

What We Learned

Agents can improve predictably when guided by structured failure taxonomies rather than open-ended prompt tweaking. When failures are classified into explicit categories (e.g., parameter mismatch, verification miss), mutations become focused mathematical operations that yield measurable Pareto gains.

What's Next for Reco

  • Dynamic MCP (Model Context Protocol) auto-discovery and sandbox execution.
  • Cross-agent transfer learning, allowing durable invariants learned in financial reconciliation to accelerate initial designs in supply-chain agents.
  • Multi-model ensemble generation, allowing Reco to assign distinct LLMs to specific nodes within the same DAG based on task complexity.

Built With

Share this project:

Updates

Submission history