AgentForge Mini | Syndicate by Maximor Demo

AgentForge Mini evaluates agents, diagnoses failures, lets AI propose fixes, reruns the same benchmark, rejects regressions, and promotes only evidence-backed improvements.

Comment

AgentForge Mini

Inspiration

AI agents are increasingly easy to build and modify, but determining whether a modification actually made an agent better is much harder.

A new prompt, model response, or repair can improve one task while silently breaking another. Benchmark results can also become misleading if expected answers leak into the improvement process.

That led to the core idea behind AgentForge:

AI proposes. AgentForge verifies.

Instead of trusting that an AI-generated change is better, AgentForge treats improvement as a measurable promotion decision.

What it does

AgentForge Mini is a test-and-improve governance loop for autonomous agents.

The workflow is:

Agent V1 → Benchmark → Evaluation → Failure Diagnosis → AI Candidate → Agent V2 → Same Benchmark → Regression Analysis → PROMOTE / REJECT

A candidate is promoted only when:

  1. Its aggregate benchmark score improves.
  2. It introduces zero regressions.
  3. It introduces zero critical failures.

Otherwise, the candidate is rejected and the existing version remains authoritative.

How I built it

The project was built in Python using Agent Orchestrator (AO) throughout development.

I used separate AO workers and branches for major parts of the system, including the project foundation, deterministic evaluator, failure diagnosis, candidate improvement, regression analysis, Ollama integration, structured extraction, and later hardening.

The AI improvement path uses a locally hosted Llama 3.1 8B model through Ollama. The model proposes improvements, but evaluation and promotion remain deterministic.

AgentForge currently demonstrates the same governance loop across two different domains.

Technical support

The first benchmark contains debugging and support tasks involving SQLite locking, Git merge conflicts, HTTP timeouts, environment variables, authentication, and related technical problems.

The baseline scored 71.2%.

After AgentForge diagnosed its failures and the local model proposed targeted repairs, the candidate reached 90.7%, with:

  • zero regressions
  • zero critical failures
  • a final decision of PROMOTE

Structured business extraction

I then reused the same architecture for structured document extraction involving invoices, purchase orders, support tickets, and shipment records.

The baseline scored 75.0%.

The improved candidate reached 87.5%, again with zero regressions and a PROMOTE decision.

This second domain helped demonstrate that AgentForge is not tied to one specific debugging benchmark.

Benchmark integrity

One of the most important design decisions was preventing the improvement model from simply learning the benchmark answers.

The model never receives the benchmark's expected output during candidate generation.

For structured extraction, it receives the original document and the names of fields that were diagnosed as missing or incorrect, but not their canonical expected values.

Already-correct fields are protected, and candidate repairs are constrained to diagnosed fields.

This keeps proposal generation separate from evaluation.

Challenges

The hardest part was not getting the model to produce an improved response. The harder problem was deciding when that improvement could actually be trusted.

During development, one extraction candidate dropped from 75% to 50% and introduced two regressions.

AgentForge rejected it.

I also encountered malformed JSON responses from the local model. Instead of trying to guess what the model intended, AgentForge fails closed and retains the baseline.

Those failures became useful evidence because they demonstrated that promotion was genuinely conditional rather than automatic.

Another challenge was working with a local language model. Even with deterministic generation settings, model behavior can vary between runs, so the surrounding evaluation and promotion system had to remain independent of the model.

What I learned

The biggest lesson was that AI-generated improvement should be treated as a proposal, not as authority.

An autonomous system needs more than generation. It needs evaluation, regression detection, explicit promotion criteria, failure handling, and evidence that can be inspected afterward.

I also learned the value of using Agent Orchestrator to separate implementation work into workers, branches, reviews, and pull requests. That created a much clearer development trail and made it easier to challenge and harden individual parts of the system.

The final implementation passes 55 tests covering evaluation, candidate improvement, benchmark integrity, regression handling, structured extraction, and model failure behavior.

What's next

AgentForge Mini is intentionally a focused prototype.

The next steps would be to add persistent experiment history, configurable benchmark suites, richer critical-failure policies, support for additional agent runtimes and models, and an operator interface for reviewing promotion decisions.

The long-term goal is simple:

AgentForge doesn't ask whether an agent feels better. It tests whether it actually became better.

Built With

  • agent-orchestrator-(ao)
  • git
  • llama-3.1-8b
  • ollama
  • pytest
  • python
Share this project:

Updates

Submission history