Task Orchestrator

• 

  logo

Inspiration

Getting an AI agent to succeed once is exciting. Getting it to succeed consistently—and understanding why it fails—is the harder problem. Task Orchestrator grew from a question: can we turn agent development into a repeatable cycle of building, testing, learning, and improving, with evidence behind every change?

What it does

Task Orchestrator creates task-specific agents from a natural-language goal, a selected evaluator, and a set of available tools. An Architect generates the agent's prompt and configuration, choosing either a single-agent workflow or a planner-and-worker workflow.

The system then gives the agent prepared tasks, repeats each task to measure consistency, and grades the answers using explicit rules. It tracks overall success and whether a task succeeds on every trial, alongside cost, latency, and execution history.

When tasks fail, an AI-powered improvement stage examines the evidence and proposes changes to memory, prompts, tool descriptions, or orchestration. Each candidate is evaluated again. The acceptance gate requires previously consistent tasks to remain consistent and the overall pass rate not to decrease. Qualifying candidates also receive a holdout evaluation before becoming the current version.

The web interface lets users inspect agents, runs, learned memory, proposed fixes, and version comparisons. The repository includes GitHub issue triage using 60 real issues and a separate support-ticket evaluator with 50 synthetic cases.

How we built it

The backend uses Python, FastAPI, Pydantic, and SQLite, with an event ledger and saved transcripts for tracing what happened during each run. Model calls use the OpenAI Python SDK through an OpenAI-compatible interface. GitHub tools provide issue context, similar issues, label information, and ownership evidence.

The frontend uses Next.js, React, TypeScript, Tailwind CSS, and Recharts. Agent versions are stored as inspectable packages containing their configuration, prompt, tools, and memory. Evaluation suites pair task datasets with Python scoring functions. A reusable playbook mechanism can extract lessons from accepted fixes and apply them during later agent creation.

Challenges

The central challenge was separating a lucky answer from a dependable capability. Repeated trials help reveal inconsistent behavior, while regression checks protect tasks the agent already handles well. Another challenge was making automated changes understandable: saved transcripts, version differences, and structured events let a user inspect the evidence behind a proposed fix.

Designing the exam is also part of building the agent. A passing score only means what its grading rules actually check, so evaluator quality and representative tasks are essential.

Accomplishments

The project brings agent creation, repeated evaluation, failure diagnosis, versioned changes, and regression checks into one inspectable workflow. The reviewed build passes 769 offline automated tests covering the supporting software. These tests validate the machinery; live evaluation results are needed to establish an agent's real-world performance.

What we learned

Agent quality depends on the entire system: instructions, tools, memory, workflow, and the evaluation itself. Improving a prompt is only one option. Keeping evidence for each run makes it possible to reason about changes instead of relying on a convincing demo.

What's next

Broaden the evaluation datasets, strengthen holdout isolation, expand grading to cover every important output, and add human review of sampled results. Future work also includes easier evaluator creation and support for new task types, including artifacts that require richer grading than structured text.

Built With

• fastapi
• github
• next.js
• openai
• pydantic
• pytest
• python
• react
• recharts
• sqlite
• tailwindcss
• typescript