AgentTune: Autonomous AI Workflow Optimizer

Inspiration

Large language models offer different trade-offs between capability, latency, and resource usage. Sending every task to a powerful model can be inefficient, while relying entirely on lightweight models can compromise response quality. We built AgentTune to explore how intelligent model routing can balance these trade-offs while maintaining reliability.

What it does

AgentTune benchmarks model-routing strategies and compares them against an all-Super baseline. It routes routine tasks to a lightweight model and complex tasks to a stronger model, with bounded fallback when responses are empty, truncated, or format-noncompliant.

The system measures criterion coverage, rubric scores, latency, failure rates, format compliance, token usage, and fallback behavior. Acceptance gates prevent a candidate configuration from replacing the baseline unless it meets the configured criteria.

How I built it

We developed AgentTune using Python, FastAPI, and a web dashboard, integrating NVIDIA Nemotron models through the Nebius Token Factory API. The benchmark suite contains 12 tasks across routine and complex categories. Experiment results are stored using SQLite and JSONL, and pytest tests validate core application, routing, evaluation, and fallback behavior.

We implemented both mock and live execution modes, allowing repeatable demonstrations alongside real model inference.

Challenges we faced

One major challenge was that lightweight-model responses could be truncated or contain reasoning transcripts instead of only the requested answer. We implemented response validation and bounded fallback to the stronger model. Live experiments also revealed that fallback overhead can increase latency, even when quality-related metrics improve.

Another challenge was evaluating quality reliably. Our current criterion-coverage metric uses literal phrase matching, so it cannot independently establish semantic correctness. Cost comparison also requires verified model-pricing configuration.

What we learned

We learned that model routing is not simply about choosing the fastest or cheapest model. Reliable optimization requires measurable baselines, explicit acceptance gates, response validation, and transparent reporting. Our live experiments showed improved recorded quality-related metrics in one run, but also higher latency due to fallback. This highlighted the importance of evaluating quality, reliability, and performance together.

What's next

Future improvements include stronger semantic evaluation, larger benchmark suites, repeated-trial statistical analysis, reduced unnecessary fallbacks, and verified cost-aware optimization. Our goal is to make AI workflow optimization more reliable, measurable, and explainable.

Built With

• css
• fastapi
• html
• javascript
• jsonl
• nebius-token-factory
• nvidia-nemotron
• pytest
• python
• restapi
• sqlite
• uvicorn