DatasetOps Autopilot

• 

  How Datasetops Autopilot Works

• 

  Why Automated RAG Evaluation Matters

• 

  Multi-Agent Collaboration

• 

  Seamless Integration

• 

  Evaluator

• 

  Intake Planner

• 

  Source Understander

• 

  Graph Viewer

Inspiration

Every team building a RAG (Retrieval-Augmented Generation) system eventually hits the same wall: how do you know if it actually works? Evaluating a RAG chatbot requires a solid benchmark — realistic questions, grounded answers, and clear difficulty tiers — but building that benchmark by hand from raw policy documents, manuals, or knowledge bases is slow, inconsistent, and easy to get wrong. A human annotator might miss edge cases, phrase questions unnaturally, or fail to catch when an "answer" isn't actually grounded in the source at all.

At the same time, most people asking for a benchmark don't specify one precisely — "build me a benchmark for our support docs" is a real request, but it's ambiguous: how many samples, what difficulty spread, which categories matter? Someone (or something) has to turn that ambiguity into a structured, executable plan.

We wanted to see whether a multi-agent system could own that entire journey — from messy documents and a vague request, all the way to a validated, exportable dataset — while still keeping a human in control at the moments that matter most. That's the problem DatasetOps Autopilot is built to solve, and it's a natural fit for Track 4: Autopilot Agent, which asks exactly for this — ambiguous inputs, tool use, and human-in-the-loop checkpoints, judged on production-readiness rather than demo tricks.

What it does

DatasetOps Autopilot converts raw source documents and an ambiguous benchmark request into a validated RAG evaluation dataset package — autonomously, with two human checkpoints along the way.

The pipeline:

1. Ingests uploaded documents and parses/chunks them into stable, retrievable units.
2. Understands the source — summarizes content, flags weak or thin sections, and maps what's actually benchmark-worthy.
3. Plans the benchmark — turns a vague request into a structured plan: categories, difficulty distribution, sample count, quality rules.
4. Pauses for human approval — Checkpoint 1. The user reviews and approves the plan before any generation happens.
5. Generates grounded samples — retrieves relevant chunks and produces question–answer pairs tied to explicit source chunk IDs.
6. Evaluates every sample against a structured, multi-dimensional rubric (grounding, answerability, clarity, hallucination risk, and more).
7. Repairs automatically — samples that fail evaluation are sent back to the generator with specific feedback, in a bounded Generator–Critic negotiation loop, capped at a fixed number of rounds to control cost and avoid infinite retries.
8. Pauses for sample review — Checkpoint 2. Only uncertain or unresolved samples reach the human; the rest flow straight through.
9. Exports the final package: rag_eval.jsonl, answer_key.jsonl, dataset_card.md, quality_report.md, all bundled into a downloadable ZIP.

The result: a team can go from "here are our docs, build us a benchmark" to a validated, ready-to-use evaluation dataset — without hand-writing a single question.

How we built it

The system is organized into five specialized agents — IntakePlannerAgent, SourceUnderstandingAgent, BenchmarkGeneratorAgent, QualityEvaluatorAgent, and ExportReportAgent — each a focused unit of reasoning rather than a monolithic do-everything agent. Between the generator and evaluator sits a NegotiationAgent: a bounded repair loop that lets the two negotiate over a failing sample for a fixed number of turns before escalating to a human.

We deliberately assigned different Qwen model tiers to different agents — a lighter model for generation, a stronger model (qwen-max) for evaluation — since the evaluator acting as a stricter "critic" than the generator is both cheaper to run and produces a more reliable quality gate. Every generated sample carries explicit source chunk IDs, so grounding can be checked concretely rather than assumed.

On the infrastructure side: FastAPI + SQLAlchemy backend, PostgreSQL with pgvector for semantic chunk retrieval, a Next.js frontend, all containerized with Docker Compose and deployed behind Nginx on Alibaba Cloud ECS, with the frontend on Vercel. Export artifacts — the final JSONL files, reports, and ZIP packages — are stored via Alibaba Cloud OSS.

Challenges we ran into

Building a multi-agent pipeline surfaced problems that don't show up in single-agent systems:

• Keeping repair loops bounded and cost-safe. A generator–evaluator negotiation loop can, in principle, run forever if failures keep recurring. We had to design explicit termination logic — a hard cap on repair rounds, with automatic escalation to human review once the cap is hit — so the system fails gracefully instead of looping or silently burning through LLM budget.
• Deciding what should be an "agent" versus deterministic code. It's tempting to let an LLM make every routing decision, but that makes the system harder to debug and easier to break in subtle ways. We had to be deliberate about which parts of the pipeline are genuine LLM reasoning (understanding, generating, judging) versus plain state-machine logic (what happens next given a decision) — and resist over-agentifying components that are really just routing.
• Evaluation consistency across dimensions. Scoring a sample on grounding, answerability, clarity, and hallucination risk simultaneously means the evaluator has to reason about several, sometimes competing, criteria at once and still produce a single coherent pass/repair/reject decision — not just a set of independent numbers.
• Retrieval quality directly determines generation quality. Because every generated question-answer pair has to be grounded in retrieved chunks, weak or imprecise retrieval propagates straight into weak samples downstream — there's no later stage that can fully compensate for bad evidence retrieval upstream.
• Coverage vs. quality tension. Pushing hard to hit a target sample count can conflict with maintaining strict quality thresholds, especially when source documents have thin sections. Balancing "get enough samples" against "never relax the quality bar" required treating coverage shortfalls as a reportable outcome rather than something to quietly patch over.

Accomplishments that we're proud of

• A complete, end-to-end autonomous pipeline — from raw documents to a validated, exportable benchmark package — with no manual question-writing required.
• A genuinely bounded, cost-aware repair loop: failing samples get real, structured feedback and a real chance to improve, without risking runaway LLM spend.
• Two meaningful human-in-the-loop checkpoints (plan approval, sample review) that put a human at the decisions that matter, without forcing them to review every single output.
• A workflow that's fully explicit and auditable at every stage — every state transition, retry, and routing decision can be traced end-to-end, which matters far more for a system meant to be trusted in production than for a one-off demo.
• A working deployment on Alibaba Cloud ECS with Qwen Cloud (DashScope) integration, running behind Docker Compose and Nginx — proof that this isn't just a local notebook experiment.

What we learned

• Multi-agent doesn't mean "let the LLM decide everything." Some of the most important design decisions were about keeping control flow in code and reserving agents for genuine reasoning — that split made the system far easier to trust and debug.
• A repair loop is only as good as its termination condition. The value of "the system can fix its own mistakes" only holds up if it also knows when to stop trying and ask for help instead.
• Grounding has to be enforced structurally, not just requested. Asking an LLM nicely to "only use the source" isn't enough — the pipeline needs explicit chunk ID tracking and evaluation checks that verify grounding rather than assume it.
• Production-readiness is a design constraint, not a checklist item. Things like bounded retries, explicit state machines, and structured quality reports weren't afterthoughts — they were what made the difference between a workflow that demos well once and one that would actually hold up under repeated real-world use.

What's next for DatasetOps Autopilot

• Guaranteed coverage via a gap-filling loop — a dedicated pass that detects when the final approved sample count falls short of the target (due to source thinness or repeated rejections) and attempts additional, targeted generation rounds — without ever relaxing quality thresholds — before falling back to an honest, reported shortfall.
• Richer retrieval — moving further into hybrid (semantic + keyword) retrieval to improve evidence quality for harder, multi-hop question types.
• Additional export formats — Alpaca and ShareGPT-style exports for teams that want to reuse benchmark data for fine-tuning, not just evaluation.
• Multi-language and multi-domain presets — pre-tuned planning templates for common verticals (support docs, legal, e-commerce policy) to reduce setup friction for new users.
• Deeper observability — expanding trace and cost dashboards so teams running this in production can audit exactly what each agent did and why, at scale.

Built With

• alibabacloud
• dashscope
• docker
• dockercompose
• fastapi
• nextjs
• pgvector
• postgresql
• python
• qwencloud
• sqlalchemy
• typescript