RasoSynthTune_CUTC

A production-ready autonomous AI dataset generation system that discovers, extracts, filters, and constructs high-quality fine-tuning datasets using multi-provider AI orchestration.

Comment

RasoSynthTune: Production-Grade AI Dataset Engineering and LLM Alignment Platform

Fine-tuning Large Language Models (LLMs) needs high-quality, diverse, and relevant instruction datasets. Traditional web scraping and simple LLM dataset generation often score poorly in terms of effectiveness, show domain drift, have duplicate content, and include irrelevant junk from the web like HTML banners, navigation menus, and anti-bot scripts.

RasoSynthTune tackles these issues by offering a production-grade platform for AI dataset engineering and quality evaluation. It automates dataset extraction, synthetic generation without seeds, multi-aspect quality scoring, and curriculum tiering based on Bloom's Taxonomy. This results in clean datasets that are ready for immediate fine-tuning with Unsloth, Hugging Face SFTTrainer, Axolotl, and OpenAI.

Inspiration

In our experiments with fine-tuning open-weight models like LLaMA-3, Qwen, and DeepSeek, we found that dataset quality, not quantity, is the main problem. Current open-source synthetic dataset tools rely on simple string templates (for example, "Explain X based on: <raw chunk>"), which produce off-topic or low-coherence samples that hurt LLM alignment performance.

We aimed to build RasoSynthTune to provide AI engineers with a reliable workspace. This workspace enforces strict domain relevance, advances prompt optimization (using PromptBreeder and DSPy), and evaluates quality through multi-tier benchmarks from Scale AI and UltraFeedback.

How We Built It

RasoSynthTune is designed as a multi-provider microservices architecture:

1. Robust Core Orchestration and LLM Router (ProviderRouter)

  • Multi-Provider Fallback: It smoothly routes tasks among DeepSeek, Groq, OpenRouter, Together AI, Hugging Face Router, NVIDIA NIM, and Google Gemini, with automatic retries and failover.
  • Observability: Fully integrated with LangSmith (ObservabilityManager) for real-time telemetry, token budgeting, and latency monitoring.

2. Evolutionary and Seedless Synthetic Generation (SeedlessGenerator)

  • PromptBreeder and DSPy Contracts: It uses evolutionary prompt mutation algorithms (PromptBreeder) and DSPy contracts to create complex multi-turn dialogs and single-turn instructions without needing human input.
  • Robust Multi-Stage Regex Parsing: It has a 4-tier JSON recovery engine (_extract_json_object and _extract_json_array) that cleans up markdown fences, trailing comments, and faulty LLM outputs automatically.

3. Multi-Aspect Quality Scoring Engine (QualityScorer)

We implemented a strict gating matrix based on four important evaluation dimensions: $$\text{Final Score} = \text{Domain Gate} \times \left( w_1 \cdot \text{Semantic} + w_2 \cdot \text{Relevance} + w_3 \cdot \text{Completeness} + w_4 \cdot \text{Coherence} \right)$$

  • Domain Relevance Gate: It scores zero ($0.00$) for off-topic samples or mismatched domain terms.
  • Garbage and Anti-Bot Gating: It instantly rejects irrelevant web artifacts (like <div align=, <img src=, Cloudflare CAPTCHAs, and README badges).
  • Bloom's Cognitive Taxonomy Tiering: It automatically categorizes datasets into six cognitive complexity tiers (Remembering, Understanding, Applying, Analyzing, Evaluating, Creating) for progressive learning.

4. LLM Fine-Tuning Exporters

  • It allows direct conversion to OpenAI JSONL (messages), Stanford Alpaca (instruction/input/response), and ShareGPT (conversations) formats.

Challenges We Faced

  1. Eliminating Greedy Regex and LLM Parsing Errors: LLMs often wrap JSON in conversational markdown. We replaced simple greedy regex matching (\{.*\}) with non-greedy multi-pass extraction to fix string decoding errors without losing data.
  2. Scraped Garbage False Positives: Scraped technical text usually included navigation headers and HTML banners. We used zero-score gating matrices to prune out garbage data before scoring.
  3. Domain Relevance vs. Subtopic Taxonomy: Strict keyword matching initially flagged valid technical terms (for instance, Raft consensus under Distributed Systems or PD-1 inhibitors under Clinical Oncology). We created domain taxonomy expansion maps to evaluate technical subtopics accurately.

What We Learned

  • Quality Over Quantity: Filtering for high-coherence, high-completeness samples greatly boosts LLM fine-tuning convergence rates.
  • Resilient Pipeline Design: Reliable production LLM applications need structured multi-provider fallback and defined schema parsing to function effectively at scale.

What's Next for RasoSynthTune

  • Direct Fine-Tuning Execution: We plan to integrate with Unsloth and Hugging Face AutoTrain to enable training jobs directly from the RasoSynthTune interface.
  • DPO / RLHF Preference Pair Generation: We will automate the creation of chosen/rejected response pairs for Direct Preference Optimization.

Built With

Share this project:

Updates

Submission history