heal-run-detail
heal-timeline

SuperTaco — AI-supervised fine-tuning sandbox

Type a plain-English goal. SuperTaco plans the run, generates data, trains on a free Colab T4, heals its own failures, and a Nemotron judge on Nebius scores before/after — fine-tuning with receipts.

Inspiration

Fine-tuning an open model today means babysitting it. You hand-write a config, watch loss curves, decode a cryptic OOM stack trace at 2am, nudge a knob, relaunch, and repeat — if you can afford a GPU at all. We kept meeting students and indie developers with a clear idea of how a model should behave but no GPU budget and no MLOps background. Meanwhile the pieces for something better already existed: free Colab T4s, strong open models, and Nemotron on Nebius Token Factory. The missing piece was supervision — something that plans the run, watches it, fixes it, and proves it worked. We built that piece.

What it does

What It Does

SuperTaco turns one plain-English goal into a finished, scored fine-tune using $0 GPU hardware costs.

Plan — Enter a goal, such as "answer like a pirate chef." NVIDIA Nemotron, via Nebius Token Factory, plans the run: model selection, LoRA hyperparameters, a bespoke synthetic dataset (or a named Hugging Face dataset), and five goal-specific evaluation prompts.
Train — LoRA fine-tuning launches on a free Google Colab T4 GPU, with live training logs streamed directly into the dashboard.
Heal — When training breaks, an eight-mode deterministic playbook classifies the failure, cross-checks it with Nemotron, and attaches relevant Tavily documentation citations. It patches the configuration into a new file—nothing is ever edited in place—and automatically relaunches the run, with up to three healing attempts and zero human input.
Prove — A Nemotron judge scores base-model versus fine-tuned-model responses on a scale of 0–10. The run passes the regression gate when the fine-tuned model's average score is at least 90% of the base model's average score:

$$ \overline{s}_{ft} \geq 0.9 \cdot \overline{s}_{base} $$

The judge runs as a Nebius Serverless Job. Its API key is injected securely through MysteryBox, never exposed in job specifications or logs. Scores are returned through presigned S3 URLs, and the job is deleted after execution. Every cloud or LLM fallback is explicitly labeled—never silent.

Iterate — Provide feedback such as "be more concise," and SuperTaco revises the plan, regenerates training data, and evaluates the updated fine-tune.

Every LLM call is recorded in an append-only audit log, and every run produces a verifiable artifact with a unique run_id.

A 10-goal evaluation sweep delivered 10/10 successful runs, 9/10 improvements, and a mean judged improvement of +2.14 points:

$$ \bar{\Delta} = \frac{1}{10}\sum_{i=1}^{10}\left(\overline{s}_{ft}^{(i)}-\overline{s}_{base}^{(i)}\right)=+2.14 $$

Every result can be traced back to the audit log for independent verification.

The v2 dashboard brings the entire workflow together in a three-page Streamlit product:

Landing: Project overview, architecture, and live verification evidence.
Live Run Wizard: A four-step workflow—Goal → Review → Monitor → Report.
Results: Evaluation sweep results, individual run details, healing timelines, and copy-paste verification commands.

One goal. Free GPU training. Automatic recovery. Verifiable results.

How we built it

Language/stack: Python 3.11+, pydantic v2 configs, Streamlit (three routed pages, worker-thread runs), google-colab-cli for the T4 transport, OpenAI-compatible client for Token Factory.
Sponsor tech, three surfaces:
- Nebius Token Factory — Nemotron with cost routing: Nano classifies $\rightarrow$ Super plans/patches/judges $\rightarrow$ Ultra reserved for novel failures.
- Nebius Serverless Jobs — eval-as-a-job: a stdlib-only container script injected via FileInjection, keyed through MysteryBox, results over presigned S3, always-deleted afterward.
- NVIDIA Nemotron models do all planning, classification, patching, and judging — called from the host and inside the job container.
Engineering discipline: immutable configs (every patch is a new file), append-only audit log (logs/llm_calls.jsonl with model, mode, tokens, latency, error, run_id), schema-versioned run artifacts, and a 357-test suite that is audit-hermetic — a full test run provably leaves the audit log byte-for-byte unchanged. Ruff and mypy gates are clean.
Honesty as architecture: circuit breakers, labeled hash-score fallbacks, direct_fallback judge mode, dashboard warning banners — degradation is always visible, never silent.

Challenges we ran into

Reasoning models break assumptions. Nemotron Nano now reasons before it answers: a 25-row dataset chunk can legitimately take $t > 90\text{s}$, and under a tiny token budget it returns no content at all. Our original 30s timeout plus a one-strike circuit breaker silently degraded the flagship synthetic-data feature. Fix: 180s chunk timeouts, a 2-strike breaker with 60s cooldown and half-open probing — verified live against the exact failing workload.
LLMs write sloppy JSON. Strict parsing of generation chunks meant one malformed reply discarded 25 rows at a time, dropping runs below the 100-row quality floor. We made the degradation honest — a labeled alpaca fallback, never a silent one — and chunk retry/salvage hardening is the first post-submission fix (see What's next).
Real cloud, real quotas. The first judge-as-a-job run failed honestly on an S3 storage-class quota (QuotaLimitExceeded); the labeled fallback preserved the run, and forcing the bucket to ENHANCED_THROUGHPUT fixed the root cause. The live smoke then passed end-to-end: job aijob-e04wgsj23r9e9vg3p1, provisioning $\rightarrow$ completed in 237s, $10/10$ real judge calls.
Free-tier flakiness is not a bug. Colab VMs occasionally hang mid-exec; we bounded every run with hard deadlines and an always-stop rule so infrastructure flakiness ends as an honest run_failed, never a leaked VM or a hung demo.
Cross-platform reality. A Windows cp1252 encoding bug mojibake'd judge prompt keys in a live run — fixed with ASCII-folding plus a regression test that reproduces the exact live error.

Accomplishments that we're proud of

A live, no-human-input self-heal: a real OOM on a 1.5B run was classified, patched (batch halved $+$ gradient checkpointing), and relaunched to success on 2026-10-11 — the moment the project stopped being a script and became a supervisor.
Judge-as-a-Nebius-Job: not an API wrapper — the evaluation itself executes on Nebius compute, with secrets hygiene (MysteryBox), S3 handoff, payload validation, and guaranteed cleanup.
Receipts, not promises: the 10-goal sweep table where every row carries a run_id a judge can independently verify against the audit log ($\bar{\Delta} = +2.14$, verified to the decimal).
357 tests, clean lint and types, and an audit-hermetic suite — for a solo hackathon project built in five weeks.
A design philosophy we're happy to be quoted on: every fallback in the system is labeled. Judges never have to trust us — they can check us.

What we learned

Labeled degradation beats fake robustness. Systems that hide their fallbacks eventually lie to you; systems that label them stay debuggable and demo-able under pressure.
Reasoning-model economics are different. Classification over long log tails can burn $8\times10^{4}$ tokens of "thinking" in a single call — cost routing has to account for reasoning tax, not just model price.
Constraints are a feature. The $0 GPU tier forced immutable configs, aggressive timeouts, whitelist validation, and cleanup discipline — the things that make it trustworthy.
Verification is a feature too. "9/10 improved, $+2.14$ avg" is marketing until every number is greppable; making receipts a first-class artifact changed how we wrote everything else.

What's next for SuperTaco

Nebius GPU training tier — a NebiusTransport that trains on Nebius AI Cloud GPUs with Colab kept as the free tier (scoped, designed, and deliberately cut for the deadline; it's the natural v2.1).
Generation hardening — retry + salvage for malformed dataset chunks so the 100-row floor is statistically guaranteed.
A public run gallery — the showcase mode already replays recorded runs; next is a browsable, shareable gallery of community goals and their receipts.
Deeper playbook — more failure modes contributed by users, with the Tavily research loop evolving from advisory citations to suggested patches (still human-approved).
Multi-run comparisons — A/B judging across feedback iterations, so "be more concise" shows up as a measurable curve, not a single delta.

Built solo for the Nebius × NVIDIA Global AI Hackathon · 357 tests · Apache 2.0 · github.com/1tsDaksh/supertaco