Inspiration
We kept watching the same failure mode. A first-time ML builder opens a tutorial, copies a few lines, and then hits a wall the tutorial never prepared them for: their CSV has missing values, their target column is imbalanced, their accuracy looks great but their recall is terrible, and they have no idea which of those things actually matters. Static tutorials can't adapt to their dataset or their specific mistake, so the project stalls and gets abandoned.
At the same time, we noticed something about our own workflow: the most useful moment in a hackathon wasn't when an AI handed us finished code — it was when it explained why a line was failing. That gap between "generate" and "explain" felt like exactly the space the Agent In The Loop theme was pointing at. So we asked: what if the AI teammate wasn't an autopilot that builds the model for you, but a patient collaborator that builds it with you — proposing steps, explaining metrics, and waiting for your call?
What it does
ModelMate is a Streamlit web app where a beginner goes from raw CSV to a trained, understood classifier in a guided conversation.
The user uploads a small CSV (or picks a built-in dataset like Iris or Titanic) and selects a target column.
An AI agent inspects a profile of the data — dtypes, missing-value counts, cardinality, class balance — and proposes a short, ordered plan: which columns to drop, how to handle missing values, how to split the data, which model to try first.
Every proposal appears as a suggestion card the user can Accept, Reject, or Ask why. Nothing is applied silently.
After training, the agent explains the results in plain English — not just "accuracy is 0.94," but what that number hides, which classes are being confused, and what one concrete next experiment to run.
Every agent suggestion is written to a timestamped collaboration log, so the human–AI interaction is auditable rather than invisible.
The defining behavior: the agent proposes, the human decides, and both are recorded.
How we built it
The stack is deliberately small so a beginner can read every line:
Python + pandas for data loading and profiling.
scikit-learn for the ML pipeline (train_test_split, LogisticRegression, RandomForestClassifier, StandardScaler, confusion_matrix, classification_report).
Streamlit for the UI — suggestion cards, accept/reject buttons, metric panels, and the agent log.
An LLM API (OpenAI / Anthropic, swappable) as the runtime agent, called through a thin wrapper in agent.py.
A structured output contract: the agent must return JSON matching a schema (action, reason, risk, confidence), which we validate before rendering any card. Malformed responses are retried, then downgraded to a safe default.
The agent's core instruction is a system prompt we iterated on heavily: "You are a tutor, not an autopilot. Prefer the simplest defensible step. Never recommend hyperparameter tuning on datasets under 500 rows. Explain every metric in one sentence without notation." We also gave it the dataset profile as structured context rather than raw CSV rows, which cut hallucinations sharply.
We evaluated the agent itself, not just the model it produced. For each dataset we compared the agent's suggested pipeline against a hand-written baseline and logged agreement, disagreement, and which one a beginner could actually explain afterward.
Challenges we ran into
The agent wanted to over-optimize. On tiny datasets it kept recommending grid search over hyperparameters, which is statistically meaningless at that scale and pedagogically backwards. We added a hard guardrail: if n rows < 500 n rows <500, tuning suggestions are suppressed. This was our clearest lesson that agent behavior is a design problem, not just a prompting problem.
Auto-apply was a trap. Our first version let the agent apply preprocessing directly — it felt magical, and it was wrong. Beginners lost track of what had happened to their data, and a rejected step couldn't be undone cleanly. We rebuilt around suggest-then-confirm, which cost us UI time but made the "in the loop" idea real instead of decorative.
Explaining metrics without drowning people in notation. We had to decide how much math to show. The compromise: prose first, formula on demand. For example, the agent explains accuracy as "the share of all predictions that were right," then reveals Structured output reliability. LLMs drift out of JSON under long contexts. We solved it with schema validation, bounded retries, and a deterministic fallback plan (drop IDs → median/mode impute → 80/20 split → logistic regression) so the app never dead-ends on an API hiccup.
Class imbalance confusions. A model reporting 0.92 accuracy while missing every positive case is worse than useless — it's actively misleading. The agent now always cross-checks accuracy against the majority-class baseline before praising a result.
Accomplishments that we're proud of
A complete, working pipeline — upload, profile, propose, train, evaluate, explain — that runs end-to-end on real data, not slides.
Suggestion cards with accept/reject/ask-why, which turned "agent in the loop" from a theme into an interaction pattern.
An auditable collaboration log that makes the human–AI relationship inspectable after the fact.
Guardrails that came from observed failures, not speculation — the small-dataset tuning block and the majority-class baseline check both exist because the agent got them wrong first.
Explanations that a genuine beginner could repeat back to us in testing. That was the real bar.
What we learned
The agent is a proposer and explainer, not an autopilot. Every time we let it act without confirmation, the experience got worse — less trust, less learning, less control. The most valuable thing it did was not write code but make the reasoning visible.
Collaboration quality is measurable. We stopped asking "did the AI help?" and started asking "how often did the agent's proposal match a sensible baseline, and could the user articulate why?" That reframing changed what we built.
Prompts are product decisions. The line "prefer the simplest defensible step" is not a technical detail — it's the pedagogical stance of the whole app. We rewrote it more times than we rewrote the model code.
Design the confirmation UI first. If we started over, the accept/reject loop would be the first thing we built, not the last. The interface is the collaboration.
LLMs are strong explainers and weak statisticians. It could make precision vs. recall intuitive in one sentence, and simultaneously recommend tuning on 150 rows. Knowing which is which is the human's job.
What's next for ModelMate — An AI Teammate for First-Time ML Builder
Regression support alongside classification, with the same explain-then-confirm pattern (and R 2 R 2 , MAE, RMSE explained in prose before formulas).
Local-first mode via Ollama so classrooms and offline users can run the agent without an API key or data leaving their machine.
Agent self-critique pass: before showing a suggestion card, a second call asks the agent to argue against its own proposal. We want disagreement surfaced, not hidden.
A benchmark of agent suggestions across ~20 public tabular datasets, scored on baseline agreement and beginner-explainability, published with the repo.
Classroom mode: instructors see anonymized collaboration logs to spot where students get stuck — turning the audit trail into a teaching tool.
Broader model menu (gradient boosting, k-NN, naive Bayes) each with a one-line "when to use this" card, so the agent's choices become a curriculum rather than a black box.
Log in or sign up for Devpost to join the conversation.