Tof-lm Project Story
Inspiration
Tof-lm started from a very familiar research pain point: machine learning projects often become harder to understand as they become more active. A single idea quickly turns into many experiment runs, partial notes, log files, config changes, bug reports, and TODOs spread across the repository. The technical artifacts exist, but the reasoning behind them is easy to lose.
The core inspiration was simple: what if a research repository could behave more like an active collaborator than a passive folder tree? Instead of asking a human to manually reconstruct what happened from logs and notes, we wanted an agent that could inspect the repository, summarize what changed, detect likely failures, and then turn those findings into concrete next actions.
We were also inspired by the idea that a useful agent should not stop at text generation. In research workflows, the most valuable output is often operational: a report, a triage issue, a plan for the next ablation, or a handoff to a project system. That led us to pair Gemini 3 with a GitLab MCP integration layer so the project could bridge reasoning and action.
What We Built
Tof-lm is a Gemini 3-powered GitLab MCP research agent for messy ML repositories. It reads experiment artifacts such as logs, notes, and configs, extracts signals like metrics and failures, builds a multi-step mission plan, and produces structured outputs:
- a reproducible Markdown research report
- a persisted agent state JSON file
- GitLab MCP issue actions in either
draftorexecutemode - a hosted FastAPI endpoint and Gradio UI for demo and interaction
At a high level, the system follows this pattern:
$$ \text{repository evidence} \rightarrow \text{structured signals} \rightarrow \text{agent plan} \rightarrow \text{report + MCP actions} $$
That pipeline matters because it turns raw ML artifacts into something that can drive collaboration.
How We Built It
We built Tof-lm as a layered system instead of a single prompt wrapper.
1. Repository Evidence Layer
The first layer scans the workspace and identifies likely research artifacts. We wanted deterministic evidence gathering first, because it keeps the system inspectable and makes debugging much easier. This layer is responsible for reading logs, notes, and experiment-related files and extracting patterns such as:
- run markers
- metric observations
- warning signals
- failure signals
- explicit next-step notes
In other words, before asking a model to reason, we first construct a local working memory.
2. Gemini 3 Mission Layer
On top of the deterministic evidence layer, we built the Gemini 3 mission layer. Gemini is used as the reasoning surface that frames the mission, interprets what matters, and guides the multi-step flow. The project is currently designed so that offline demos can run safely in dry-run mode, while the architecture is prepared for live Gemini-backed execution when credentials are available.
The mental model we used was:
$$ \text{Agent Output} = f(\text{Question}, \text{Evidence}, \text{Tool State}) $$
where the answer is not just prose, but a combination of synthesis and action-ready artifacts.
3. Multi-Step Planner
We did not want the system to behave like a one-shot chatbot. So we added an explicit planning layer that breaks the mission into tool-oriented steps, such as:
- scan the workspace
- analyze evidence
- write the report
- prepare or execute GitLab MCP actions
This made the system easier to explain, test, and extend. It also made the agent visibly more than a text interface, because it leaves behind an execution trace instead of only a response.
4. GitLab MCP Partner Integration
The partner-power part of the project is the GitLab MCP integration layer. We designed it with two modes:
draft: export MCP-ready requests for offline demos and safe testingexecute: connect to a real GitLab MCP server and invoke issue-creation tools through the MCP Python SDK
This was an important architectural choice. It let us keep the project demoable in constrained environments while still making the core integration real rather than hypothetical.
5. Hosted Interface
To make the project easier to demo and interact with, we added:
- a FastAPI service with clean endpoints like
/healthand/run - a Gradio UI mounted at
/ui
This made it possible to present Tof-lm as both an API and a human-friendly interface, which is especially useful in a demo setting.
What We Learned
One big lesson was that useful agents need structure, not just model quality. A strong model helps, but the real leverage comes from making the workflow explicit: what data is gathered, how it is normalized, when tools are called, and what artifacts are produced.
We also learned that partner integrations are most valuable when they are treated as operational endpoints rather than just branding. GitLab MCP became meaningful once it was part of the agent loop: not only "the agent can think about issues," but "the agent can prepare or execute issue actions as part of the mission."
Another lesson was about trust. Research teams are more likely to adopt an agent if they can inspect the evidence trail. That is why Tof-lm exports not just a report, but also agent state and execution metadata. The goal is not only intelligence, but legibility.
Finally, we learned that deployment constraints shape product decisions. It is easy to imagine a fully live cloud agent, but practical constraints like offline development, billing requirements, and access to real external services push you toward designs with graceful fallback modes. That turned out to be a feature, not a compromise.
Challenges We Faced
Designing For Real Work, Not Just Demo Output
One challenge was resisting the temptation to build a flashy summarizer. It would have been easy to stop at "read a log, produce a summary." But that would not satisfy the deeper goal of building an agent that actually helps move research forward. We had to keep asking whether each system component created operational value.
Balancing Deterministic Parsing With LLM Reasoning
Another challenge was deciding what should be hard-coded and what should be delegated to Gemini. If we pushed too much into model reasoning, the system became harder to debug. If we made everything deterministic, the outputs risked becoming too brittle. The resulting architecture intentionally combines both:
$$ \text{use deterministic extraction for evidence} + \text{use Gemini for mission-level reasoning} $$
That balance was one of the most important design choices in the project.
Real GitLab MCP Integration
Making the GitLab MCP layer real was a meaningful challenge. The key problem was that we wanted the project to truthfully support live MCP execution while still working in offline or restricted environments. The draft-versus-execute split solved that, but it required additional design work in configuration, reporting, and error handling.
Deployment Friction
Hosting turned out to be a practical challenge too. Some deployment paths, especially cloud-managed ones, introduce extra setup requirements such as billing or project configuration. That pushed us to think more carefully about multiple deployment paths and what would make the project easiest to evaluate.
Why We’re Proud Of This Project
We are proud that Tof-lm is not just a repository analyzer and not just a prompt wrapper. It is a small but coherent agent system with:
- a research-specific problem framing
- a multi-step mission model
- concrete artifact generation
- a real partner integration strategy through GitLab MCP
- both CLI and hosted interfaces
Most importantly, it points toward a version of AI tooling that is useful in day-to-day research work: not simply answering questions about experiments, but helping teams turn experimental evidence into organized action.
What’s Next
The next steps are clear:
- expand live Gemini-backed synthesis
- broaden GitLab MCP support beyond issue creation into updates and comments
- improve structured parsing for CSV, JSON, and tracker exports
- compare runs across config changes and experiment lineage
- make the hosted workflow smoother for real teams
The long-term vision is a research workspace that remembers what happened, explains why it matters, and helps decide what to do next.
Log in or sign up for Devpost to join the conversation.