Your next experiment starts with a benchmark you need to trust
You find a published result that looks like the right baseline. Before spending time reproducing it, you need to know: was it measured on the same data, at the same forecast horizon, with the same metric calculation?
Relay Research helps ML researchers answer that question—and revise their argument when the evidence changes.
Our demonstration starts with a real detail in the GTS paper. Its original METR-LA table reports a 60-minute MAE of 2.99. Appendix D explains a metric-calculation correction, and the corrected table reports 3.41. Both numbers are published, but treating them as interchangeable evidence can lead to a misleading comparison. Relay did not produce these experimental results; it helps inspect their context.
What the researcher actually does
- Ask a question. Enter a research question and optionally add arXiv papers you are considering as baselines. Relay discovers papers and extracts source-linked experimental evidence.
- Inspect the comparison. Results retain their dataset, split, horizon, aggregation, metric definition, preprocessing, and provenance. Relay explains whether the comparison is comparable, incompatible, or has insufficient evidence.
- Investigate a gap. Choose Investigate on a comparison. Nemotron examines relevant paper passages and supported author-code repositories. If configured, Tavily searches the web for details that remain unresolved.
- Review a proposal. The researcher inspects its source quotation and accepts or rejects it. Discovery does not fabricate corrections, and an investigation can legitimately find no answer.
- Follow the consequences. An accepted correction recomputes comparisons and marks dependent manuscript claims stale. Regenerate those claims, inspect the earlier checkpoint, or explore an independent branch.
- Design the next experiment. Turn reviewed evidence into cited related work and a proposed evaluation, edit LaTeX/BibTeX, and export a manuscript and evidence bundle.
The deliverable is an inspectable baseline decision and a proposed experiment grounded in the available evidence.
The idea behind Relay: validation → feedback → correction
Our earlier WebMCP project taught us that giving an agent tools is only part of the problem. The application must return useful feedback so the agent and researcher can inspect a mistake and recover.
Relay applies that lesson to one shared evidence state. A comparison conflict becomes an investigation. A reviewed correction changes that state. Dependent claims become stale. Saved checkpoints and branches let the researcher examine what changed.
The recorded five-paper journey includes a correction to the description of GTS's metric calculation, impact records for ten comparisons, and four manuscript claims marked stale and subsequently regenerated. Some comparison outcomes stayed the same while their explanations changed; the impact view makes that distinction visible. Reviews and proposal decisions in this recorded journey were made by coding agents/test scripts, not independent human researchers.
Where Nebius, NVIDIA, and Tavily fit
NVIDIA Nemotron 3 Super on Nebius Token Factory performs search planning, relevance assessment, experimental evidence extraction, paper/code reasoning, synthesis, and manuscript generation or revision. We use nvidia/nemotron-3-super-120b-a12b through actual inference API calls, not a branding-only integration.
OpenAlex and arXiv supply the initial scholarly records and available full text.
Tavily is an optional runtime fallback during investigation. Add a Tavily key in Model settings; when paper/code analysis leaves protocol fields unresolved, Relay searches the web and passes source-linked passages to the investigation. A web result remains evidence to assess, not an automatic correction. The Tavily fallback was live-smoke-tested on October 28 and returned four results, including the MTGNN author repository. The earlier prepared Nemotron studies predate this integration.
Deterministic Python checks validate source/table context, calculate comparison outcomes, protect revisions, track dependent claims, and verify checkpoint hashes. The website and API clients use the same research operations.
What is working today
- Interactive paper/topic maps and source-linked experimental evidence.
- Three-outcome comparisons and paper/code/web investigations.
- Reviewable proposals, correction impact, stale-claim revision, and independent branches.
- A LaTeX/BibTeX workspace, local sandboxed PDF compilation, and Overleaf-compatible exports.
- Browser-held project copies, project import/export, and an in-app research/API desk.
- 103 passing automated tests.
The included fresh discovery run assessed four papers using 12 Nemotron calls and 114,655 tokens. The five-paper correction record contains 22 cumulative calls and 312,770 tokens. Model usage, source evidence, and execution records are inspectable.
How we tested—and what remains uncertain
We compared Relay with a single-pass baseline using the same Nemotron model and source papers. In the latest small development run, Relay incorrectly accepted 1 of 13 non-comparable cases, versus 6 of 13 for the baseline. Decisive-answer coverage was 80% versus 90%, so caution comes with a trade-off.
The evaluation is agent-annotated, development cases informed fixes, and held-out cases share papers with development. These are demonstration results, not independently validated general scientific accuracy.
The current schema specializes in ML/time-series benchmarks, especially traffic forecasting. Numeric results remain provisional until reviewed. We have not run new forecasting experiments, conducted an independent researcher study, or measured user time savings.
What we learned building it
A number appearing in a source quotation does not prove it belongs to the correct table column. A missing extracted setting does not prove the paper omitted it. A cited baseline is not necessarily an independent reproduction.
Those failures shaped Relay's table-aware extraction, explicit unknowns, result provenance, bounded retries, and visible correction history. Nemotron output also varied across repeated runs, so we retained failures and costs rather than presenting one successful run as universal accuracy.
Codex assisted implementation, API acceptance, source review, and manuscript amendment. Agent activity is explicitly attributed. The research product was built during the hackathon from an earlier document-handoff prototype.
Try the complete workflow
Hosted app: https://relay-research.onrender.com/ Public MIT-licensed repository: https://github.com/mohammed1916/ai_workflow_manager
Open the Research desk and choose Open prepared study / Load recorded study, or Open recorded discovery. These are clearly labeled saved live-run records. For a fresh investigation, supply a Nebius key; Tavily is optional. Hosted PDF compilation is disabled: compile locally or export to Overleaf.
What's next
Independent domain annotation, evaluation on papers without development overlap, stronger table-context recovery, and a controlled replication of selected baselines.
Our most useful platform improvement would be stronger structured-output guarantees and clearer finish/error diagnostics for long scientific extraction calls. Token Factory let us use the same Nemotron model for the application and baseline without provisioning GPU instances.
Built With
- arxiv
- css3
- html5
- javascript
- nebius-token-factory
- nvidia-nemotron
- openalex
- python
- tavily
- tectonic