BenchPilot

• 

• 

• 

  The frontend

Inspiration

We were inspired by a simple problem: experiments are expensive, but choosing what to try next is often surprisingly inefficient.

Whether you're testing a chemical reaction, optimizing a biological process, tuning a manufacturing system, or running an expensive computational experiment, every trial consumes something valuable. Researchers may rely on intuition, manually explore parameter combinations, or use grid search—approaches that can waste a large portion of a limited experimental budget.

We wanted to build something that could actively learn from every experiment and use that information to decide what should happen next. That led us to BenchPilot: a closed-loop experimental optimization tool that treats every trial as information and continuously improves its recommendations.

What it does

BenchPilot helps researchers decide which experiment to run next when time, money, or materials are limited.

A researcher defines their experimental parameters, constraints, and objective, then enters previous trial results. BenchPilot fits a Gaussian Process to the data and uses Expected Improvement to recommend the next experiment, including its predicted result and uncertainty.

The researcher can then run the real experiment, enter the measured result, and repeat the process. The model continuously updates as new data comes in.

We also built a demonstration that races BenchPilot against random experimentation across multiple synthetic landscapes and repeated trials. This lets users see not just one lucky example, but how the optimization strategy performs across different conditions.

Importantly, the LLM does not make the scientific recommendation. The mathematical optimizer makes the decision; the LLM simply translates the optimizer's computed results into an understandable explanation.

How we built it

We built BenchPilot as a full-stack web application.

The backend uses FastAPI and scikit-learn to handle experiment sessions, validation, Gaussian Process regression, and our implementation of Expected Improvement. We normalized the search space and used multi-start L-BFGS-B optimization to search for promising experimental parameters.

The frontend was built with React, TypeScript, and Vite. We created interfaces for defining experiments, entering historical trials, viewing recommendations and uncertainty, logging results, and visualizing optimization progress. Recharts powers the interactive graphs and comparison visualizations.

For the explanation layer, we optionally use Claude to turn the optimizer's existing numbers into plain English. If an API key isn't available, BenchPilot falls back to a deterministic explanation template.

For the demo, we created synthetic landscapes based on the user's parameter space and repeatedly compared BenchPilot with random search. This gave us a way to demonstrate the underlying optimization method without pretending that our synthetic results represented real laboratory experiments.

Challenges we ran into

One of our biggest challenges was implementing Bayesian optimization ourselves instead of relying on a framework like BoTorch or Ax. We wanted to keep the project lightweight and understandable, so we implemented the Expected Improvement acquisition function and optimization process directly.

We also had to think carefully about what could go wrong with experimental data. Parameter names need to be unique, ranges must be valid, and historical trials have to fall within the declared search space. We also made sure that the predicted result is never automatically submitted as if it were an actual experimental measurement.

Another challenge was making the demo scientifically meaningful. A single comparison against random search could easily be misleading because random search can get lucky. We therefore ran multiple replicates across multiple landscape seeds and reported the median, interquartile range, and win rate rather than showcasing only our best result.

Finally, we had to balance technical sophistication with a user experience that a researcher could understand without needing to know how Gaussian Processes or acquisition functions work.

Accomplishments that we're proud of

We're especially proud that BenchPilot isn't simply an AI chatbot pretending to do science. The core recommendation comes from an actual optimization algorithm.

We built a complete closed loop:

experiment → result → model update → recommendation → experiment

We are also proud of the evaluation system we built into the demo. Across our synthetic benchmark of 8 landscape seeds and 8 replicates per landscape, BenchPilot won approximately 73% of individual races and finished closer to the optimum on 7 of 8 landscapes.

Just as importantly, we designed the demo to show its failures too. BenchPilot doesn't win every time, and users can reproduce cases where random search keeps up. We think being transparent about the limitations makes the demonstration much more meaningful.

What we learned

We learned that building an effective scientific tool is about much more than getting the underlying algorithm to work.

We learned how Bayesian optimization can make decisions under uncertainty, how Gaussian Processes can model relatively small experimental datasets, and how acquisition functions turn those models into actionable decisions.

We also learned the importance of evaluation design. It is easy to build a demo that produces an impressive-looking result, but much harder to build one that fairly tests whether the method actually works.

Most importantly, we learned that AI can be much more useful when it is given a clearly defined role. Instead of asking an LLM to invent scientific recommendations, we used mathematics for the decision-making and AI for communication. That separation makes the system more interpretable and trustworthy.

What's next for BenchPilot

The biggest next step is testing BenchPilot on real experimental data rather than synthetic landscapes.

We want to support more complex experimental workflows, including categorical parameters, multiple objectives, noisy measurements, and more sophisticated constraints. We also want to add integrations that allow researchers to automatically import experiment results and eventually send recommended parameters directly to laboratory or computational workflows.

Our long-term goal is to make BenchPilot a general-purpose **closed-loop experimentation platform- one that helps researchers spend less of their limited experimental budget searching blindly and more of it learning something useful.

Built With

• api
• bayesian
• claude
• css
• fastapi
• html
• l-bfgs-b
• numpy
• python
• react
• recharts
• scikit-learn
• scipy
• typescript
• vite