Inspiration

Different search providers are good at different things. The best provider for finding a product might not be the best one for technical research or travel planning. We wanted to build a system that could discover those differences through continuous experimentation, using the outcomes users actually care about.

That became Search Optimizer: a feedback loop that learns which search provider to use for each query category.

What we built

Our system routes queries to search providers, then learns from downstream outcomes such as purchase conversions, users confirming they reached their goal, and follow-up behavior.

We built three connected interfaces:

  • A live chatbot that asks the optimizer which provider to use, performs the search, and reports observed user outcomes through a feedback tool.
  • A simulation bed that generates traffic with configurable category weights and outcome probabilities. We can change the conditions live and watch how the optimizer responds.
  • An optimizer dashboard that shows traffic, feedback, and how routing preferences evolve.

The demo presets let us test scenarios where one provider wins everywhere or different providers win in different categories. Switching between them makes it easy to explore how the system adapts.

How we built it

Each search goes through a simple loop: identify the query category, choose a provider, perform the search, and collect feedback when an outcome becomes observable.

The optimizer balances trying different providers with sending more traffic to those that have performed well. It updates its routing preferences in the background as new feedback arrives.

We built the simulator alongside the live experience so we could test known conditions, introduce changes, and inspect how the system responds.

What we learned and the challenges we faced

Defining feedback was one of the hardest parts. A relevant-looking result doesn’t establish that the user accomplished their goal. Outcomes can arrive later, and an unknown outcome must remain distinct from a confirmed negative one.

We also saw how easily early feedback can influence routing. A provider getting lucky on its first few queries shouldn’t settle the experiment. Exploration and uncertainty matter.

Building the simulator helped us make those behaviors visible. Live controls, outcome charts, and markers for configuration changes gave us a way to inspect the feedback loop and test whether it was learning what we intended.

Built With

Share this project:

Updates

Submission history