Confiron

• 

  Our Benchmark results for Qwen3.5-4b

Inspiration

I've been fascinated by LLMs for a while. During the Nebius AI Performance Engineering course, I discovered optimized inference engines such as vLLM, SGLang, and the less widely known TensorRT-LLM. I saw an opportunity to help users get more value from their models by making inference optimization easier to explore and apply.

What it does

Confiron helps users from choosing a model and inference engine to finding a serving configuration suitable to their workflow and workload. It connects model recommendations, configuration search, and remote GPU execution in one experience.

The web app currently supports Qwen3-4B, Qwen3-8B, Qwen3.5-4B, Qwen3.5-9B, and Qwen3-14B, along with compatible DSpark draft models for the Qwen3 models. Regarding the data, ShareGPT, Alpaca dataset, messages(role/content) formats are supported .

Keep in mind, Bayesian optimization needs an initial exploration period before it can use previous results to guide the search. With our current settings, the first 10 trials use random sampling, so choose more than 10 trials to benefit from guided optimization. We recommend starting with 20 trials.

Our offline studies also support models such as Gemma 4 12B and MiniCPM5. The web app's tested, stable worker images currently use older versions of vLLM and SGLang, so its supported model selection is narrower than that of our offline studies.

How we built it

We built Confiron around three components:

1. Recommender: An agent that combines Tavily research, Hugging Face lookups, model scoring, and use case mapping to recommend models suited to the user's needs.
2. Optimizer: A Bayesian configuration explorer that searches serving parameters across vLLM, SGLang, and TensorRT-LLM. We focused on efficient exploration, prepared workload reuse, caching, and duplicate suppression to reduce avoidable work during optimization.
3. Orchestrator: Launches and monitors Nebius GPU jobs, coordinates optimization runs, and preserves results, logs, and artifacts.

Challenges we ran into

1. Supporting different models and inference engines requires handling many compatibility constraints. We built a flexible foundation, but broader coverage still requires testing each model and runtime combination.
2. We initially focused on discovering overlooked models. As we tested them, we came to appreciate why established model families are widely used and shifted our attention toward Qwen, Gemma, and MiniCPM.
3. Configuring Nebius infrastructure, credentials, and environment variables took time. Nebius AI Assistant helped us work through the setup.
4. In our experiments at concurrency 1, changing server parameters often had a smaller effect than we expected, making improvements harder to identify. We need to further investigate it.
5. Runs may take some time because each Nebius job uses the user’s selected dataset, and we currently do not reuse random trials from offline studies for faster Bayesian optimization.
6. The performances depend on gpu resources and we observed slight variance across same parameters and seed runs.
7. In our experiments, speculative decoding did not consistently improve performance at high concurrency and request counts. KV-cache memory pressure may have contributed, but we have not verified the cause.

Accomplishments that we're proud of

We observed performance improvements across multiple models and inference objectives:

built a shared workflow for comparing serving frameworks, and developed a model recommendation tool that connects research with practical execution choices.

We're also proud of the evidence behind each run: saved configurations, benchmark metrics, startup diagnostics, and artifacts that make results easier to inspect and compare.

What we learned

Serving parameters can make a substantial difference to throughput and latency, but the gains depend on the model, workload, hardware, and inference engine. Careful measurement matters as much as the search strategy.

Nebius, NVIDIA, and Tavily tools helped us turn model research and performance experiments into a connected workflow. We also learned to distinguish a promising configuration from one that performs consistently under repeated testing.

Next for Confiron

We explored several Bayesian samplers and all three serving frameworks. Next, we want to test larger models, a wider range of workloads, different worker and request counts, and more dataset formats and task types.

We also plan to update and validate the web app's worker images so that more models can become stable options in the app.

Built With

• ai
• bayesian
• huggingface
• nebius
• optuna
• probabilistic
• python
• serverless