SLOKern

SLOKern optimizes LLM inference by benchmarking serving configurations, finding the lowest-cost option that meets defined SLO, and independently verifying its performance under reproducible workloads.

Comment

Inspiration

When working with LLMs, it’s easy to focus on the model itself and forget that serving it efficiently is a whole different problem.

A configuration might give good throughput but terrible latency. Another might respond quickly but struggle under load. And even if a benchmark looks promising, how do you know the result is actually reliable?

That’s where the idea for SLOKern came from. I wanted to build something that could help answer a practical question: given a set of performance requirements, which serving configuration should we actually trust?

Not just the one that looks best in a single benchmark, but one that meets the requirements, has evidence behind it, and can be tested again.

What it does

SLOKern is a tool for testing and comparing LLM-serving configurations against specific performance requirements, or SLOs.

The idea is pretty straightforward. Define the workload and the limits you care about, measure a baseline, test a small set of alternative configurations, and evaluate the results against those limits.

It looks at metrics such as time to first token (TTFT), inter-token latency (ITL), end-to-end latency, throughput, and errors. Instead of blindly picking whichever configuration has the highest throughput, SLOKern is designed to reject configurations that violate hard requirements.

It also separates a genuinely bad result from a run where the evidence itself is unreliable. If the measurements aren't good enough to support a decision, the system should say so instead of pretending it knows the answer.

The final goal is to identify a promising configuration, run it again independently, and explain why it was selected.

How we built it

We started by working out the decision process before trying to build everything around it. The core flow is:

Specify → Validate → Measure → Evaluate → Optimize → Verify → Explain.

We used Pydantic v2 for structured contracts so that different parts of the system could agree on what a workload, experiment, metric result, and final decision should look like.

We also designed the evaluator around explicit outcomes: feasible, infeasible, or inconclusive. That distinction matters because a configuration failing an SLO is not the same thing as a benchmark failing to produce trustworthy evidence.

For optimization, we kept the search deliberately small. The planned experiment budget consists of a baseline, four candidates from a fixed 2×2 search, and an independent rerun of the winner. There's also a separate preflight smoke test.

We designed a dedicated subprocess for load generation, so the component producing requests would be separated from the API and dashboard. The intention is to make measurements more controlled and reduce the chance that the testing machinery itself distorts the results.

We're keeping the scope focused on one model and one serving environment rather than trying to support every possible combination from day one.

Challenges we ran into

One of the hardest parts was figuring out what a successful optimization should actually mean.

It's tempting to say that a configuration is better because it has higher throughput or lower latency. But real workloads involve trade-offs, and a configuration that breaks a hard latency requirement shouldn't win just because it performs well somewhere else.

Another challenge was dealing with unreliable measurements. A failed run, invalid metric, or overloaded load generator can make a configuration look worse than it really is. We needed to think about the quality of the evidence, not just the numbers it produced.

Scope was another big one. Supporting multiple models, serving engines, hardware setups, and search strategies would have made the project much larger. We chose to focus on a small, controlled experiment that we could reason about and eventually reproduce.

We also had to be careful about the gap between designing the system and actually validating it on live infrastructure. Having an architecture and a benchmark plan is not the same as having verified performance results, so we're treating those as separate milestones.

Accomplishments that we're proud of

I'm proud that SLOKern is being designed around making a defensible decision, not just generating a benchmark chart.

The separation between execution status, evidence validity, SLO feasibility, and independent verification is particularly important to us. A run completing successfully doesn't automatically mean its result is trustworthy, and a promising candidate isn't automatically a verified winner.

We also put effort into defining the experiment budget, data contracts, decision rules, and reporting expectations before expanding the implementation. That gives the project a clearer direction and makes it easier to identify what is implemented, what still needs work, and what evidence is missing.

There's still a gap between the intended system and a fully validated live deployment, but having that gap documented is useful in itself. We'd rather be honest about what works than claim improvements we haven't measured.

What we learned

We learned that optimizing inference is not just about finding the fastest configuration. It's about understanding what you're optimizing for and what you're willing to sacrifice.

We also learned how much the measurement process matters. If the workload isn't controlled, the metrics aren't valid, or the results can't be reproduced, then even an impressive-looking benchmark can lead to the wrong decision.

Another takeaway was that uncertainty needs to be part of the system. Sometimes the right answer is that a configuration violates the requirements. Sometimes it's that the evidence isn't sufficient to decide. Those shouldn't be treated as the same thing.

And probably the biggest lesson is that a smaller system with clear rules and trustworthy evidence is a better starting point than a huge optimizer that we can't properly validate.

What's next for SLOKern

Once we have the core workflow working reliably, there's a lot more we want to explore.

Right now, we're keeping SLOKern focused on a small, controlled search space. In the future, we'd like to expand that into smarter optimization strategies that can explore more configurations without wasting time and compute on options that are unlikely to work.

We also want to support more models, serving engines, hardware setups, and workload patterns. Different workloads have different needs, and a configuration that works well for one use case might not make sense for another. SLOKern should eventually be able to account for those differences instead of treating every inference workload the same way.

Another direction is cost-aware optimization. Performance is only one part of the equation. We'd like to help users understand the trade offs between latency, throughput, resource usage, and serving cost, while keeping cost estimates grounded in actual pricing data rather than guesswork.

Longer term, we see potential for SLOKern to become useful beyond one-off benchmark runs. It could compare results across deployments, detect performance regressions after configuration changes, and help teams understand when their serving setup no longer meets its SLOs.

The bigger vision is to move from a tool that helps find a good serving configuration to one that helps teams continuously understand, validate, and improve their inference infrastructure. There's plenty to build before we get there, but that's the direction we'd like to take SLOKern.

Built With

Share this project:

Updates

Submission history