LLMFit

Inspiration

Running LLMs locally is often a guessing game. A model may look like a good fit based on its parameter count and memory requirements, but actual performance can vary significantly depending on the CPU architecture, available RAM, GPU, and runtime.

We wanted to make that decision hardware-aware and evidence-driven, especially for ARM-based machines where benchmarking data can be limited. The goal was simple: instead of asking "Which model should I run?", help answer "Which model actually fits and performs well on my machine?"

What it does

LLMfit analyzes the user's hardware and helps find suitable LLMs for that specific machine.

It:

• Detects hardware architecture, CPU, RAM, GPU availability, and runtime availability.
• Analyzes thousands of models and determines which ones can realistically fit.
• Recommends appropriate quantization and context limits.
• Supports Ollama-based model benchmarking.
• Measures real-world inference throughput in tokens per second.
• Stores benchmark results locally.
• Matches benchmark results back to the corresponding model and hardware.
• Distinguishes between measured performance and predicted performance.
• Provides an interactive TUI for browsing models and inspecting their hardware fit.

For example, on our ARM64 machine, LLMfit measured Qwen 2.5 1.5B at 16.87 tok/s and identified it as an exact hardware match rather than treating it as another generic estimate.

How we built it

LLMfit is built in Rust, with the core model-fitting and hardware evaluation logic separated from the TUI.

The main flow is:

Hardware detection → Model fitting → Benchmarking → Hardware matching → Performance evidence → Recommendation

We extended the existing architecture with ARM64-aware hardware matching, including architecture normalization for aliases such as arm64 and aarch64.

For benchmarking, we integrated Ollama and created a local benchmark evidence pipeline. Benchmark results are associated with model identity, runtime, and hardware so that measured throughput can feed back into model selection.

We also added an optimization layer that keeps:

• best_measured — backed by real benchmark evidence
• best_predicted — based on performance estimation

The interactive interface is built using Ratatui, allowing users to browse models and inspect their fit directly from the terminal.

Challenges we ran into

The biggest challenge was making sure a benchmark result didn't simply become a number attached to a model.

We had to solve several identity and hardware-matching problems:

• Ollama runtime tags such as qwen2.5:1.5b needed to resolve to the correct catalog model.
• ARM systems can report architecture differently (arm64 vs aarch64).
• ARM CPUs can have different vendor-specific names while still belonging to the same architecture class.
• Benchmark evidence needed to survive the entire path from local storage → model matching → optimizer → final recommendation.
• We had to preserve compatibility with older benchmark schemas and existing x86 hardware behavior.
• Finally, we had to make the measured-vs-predicted distinction clear enough to be useful in the TUI.

Accomplishments that we're proud of

We're especially proud that LLMfit doesn't just predict performance anymore — it can use real measurements from the user's machine as evidence.

On our ARM64 environment:

• Hardware was detected as AArch64 / Neoverse-N1.
• Qwen 2.5 1.5B was benchmarked through Ollama.
• Three benchmark runs produced an average of approximately 16.87 tok/s.
• LLMfit correctly attached that result to the model.
• The result was classified as an exact hardware match.
• The optimizer exposed it as best_measured with performance_provenance: measured.

We also maintained a fully tested Rust implementation with the core and CLI test suites passing after the changes.

What we learned

We learned that model selection and model benchmarking shouldn't be separate features.

A theoretical estimate is useful when no benchmark exists, but once real performance data becomes available, the system should be able to trust and reuse that evidence.

We also learned that hardware identity is more complicated than simply comparing CPU names. Architecture, hardware class, runtime, and benchmark provenance all matter.

Most importantly, we learned that a good LLM recommendation system should communicate why it recommends a model — and whether that recommendation is based on an estimate or something the machine has actually measured.

What's next for LLMFit

The next step is to turn LLMfit into a continuously improving hardware-performance knowledge base for local AI.

We're interested in:

• More ARM64 hardware and cloud instances.
• Community benchmark sharing across machines.
• More inference runtimes beyond Ollama.
• Better model/runtime compatibility detection.
• More sophisticated hardware similarity matching.
• Using accumulated benchmark evidence to improve predictions.
• A richer dashboard for comparing models and hardware.
• Eventually, automatically benchmarking and calibrating recommendations as new hardware is detected.

The long-term vision is for LLMfit to become the "Can I run this model, and how fast will it actually run?" layer for local and edge AI.

Built With

• cli
• opensource
• rust