Inspiration
Local models are crucial to many developer workflows. They allow developers to work without a network connection, avoid per-token API costs, and give more control over how they build. Choosing which local model to run still comes with many pain points. A single model release can include a dozen quantized files. Should I download the 4-bit version or the 5-bit version? Is the larger model actually useful on my machine, or will it just be slow? Will it fit at the context length I care about?
Right now, I usually fall back to abbreviated quant notes from uploaders such as bartowski on Hugging Face, then search Reddit and hope someone has posted tokens-per-second numbers from a setup similar to mine. Those posts are useful, but they rarely tell me everything I would need to reproduce the result. I often do not know the exact model file, runtime, settings, prompt, context length, or whether the model was actually good at the task instead of merely fast.
There have been good capability benchmarks. I used to look to projects like the Open LLM Leaderboard, and Artificial Analysis is now a strong example of reliable benchmarking presented in a way that is easy to compare. That work is an important inspiration for InferGrade.
The gap is that those benchmarks are not trying to tell me which quant to download or how it will perform locally on my hardware. A model can look great in a general capability comparison and still be the wrong local choice because it does not fit, runs too slowly, or has a nearby quant that offers a better tradeoff. I wanted a way to run the actual model and quant on the actual machine, then get an answer based on what happened.
That is why I started InferGrade. The question I want it to answer is simple: “Which quant of which model should I run on my hardware?”
What it does
The InferGrade Runner runs quantized GGUF models on the machine being tested. On Apple Silicon it can use native llama.cpp with Metal. It also has container-based paths for benchmarks to run in isolation on Nvidia and AMDI GPUs, but Apple-metal accelerated containerization is not an option today.
We measure more than tokens per second. Depending on the test, the report includes whether the model completed the task, time per task, time to first token, generation speed, memory use, and perplexity or quant-fidelity results. Runner writes a Markdown report that a user can read directly, along with a structured result bundle that records the model file, hardware, benchmark, scoring rules, and runtime used for the run data to be ingested.
There are short tests for making a practical decision and deeper tests for investigating a model more carefully. I do not want a five-minute local check to be presented as if it proved which model is best everywhere.
How I built it
The main execution code is Python. I use Rust for the engine, CLI, credentials, pairing, and managed-runtime work. The desktop app is Tauri with a small JavaScript and Vite interface. llama.cpp runs the GGUF models, Docker handles several isolated benchmark paths, and JSON Schemas define the result format. The Runner started in Python but migrated to Rust for "Install and Go" functionality without a user manually installing dependencies.
InferGrade Runner existed before Build Week. During July 13–21, I used Codex with GPT-5.6 Sol to make a large pass over the public Runner. I would bring it a product problem or a suspicious result, and Codex would investigate the code and real run artifacts, implement a fix, add tests, prepare pull requests, and help verify the release. I still made the product calls, set the evidence and security boundaries, and decided what could be merged or released.
The public release moved from 0.3.13 to 0.3.39 during that period. Some of the most useful work came directly from dogfooding. When an older Qwen2.5 7B model appeared to score 100/100, I knew the benchmark was saturated and the presentation was misleading. I replaced that component with a harder test and removed the suggestion that reaching a suite ceiling meant a model was perfect. When Qwen3.5 spent its whole answer budget inside a thinking block, I changed the Runner to use the model’s structured chat protocol. When long benchmarks were interrupted, I added per-case checkpoints so completed cases could be reused safely instead of starting from zero.
I also added exact benchmark fingerprints, one immutable llama.cpp runtime lock per run, path-free runtime receipts, and a signed runtime catalog. The catalog lets Runner learn about a reviewed runtime without silently trusting or installing whatever happens to be newest upstream. GPT-5.6 helped me implement and test this work; it is not a model that runs inside InferGrade.
Challenges I ran into
The hardest part was deciding what a failed result actually meant. A model that gives a malformed answer should usually be scored as wrong. A runtime that crashes should leave the benchmark incomplete. Treating those as the same failure either flatters the model or throws away valid evidence.
Real runs found problems that unit tests alone would not have caught. One 27-minute benchmark was recorded as taking 30 seconds because of a placeholder. An EvalPlus subset run accidentally asked the upstream evaluator to load the full dataset. Some newer model architectures would only reveal an incompatible llama.cpp build after expensive work had already started. Each of those became a concrete fix and regression test.
Runtime identity was another surprisingly deep problem. A label like llama.cpp b10069 is not enough to prove which executable and libraries produced a result. Runner now locks the exact runtime content for the run, checks it again after measurement, and records a receipt without publishing private filesystem paths.
Accomplishments that I’m proud of
I am proud that Runner produces something I can audit instead of only printing a score. A result can carry the exact model artifact, benchmark protocol, runtime identity, hardware path, and scoring outcome that produced it.
I also built a bounded autopilot for benchmark work. It can run jobs I authorize, but it cannot quietly choose a different artifact, expand its download budget, change publication settings, rent hardware, or grant itself more work.
The signed runtime catalog was the biggest architectural addition. It has expiry and rollback checks, revocation, explicit install consent, offline last-known-good behavior, and separate signing responsibilities. I used it to prove exact Gemma 4 and MiniCPM5 artifacts against a reviewed runtime on my M1 Pro instead of making a broad compatibility claim.
The submitted Runner is public and Apache-2.0 licensed. Judges can inspect the implementation and run the focused tests without access to my private Hub repository.
What I learned
Tokens per second is not enough to choose a local model. In one quant comparison, the Q8 model decoded more slowly than Q5 but finished the task sooner because it generated fewer tokens. Time per task told the more useful story.
I also learned to be suspicious of a clean-looking score. If a small older model gets 100/100, that may say more about the benchmark ceiling than the model. Capability, speed, memory, output length, quant fidelity, and the strength of the evidence all need to stay visible.
Finally, GPT-5.6’s autonomy worked best when I gave it real evidence and firm boundaries. It could carry a problem through investigation, implementation, tests, review, and release, while the contracts and approval points kept the result inspectable.
What’s next for InferGrade Runner
The next goal is to make the first successful run boring: download the app, pair the machine, pick a model and quant, run it, and get a useful report without needing terminal knowledge.
I also want to add a stronger Windows and NVIDIA path, test more current model families and neighboring quants, and collect repeated results on more hardware. The goal is not to declare one universally best model. It is to give someone enough real evidence to make a good decision for the machine and task they actually have.
Log in or sign up for Devpost to join the conversation.