inferit

Real World Problem Solved: Comparable inference output

inferit is a visual take on llm inference. Most inference frontends are limited to a single visual input/output "thread". This makes it hard to compare output from different models, prompts and sampler settings. inferit solves this with its UI that allows for an unlimited number of side-by-side generations. This makes it a perfect fit to compare and experiment with different models, prompts and sampler settings.

Beyond tuning

While created with the idea of easily doing side-by-side generations. I saw users including myself using it for more than that. It turned out that the visual playground is extremely helpful to group and organize conversations and thoughts in-pair with their llm output. Which makes it useful for any kind of thinking in collaboration with llms.

Also, if you do a tree view of this, and let each response generate its own trees (less so regarding the comparison aspect), you have yourself a major contender for creative writing -and- just plain LLM output management, that could possibly usurp some of the big names in LLM output (ooba, etc).

~ Comment in the local llama subreddit

Always online

Distributed as both, a web application as well as a browser extension. The browser extension is the preferred interaction method, since it can run completely offline.

As soon as the user goes offline inferit immediately falls back to Chrome's built-in llm (Gemini Nano) by leveraging the prompt api. Sampler settings can be customized like for any other local or remote llm that is plugged into inferit. When back online, the user's previous model selection is used, providing a seamless user experience.

Built With

• svelte
• typescript