LLMDash OpenAI Cache

Inspiration

While developing our own GenAI applications, we started to realize how our OpenAI spend was quickly skyrocketing- even though most of our queries were the same. The APIs for our LLMs are a core part of our products as developers and are used with high frequency- especially as we iterate and develop. We recognized a distinct lack of performant, purpose-built caching software for LLMs- which is why we created one in Rust. We want to go above and beyond, by building an all-in-one tool for LLMs that can run within the comfort of Docker Desktop. Our vision is to build a studio for prompt engineering, caching, logging, local LLMs, and more- everything a developer needs to succeed with GenAI on Docker.

What it does

The extension is designed around LLMDash, a caching system written in Rust. It's fully compatible with the OpenAI API- the developer simply points their SDK to the cache's URL. It supports two modes of operation: semantic caching (similarity between queries), and exact caching (same queries).

The Docker extension consists of two components: real-time stats, and a developer-centric chat client specialized for prompt engineering. The interface provides quick installation and configuration of the cache. The chat client stores LLM parameters (temperature, model, etc) per-conversation, empowering developers to easily organize and iterate on their prompts.

How we built it

The extension's frontend is built with React & MUI- we designed it support all screen sizes and system themes. It serves live statistics through local instances of Prometheus and Grafana that scrape the cache for metrics. The backend is programmed in Golang, with a specially designed long-polling system that streams asynchronous chat & conversation updates in real-time.

Accomplishments that we're proud of

We developed a super-high-quality chat client that rivals ChatGPT. The product we've built is fully functional and will provide great utility to developers out-the-box.

Challenges we faced

Getting up and running with the Docker extension SDK was easy- we only ran into minor challenges. Not having a xs (0 px) breakpoint with the Docker MUI theme was very frustrating. It would have also been nice to have fine-grained controls over request headers sent to our extension's backend.

What's next for LLMDash OpenAI Cache

This isn't the end for LLMDash- we're going to continue adding features that will make it the go-to tool for LLMs; even for those that aren't already using Docker Desktop. Our vision is to facilitate every necessity a developer needs when building GenAI applications. Here's some of our plans:

Local LLMs: Integrate GPT4All models with the Huggingface inference server to download and serve models locally. We'll provide a unified API (OpenAI) for all LLMs.
Logging: Deep-dive into every request with advanced metrics
RAG: We'll integrate Modela, our AutoML platform with a purpose-built RAG engine based on LlamaIndex. Drag-and-drop documents or use a vast array of data connectors. Fine-tune your RAG's performance with advanced query engines