Inspiration
We all use AI to study, build projects, learn new skills, and solve problems. But every time an LLM receives a prompt, it has to process the context we send with it.
That becomes expensive fast.
Modern AI applications often use retrieval-augmented generation, or RAG, to pull information from documents before sending it to an LLM. The problem is that retrieval can return far more information than the model actually needs. Every irrelevant token still gets processed. Every irrelevant token can still cost money.
We started asking a simple question:
What if AI only had to read the context that actually mattered?
That became Uma.
What it does
Uma is a context optimization layer for AI applications.
It sits between retrieval and the final LLM call. Instead of blindly sending all retrieved context to the model, Uma analyzes that context first. It scores individual sentences based on how relevant they are to the user's question, removes low-relevance information, then passes the smaller context to the same LLM.
The user still asks the same question. The application can still use the same model.
The difference is that the model has much less to read.
In our benchmark, Uma reduced retrieved context by 52.3%. Total token usage fell by 43.0%, while estimated API cost fell by 30.4%. The factual answer was preserved in the benchmark.
Uma's goal is simple:
Stop paying AI to read what it doesn't need.
How we built it
Uma works as a preprocessing layer before LLM inference.
First, an application's RAG system retrieves context normally. Uma receives that retrieved text along with the user's query.
The retrieved context is split into individual sentences. A local cross-encoder relevance model compares each sentence with the user's question and assigns it a relevance score.
Uma then applies a threshold to those scores. Low-relevance sentences are removed while the strongest context is preserved.
Only that optimized context is passed to the LLM.
Because Uma changes the context before generation rather than modifying the model itself, the system is model-agnostic and can work with OpenAI-compatible model APIs.
We also built benchmarking into the project so that we can compare a normal LLM request against an Uma-optimized request using the same query, model configuration, plus original retrieved context.
Challenges we ran into
Our biggest challenge was realizing that simply removing tokens is not enough.
A system that cuts 90% of a prompt but destroys the answer is useless. We needed to optimize context while preserving the information required for the model to solve the task.
That meant experimenting with relevance thresholds and sentence-level filtering. We also had to create a fair benchmark where the only meaningful difference between the two requests was whether Uma filtered the context first.
Another challenge was designing Uma to work independently of a specific LLM provider. Instead of building optimization directly into one model, we placed Uma before the model call so the same idea could work across different OpenAI-compatible systems.
Accomplishments that we're proud of
Our biggest accomplishment is that Uma produced measurable savings on a controlled benchmark while preserving the factual answer.
In our test:
52.3% less retrieved context reached the model.
43.0% fewer total tokens were processed.
30.4% lower estimated API cost was achieved.
Most importantly, both versions returned the same key factual answer.
We're also proud that Uma does not require developers to build a new model. It is an optimization layer that can sit inside an existing AI pipeline.
That means the idea can potentially scale from a small RAG application to much larger AI systems where millions of unnecessary tokens can translate into significant infrastructure costs.
What we learned
Building Uma changed how we thought about AI optimization.
When people discuss making AI cheaper, the conversation often focuses on using smaller models or cheaper providers. Uma approaches the problem from another direction.
Sometimes the model is not the problem.
The context is.
We learned that improving an AI system does not always mean giving it more information. Giving a model less information can sometimes make the system significantly more efficient, as long as the information being removed is actually irrelevant.
We also learned how important controlled benchmarking is. Token reduction alone does not prove that an optimization works. Answer quality has to be evaluated alongside efficiency.
What's next for Uma
Uma currently proves the core idea: relevant context can be selected before inference to reduce unnecessary token usage.
The next step is making that selection smarter.
We want to experiment with adaptive relevance thresholds, stronger retrieval models, semantic caching, plus learned context selection. We also want to benchmark Uma across larger datasets and different LLM providers to understand how aggressively context can be reduced before answer quality begins to decline.
Long term, we see Uma becoming an infrastructure layer that sits between AI applications and their models.
Every request would ask one question before expensive inference begins:
What is the minimum context this model actually needs?
Log in or sign up for Devpost to join the conversation.