Inspiration
Especially in the modern era, agents are handling more and more complex tasks which involves making many tool calls sequentially, potentially taking a long time. But what if your agent already had the information?
What it does
When an agent takes in a prompt, it often has a chain of thought before making a sequence of MCP tool calls. We read that, predict which MCP tool calls it will make, and fetch them all in parallel before execution to put in a cache. When the agent starts executing, it reads the cached responses ,which makes it much faster.
How we built it
A drop-in MCP proxy that can be used with any agent making any types of MCP tool calls. It uses multiple systems like a small classifier AI model, and a mathematical Markov model, to predict the tool calls in advance based on the chain of thought.
Challenges we ran into
Building the system to be effective with any type of MCP tool calls was difficult, as different types of prompts can lead to very different tool calls.
Accomplishments that we're proud of
We are proud of building something that generalizes fairly well in testing, and can have very significant speedups (3x or faster) in complex prompts.
What we learned
We learned a lot about the middleware layer between the agent and the tool it is accessing.
What's next for engram
Researching more advanced systems to make the tool call predictions more accurate.
Log in or sign up for Devpost to join the conversation.