Agent-Depot

A platform where you can create, test, run and monitor AI agents without writing code. Every agent is a saved spec and one shared engine runs all of them.

Comment

Inspiration

We kept coming back to one question: what would a company need before it could trust an AI agent with its customers?

When we looked at how agents are usually built, we noticed two problems. The first is repetition: every new agent means connecting tools, configuring a model, writing instructions and setting up monitoring again from the start. The second, which we thought mattered more, is what happens after an agent goes live. A small change to its instructions can change how it behaves, and there is often no easy way to notice. A support agent that answered correctly one day could start describing a policy that doesn't exist, or take an action no one approved.

Many tools focus on making agents quick to create. We wanted to spend our time on the part after creation: reusing what's already been set up, checking changes before they go out, keeping people involved in risky actions, and being able to look back at what an agent actually did.

What it does

AgentDepot is a platform for building, testing, running and monitoring AI agents without writing agent code. We organised it around four questions we expected a company to ask.

Can we reuse it? An agent in AgentDepot is a saved configuration (its instructions, tools and safety rules) rather than separate code. One shared engine runs every agent, and tools are set up once so any agent can use them. A user can describe a goal in plain English, and Gemini drafts a configuration, explains why it chose each tool and suggests starter tests. Agents can also be created from templates or duplicated.

Can we trust a change? Saving an agent creates a new version and reruns its tests in what we call the Evaluation Lab. A test checks behaviour, such as which tools the agent should or shouldn't use and what its answer should include, and can optionally use Gemini to judge whether the answer followed policy. If a test fails, the server won't let that version be published. Earlier versions can be restored.

Can we stay in control? When an agent wants to take an action that changes something, such as sending an email, the run pauses and shows the exact action so a person can approve or reject it. We decided these rules should come from the platform rather than the model, since a model can misjudge what is risky. Each run also has limits on model calls, tool calls, estimated cost and time.

Can we explain what happened? Each run is recorded step by step, including every model decision and tool call, with inputs, outputs, timing, token usage and estimated cost. A runs page summarises success rate, duration and cost across agents and versions.

How we built it

AgentDepot is a TypeScript application built with Next.js and React, using Tailwind CSS and shadcn/ui for the interface and Better Auth for sign-in and separate workspaces. Gemini 3.1 Flash-Lite, through the Vercel AI SDK, is used to draft agents, call tools and grade tests. SerpApi provides Google Shopping, Google News and web page data. Data is stored in PostgreSQL using Prisma, and the app is packaged with Docker.

The core of the project is what we call the guarded agent loop: one runtime that runs every agent from its saved configuration. When the model asks to use a tool, the runtime first checks the agent's rules. The tool then either runs or the run pauses for approval. Each step is saved to the database before it is shown on screen, which is what allows a run to be replayed and an approval to survive a page refresh.

Tools share one common format in a registry, so adding a tool doesn't require changes to individual agents. We used Zod to validate data throughout the app, including Gemini's responses, because we didn't want to assume the model would always return exactly what we asked for.

Challenges we ran into

  • Pausing and resuming runs. An approval needed to survive a page refresh, and approving it needed to run exactly the action the person had reviewed. We handled this by saving each step before displaying it and storing the exact tool inputs with each approval.
  • Testing without side effects. We wanted tests to run the real agent rather than a simplified version, but without sending real emails. The test runner uses the same loop as normal runs, with write actions automatically denied.
  • Unpredictable answers. The same question can get different answers on different runs, which makes testing harder. We focused our checks on behaviour, such as which tools were used, instead of exact wording, and added the Gemini grader for questions about policy.
  • Not relying on the model for safety. Instructions in a prompt can be ignored or misread, so we enforced approvals, limits and publishing rules on the server instead.

Accomplishments that we're proud of

  • A user can go from a one-sentence goal to a working agent with starter tests in about a minute.
  • When a change breaks an agent's tests, that version is blocked from publishing and the previous version can be restored.
  • Safety rules apply whether or not the model asks for them.
  • Tests run through the same engine as real runs, so their results reflect how the agent actually behaves.
  • Two agents with different purposes run on the same engine and share the same safeguards.

What we learned

We came away thinking that the harder part of using AI agents in a company is building trust in them, not getting them to produce answers. Testing, approvals, run records and rollback ended up being more important to us than adding more ways to build agents, and we deliberately put our limited time there. Treating each agent as a versioned configuration made reuse and rollback much simpler than we expected. We also learned that because model output varies, safety decisions are better handled in code than left to the model.

What's next for AgentDepot

There is a lot we would still like to add:

  • More thorough testing: checking reworded versions of each test question, so an agent can't simply be tuned to one phrasing
  • Team roles: controlling who can approve actions, publish versions and manage agents
  • More integrations: connecting tools like email, Slack and CRMs through the existing registry
  • Sharing agents: letting teams share and build on each other's agents
  • Protecting personal data: removing personal information from agent outputs automatically

Built With

Share this project:

Updates

Submission history