Gateway

Inspiration

AI assistants are starting to read invoices and inboxes for us, but they can't tell a document's content from an order hidden inside it. This is an active research area, including here at UCSC. The AI Security reading group, sponsored by Professor Alvaro Cardenas, studies how LLM-powered computer-using agents are both a new tool for defense and a new target. Its reading list covers indirect prompt injection (Greshake et al.), the OWASP Top 10 for LLM Applications, and benchmarks like AgentBench, WebArena, and ToolEmu that test what agents do when something goes wrong. The same problem shows up in all of it: once an agent can act, untrusted text becomes a way to attack it. We wanted to see how bad that gets, and then build something that stops it.

What it does

Protection Gateway checks a document before an AI agent acts on it. You upload a text file, email, or folder and pick a task, like extracting invoice fields. The gateway inspects the content and either releases a source-backed result or holds it for review. Text inside a document can never change the task or approve an action.

How we built it

Isolated, sandboxed workers inspect each document. Three inspectors (instructions, data and tools, behavior) run on Kimi K3, and a separate core makes the final decision with a fixed rule table. Workers talk to the core over a SLIM transport. The core is plain Python standard library. Email actions are simulated, and sending needs a separate typed action plus the user's exact confirmation. We added a browser UI, a read-only MCP server, and a replay mode so the demo runs offline.

Challenges we ran into

Models sometimes return malformed reports, so we validate every one and allow only two retries. When anything fails, the content stays held instead of slipping through. We also had to avoid false alarms: six of our benign test documents use words like "ignore" and "send" on purpose. Making the demo repeatable was its own project.

Accomplishments that we're proud of

We red-teamed a plain agent with 10 realistic attacks, like a fake compliance rule in an invoice footer. It took the harmful action in 14 of 30 runs. In one case it paid the attacker's account, then told the user "Payment not processed." The gateway held 7 of the 10 inputs, and none of the ten could cause harm.

What we learned

Papers like ToolEmu and AgentBench measure what agents do when they're tricked. We ran a small version of that test ourselves, and the results matched the research: a plain agent took the harmful action in 14 of 30 runs. We also learned that detection will miss things. Three of our attacks looked clean to every inspector. They were harmless anyway because inspected tasks can't pay, fetch links, or send email on their own. Strong limits matter more than a perfect detector.

What's next for Gateway

Support PDFs and images, which are currently flagged as uninspected. Then add a production deployment setup and real email sending behind the same confirmation step. After that, we want to cover the paths outside our app, like native chat uploads.