AI Runtime Firewall

A centralized security gateway that protects AI applications by detecting sensitive data, blocking prompt attacks, enforcing security policies, and auditing every request before it reaches the LLM.

Comment

Inspiration

Most of the AI applications I have seen at work just fire requests straight at an LLM. It works, sure, but there is no real chokepoint — nowhere to enforce policy, inspect what's actually being sent, or go back later and see what happened. That got me thinking about what a centralized runtime security layer for AI applications would look like, so I built AI Runtime Firewall.

What it does

It sits between the app and the model, so every request has to pass through it first. It can inspect prompts for risky content, catch prompt injection attempts, mask sensitive data, and land on an Allow / Mask / Block decision. It also caches semantically similar requests (so you're not re-paying the token cost for near-duplicate prompts) and logs every decision so there's an actual audit trail afterward.

I also built a small sample RAG application to demonstrate how an AI application can integrate with the firewall in practice.

How I built it

Backend is ASP.NET Core, UI is Angular. The sample RAG app pulls documents from ChromaDB and routes everything through the firewall instead of hitting the model directly.

The flow: deterministic checks run first, then GPT-5.6 does contextual risk analysis on top of that, and the policy engine combines both to make the final call. Once something is approved, it goes to Ollama. Every step gets logged.

Challenges I faced

The processing flow took much longer than I expected. A blocked request, a masked one, a cache hit, a disabled app — those are four different paths, and earlier versions shared too much logic, which made audit logging inconsistent. Separating each request path into its own workflow took a few iterations.

I also went back and forth on where GPT-5.6 should sit. My first instinct was to just let it make the call directly, but that felt wrong — I didn't want a model quietly becoming the enforcement mechanism. So I pulled it back to contextual analysis only, and kept the actual allow/mask/block decision deterministic.

What I learned

Mostly that AI security isn't really "prompt filtering" the way I assumed going in. It's closer to normal app security plus governance and auditability bolted on — you need to know not just what got blocked, but why, and be able to show that later. Also learned more than I expected about wiring an LLM into a pipeline as an advisor rather than a decision-maker, which is a different design problem than I thought it'd be.

Accomplishments

Getting the whole thing to actually work end to end — RAG app, GPT-5.6 analysis, policy engine, cache, dashboard, all talking to each other through one firewall — without it turning into a pile of special cases. That's the part I'm most satisfied with. I also got it running in production rather than leaving it as a local demo, which forced me to deal with a bunch of rough edges I could've otherwise ignored.

What's next

There's a lot I still want to add. Supporting more LLM providers is probably next thing I would work on. Right now firewall talks to Ollama, but I would like to make switching between different model providers much easier. I also want to make the policy engine more configurable so organizations can define their own security policies without changing the code, and take a closer look at the semantic cache — cache invalidation is the area I think still needs the most work.

Beyond that: role-based access so it's not single-user, and a dashboard that does more than just show a log — actual analytics and reporting on what's getting blocked and why.

Built With

Share this project:

Updates