PromptShield

PromptShield is an LLM firewall on Nemotron that screens every prompt, blocks injections before they reach your app, and proves it with live protected vs unprotected demos

Comment

Inspiration

Every team building an LLM app eventually hits the same problem: the model will follow instructions from anyone who can type into it, not just the developer.

A user prompt can say "ignore your previous instructions", paste a fake system message, or ask the model to reveal data it should not reveal, and a normal chatbot will often just comply. That is not a theoretical risk. It is the reason small teams hesitate to ship LLM apps at all: one clever prompt can undo all the product work around the model.

PromptShield started from a small, public experiment. I built a prompt-injection detector with TF-IDF and Logistic Regression, tested it on a small synthetic set, and shared it openly. The feedback that changed the project came from that post: test it on a real external set, report false positives on benign prompts separately from accuracy, and try rewritten injections that do not use the usual trigger words.

That feedback became the whole project. A detector that only works on synthetic prompts, or that blocks normal users to look safe, is not a firewall. It is a demo.

What it does

PromptShield is an LLM firewall on Nemotron, built on Nebius Token Factory.

A prompt goes through the guard before it reaches your app's model:

  1. Nemotron Nano screens every prompt: fast, cheap, first pass.
  2. Nemotron Super analyses flagged or ambiguous prompts: semantic understanding, not just keyword matching.
  3. Nemotron Ultra adjudicates high-risk cases only and writes a plain-English explanation for why a prompt was blocked.

That tiering matters. Running the biggest model on every prompt is slow and expensive. Running only a keyword classifier is brittle. PromptShield routes by risk, so most prompts stay fast and the expensive reasoning is spent where it actually changes the decision.

The baseline TF-IDF detector runs beside Nemotron in the demo, so you can see the difference directly, especially on rewritten injections that avoid trigger words.

The demo also shows the point that matters most to a developer: the same injection that succeeds against an unprotected app is blocked against a protected one, with the reason shown. A benign prompt is allowed through. Blocking everything is easy. Blocking attacks without blocking normal users is the hard part, and that is what the dashboard measures.

How we built it

  • Nebius Token Factory for Nemotron inference, using its OpenAI-compatible API. "Runs on Token Factory" in this project means a real runtime call to the Token Factory inference API, not a screenshot of a model name.
  • NVIDIA Nemotron at three tiers: Nano for screening, Super for semantic analysis, Ultra for adjudication and explanation.
  • Python and Flask for the guard API and demo app, with a middleware-style design so an existing LLM app can add the guard without rewriting its chat flow.
  • TF-IDF + Logistic Regression as the baseline detector, carried over from the original public experiment.
  • An evaluation dashboard that reports accuracy, false positives, and false negatives separately, plus latency and cost per 1,000 prompts by tier. False positives get their own number because a guard that blocks 5% of normal users will not survive in a real product, even if its headline accuracy looks good.

Challenges we ran into

Synthetic accuracy is misleading. The first detector scored 100% on a held-out synthetic split, but that split was only 10 prompts. That number looked great and meant almost nothing. On a real external set, the picture changed completely: 67.2% overall, with 3 false positives out of 56 benign prompts (5.4%), injection recall of 25 out of 60, and only 6 out of 10 rewritten injections caught, with the 4 misses listed openly.

That gap is the reason PromptShield exists. Keyword and TF-IDF methods catch obvious injections and miss rewritten ones. Semantic screening with Nemotron is there specifically for the cases the baseline misses.

False positives vs. false negatives is a real trade-off, not a slider you can max out. Block too aggressively and you break normal use. Block too loosely and injections get through. There is no threshold that makes both numbers perfect, so the project does not hide behind a single accuracy figure. It shows both, and lets the developer choose a policy for their app.

Cost and latency are part of the design, not an afterthought. If every prompt needs the largest model, the firewall becomes too slow and too expensive to ship. Tiered routing (Nano first, Super next, Ultra only for adjudication) is how PromptShield keeps the common case fast while still giving a reasoned explanation for the hard cases.

A guard is a layer, not a guarantee. PromptShield does not claim to stop every possible injection, and it does not claim real-world fraud or security proof from synthetic data. It is a screening and explanation layer that makes LLM apps measurably safer to ship, with its limits stated in the open.

Accomplishments that we're proud of

  • A working guard with a clear, demoable story: blocked injection vs. allowed benign prompt, side by side, with a reason.
  • An honest evaluation habit carried through from the first public post: synthetic results labelled as synthetic, external-set results reported with false positives separate, and misses listed instead of hidden.
  • A design that uses Nemotron for different jobs at different tiers, rather than using one model name as a label.
  • A project that a developer can actually integrate, with an API and middleware pattern, not just a notebook, and a dashboard they can use to judge whether the guard is helping or hurting their app.

What we learned

  • Accuracy alone is a bad metric for security tools. False positive rate, false negative rate, and performance on rewritten attacks tell you far more.
  • Real data humbles a model fast. A 100% synthetic score falling to 67.2% on an external set was the most useful result in the whole project, because it forced the design to change.
  • The best feedback is specific and checkable: "test on this set, report this number separately, try these rewritten cases." That is now how the evaluation suite is built.
  • Open models on open infrastructure are only useful to small teams if trust tooling ships with them. A firewall is not a side feature for LLM apps. It is part of shipping.

What's next

  • An adversarial self-test loop: Nemotron generates new rewritten injection variants, the guard is tested against them, and failures feed the dataset, so the guard gets harder to beat over time.
  • Tavily enrichment for the latest injection techniques and research, so guard patterns do not go stale.
  • A packaged SDK and clearer integration docs, so adding PromptShield to a Nebius LLM app is a few lines, not a fork.
  • More evaluation sets, with the same rule as always: report false positives and false negatives separately, and list the misses.

Demo: The demo runs locally from the repo. There is no hosted demo URL. Try it out: https://github.com/Duke2nuke/promptshield

Built With

Share this project:

Updates

Submission history