-
-
image1
-
image2
-
image3
-
image4
Inspiration
Developers paste API keys, customer emails, card numbers and database passwords into AI assistants every day. Once that text reaches a hosted LLM provider, it can be logged, retained or exposed in a breach, and you cannot take it back. A hosted model has to read the text it answers, so true end-to-end encryption is not possible. We asked a more practical question: how do we make sure the provider only ever reads what it truly needs?
What it does
VeilProxy is a local gateway that sits between any OpenAI-compatible client and OmniRoute, which routes to hundreds of models. For every request it:
- Detects secrets and personal data: API keys, JWTs, private keys, card numbers (Luhn-checked), IBANs, emails, phone numbers (including Nigerian formats), IPs, URL credentials and passwords.
- Encrypts each value locally with AES-256-GCM and sends the model only placeholders such as
[EMAIL_1]and[CARD_1]. - Rehydrates the model's reply locally, including streamed replies, so you see the real values and the provider never does.
- Blocks the request (fail-closed) if a detected secret somehow survives redaction.
ShardKey protects the vault key. The master key is generated inside a Rust core, split with Shamir's secret sharing (3-of-5) across separate custodian processes, and held only in zeroized memory after a threshold of custodians approve. Locking the vault, or an idle timeout, wipes it. With the vault locked, nothing is forwarded at all.
A three-pane terminal UI shows what you see, what the provider receives, and the vault status with detections, side by side.
How we built it
- Rust core (PyO3 and maturin): AES-256-GCM, HKDF per-session subkeys, HMAC pseudonym tags, and our own Shamir implementation over GF(2^8) with a branch-free multiply and zeroize-on-drop key custody. The key never crosses into Python.
- Python gateway (FastAPI and httpx): an OpenAI-compatible
/v1/chat/completionsendpoint with streaming, forwarding to OmniRoute. - Detection engine: regex rules with checksum validators, entropy scanning, optional Presidio/spaCy for names and places, and YAML custom rules and allowlists.
- Vault: SQLite storing only ciphertext and keyed tags; custodian processes that release shards only on approval; idle auto-lock.
- Textual TUI with
/unlock,/lock,/demo,/copyand live detections. - Testing: 7 Rust unit tests, 67 Python tests, a randomized 400-prompt leak test, a Rust-to-Python interoperability test, and property-based tests that split streamed replies at arbitrary points.
Reconstruction uses Lagrange interpolation at zero over GF(2^8):
$$ s = f(0) = \sum_{i=1}^{k} y_i \prod_{j \ne i} \frac{x_j}{x_i \oplus x_j} $$
The key is rebuilt only when at least $k = 3$ of $n = 5$ shares are supplied, and a key-check value confirms it is correct.
Challenges we ran into
- Streaming: a placeholder like
[EMAIL_1]can arrive split across chunks. We built a buffering rehydrator and property-tested it against arbitrary chunking. - Our own safety net had a bug. The fail-closed guard flagged a model name as a "secret" because of its entropy and blocked a legitimate request. We fixed it so the guard scans exactly the fields the redactor handles, and tightened the entropy rule.
- False positives that leave fragments: a card number that failed the Luhn check was being partially redacted as a phone number. We added context checks.
- The terminal UI: the log widget's default minimum width caused sideways scrolling in narrow panes. We fixed wrapping, reserved scrollbar space, and re-wrap content on resize.
- Honest scope: a hosted LLM must read the prompt, so we protect secrets and identifiers rather than claiming impossible end-to-end encryption.
Accomplishments that we're proud of
- The master key never exists in Python memory, and nothing is forwarded while the vault is locked.
- Real calls to real models through OmniRoute, with a leak check showing no secret reaching the provider.
- Rust and Python backends that interoperate byte-for-byte, with tests proving it.
- A fail-closed design that has already caught its own bugs.
What we learned
- Safety checks must look at exactly the same data as the thing they guard.
- Fail-closed is the right default for security tooling, and it needs tests for false positives too.
- Threat models need to be stated plainly. Our limits: anything the detector misses is still sent, placeholders reveal the type of data, and custodian shards travel over localhost.
What's next for VeilProxy
- An encrypted (FHE) sensitivity classifier to decide which prompts go to the cloud and which stay on a local model.
- TLS or hardware-key custodians for multi-machine use.
- Proactive share refresh, and name and organization detection enabled by default.
- A browser extension and an IDE plugin on top of the same gateway.
