Foxy Audit

Stop promising your AI is safe. Start proving it with cryptographic proof, not just your word.

Comment

About the Project

Inspiration

Every AI startup selling into healthcare, finance, or legal eventually hits the same wall: a buyer's security team asks, "Can you prove your AI didn't leak or tamper with our data?" The answers founders usually have on hand are weak "just trust us," a slow five-figure manual audit that's outdated the day it's finished, or a governance platform that ironically forces you to upload your most sensitive raw logs to its cloud just to watch them.

That last part is what really bothered us. Every existing "solution" solves a privacy problem by creating a bigger one. We wanted to know: could you generate cryptographic proof that an AI system behaved — without ever needing to see, store, or transmit the actual conversations? That question became Foxy Audit: a privacy-preserving runtime evidence layer, not another dashboard that asks for your trust.

What it does

Foxy Audit wraps an AI call with a lightweight SDK that:

  • Blocks or redacts sensitive data locally — PHI, PII, secrets, and prompt-injection attempts — before it ever leaves the host.
  • Hashes every interaction into a content-blind 64-character digest. The raw prompt or response is never sent to us.
  • Chains every event into a tamper-evident ledger using a simple, verifiable construction:

$$H_i = \text{SHA256}(\,data_i \,|\, H_{i-1}\,)$$

Change any earlier record, and every hash after it breaks — detectable instantly, and independently checkable, since we anchor the chain head to a public blockchain (Sepolia) rather than asking anyone to trust our own database.

That evidence rolls up into a Compliance Passport: a signed report with the audit summary, exactly which policies fired, and full chain verification — safe to hand to a customer's legal or security team. Buyers also get a live dashboard (capture coverage, threat analytics, an audit ledger) and a desktop companion that reacts to audit events in real time.

How we built it

The core is a Python SDK (@foxy.audit) that instruments a model call, run against a live backend at app.foxyaudit.tech. On top of that sits a Next.js dashboard, a policy/detection engine (HIPAA, GDPR, secrets, and injection detectors), a SHA-256 hash-chain service, and a public-chain anchoring job.

Built with GPT-5.6 and Codex

Two OpenAI tools did most of the heavy lifting during Build Week:

  • GPT-5.6, via the OpenAI Responses API, is the pluggable "AI Judge" inside the product itself — it grades interactions against policy (HIPAA, GDPR, custom rules) and produces the verdicts that get hashed into the chain. Judges are swappable (Gemini, GPT-5.6, or a customer's own key), but GPT-5.6 is our default and the one wired into the live demo.
  • Codex was our build partner, not just a code-completion tool. We used it to analyze the existing repo end-to-end, harden the evidence boundary (making sure nothing that touches raw prompt text can accidentally leak into logs, alerts, or the passport export), add the no-LLM judge test path so anyone without an API key can still run the full test suite against the live backend, and — just as importantly — go back through our own marketing copy and strip out claims we couldn't actually back up in the code. That last part mattered a lot to us: we wanted "evidence layer, not a compliance guarantee" to be something Codex helped us enforce, not just a tagline.

Between the two: GPT-5.6 is the brain doing the judging at runtime, and Codex was the pair-programmer that made sure the system around that judging was honest and airtight.

Challenges we faced

  • Proving a negative. "We don't see your data" is easy to claim and hard to prove. We had to design the SDK so that content-blindness wasn't a policy promise but a structural fact — the raw text is hashed and discarded on the client before a network call is even made.
  • Making "unknown" honest. Early on, the evaluator either passed or failed every event. But an AI judge can be unavailable, time out, or contradict itself. We added a third, explicit state — "Evaluator Could Not Determine" — so an outage never silently gets counted as compliant. That was a small feature with an outsized amount of debate behind it.
  • Judge-friendly testing without an LLM key. Not every reviewer running our demo has an OpenAI key on hand. We built a no-LLM test script that still exercises the real SDK path — blocking, redaction, hashing, chain verification — against the live backend, so what a judge sees isn't seeded or faked data.
  • Keeping the marketing honest. It's tempting to call an evidence layer a "compliance guarantee." We deliberately don't, and spent real time (with Codex's help) making sure every sentence on the sales page and in the Compliance Passport is something a reader could go verify in the code or the git history themselves.

What we learned

Privacy-preserving evidence is a genuinely different design problem from ordinary logging — you have to design for what you refuse to collect just as carefully as what you do. We also learned that pairing an LLM-as-judge (GPT-5.6) with a deterministic, cryptographic backbone (the hash chain) gives you the best of both: judgment where you need nuance, and math where you need certainty.

Built With

Share this project:

Updates