We will be undergoing planned maintenance on Oct 7th 6:00AM UTC / Oct 7th 2:00AM ET

LoopHole AI

Every law has a loophole. We find it first.

Comment

Inspiration

Software engineers would never ship critical code to production without CI, penetration testing, and fuzzing.

But governments, regulators, and compliance teams routinely deploy high-stakes laws and policies without systematically testing how adversarial actors might exploit them.

Sophisticated actors rarely break rules outright. They comply with the literal wording while defeating the underlying intent.

California's 2018 Consumer Privacy Act is a classic example. The CCPA restricted the "sale" of personal data for "monetary or other valuable consideration." Ad-tech networks found an opening: instead of paying cash, they exchanged data through reciprocal advertising networks.

The conduct survived the wording even though it undermined the policy goal. California later had to expand the law through the CPRA and introduce the separate concept of "sharing."

That exposed the core problem:

Rules are deployed before they are adversarially tested.

As autonomous AI agents begin operating at machine speed, regulatory arbitrage can happen faster and at far greater scale.

So we asked:

What if we could fuzz legislation before deployment, the same way engineers fuzz software?

That became Loophole.

What it does

Loophole is an adversarial red-team War Room for legislation, regulations, and corporate policies.

It does not try to be a generic chatbot or an "AI lawyer."

Instead, it turns rule-making into something that can be attacked, tested, repaired, and attacked again.

Our core principle is simple:

AI argues. Code checks the quotes. Humans decide.

1. Define the intent

The user pastes a draft bill, regulation, or policy and creates a Purpose Contract:

  • What is this rule trying to prevent?
  • What legitimate behavior must remain allowed?

This gives the system both a target and a safety boundary.

2. Launch the red-team attack

An autonomous Attack Agent probes the text across 9 adversarial evasion lanes, including:

  • threshold splitting
  • relabeling
  • uncompensated sharing
  • nominal oversight
  • structural workarounds

Instead of producing vague legal criticism, it generates concrete scenarios describing how a real actor could obey the literal text while defeating its purpose.

3. Pass through the deterministic Grounding Gate

Before an attack can reach review, it must survive a non-AI verification layer.

Pure TypeScript code checks that every statutory quote cited by the attacker exists character-for-character in the source text.

If the model invents a word, misquotes a clause, or cites text that does not exist, the attack is rejected immediately.

No LLM decides whether evidence is real.

Code does.

4. Cross-examine it with a three-judge jury

Every grounded scenario is independently reviewed by three AI judges representing different legal perspectives:

The Textualist

Do these exact words prohibit this conduct?

The Purposivist

Would the legislative purpose support reading the rule this way?

The Enforcer

Could a regulator realistically bring an enforcement action today?

The jury runs at temperature 0.

More importantly, the jury never sees the attacker's persuasive argument.

The Attack Agent may argue aggressively for why its loophole works, but those fields are stripped before evaluation.

The judges see only:

  • the raw scenario facts
  • verified statutory text
  • the Purpose Contract

This prevents the attacker from persuading its own judges.

5. Give the human the gavel

If the jury agrees that the conduct survives the rule, it becomes a Confirmed Loophole.

If the existing text clearly stops it, it becomes Blocked.

If the judges disagree:

Jury split. Your call.

The human reviewer breaks the tie.

Loophole never removes human authority from the final legal decision.

6. Repair, then attack again

For a confirmed loophole, a Repair Agent drafts the smallest possible amendment intended to close it.

Once the human approves the change, Loophole automatically runs three regression tests.

Previously discovered loopholes must now fail:

$$ \text{Status}(\text{Loophole}_i)=\text{Blocked} $$

The patch must not introduce new loopholes:

$$ \text{Count}(\text{New Confirmed Loopholes})=0 $$

Legitimate conduct must remain legal:

$$ \text{Status}(\text{Legitimate Conduct}_k)=\text{Preserved} $$

The loop becomes:

Attack → Verify → Judge → Repair → Re-attack

Essentially, regression testing for rules.

How we built it

We built Loophole as an end-to-end web application with a strict separation between deterministic code, probabilistic AI agents, and human judgment.

Single-screen War Room

The frontend uses:

  • Next.js 16 App Router
  • Tailwind CSS v4
  • shadcn/ui
  • Motion

We initially had a multi-step workflow, but it hid the core experience behind forms.

We replaced it with a single War Room where the statutory text and live Attack Arena sit side by side.

Dynamic SVG connectors visually link each attack to the exact statutory clause it depends on.

A judge can understand what the product does almost immediately.

Deterministic Trust Core

The trust boundary lives inside src/core.

It is pure TypeScript with zero framework dependencies.

It handles:

  • quote normalization
  • exact substring verification
  • deterministic evidence validation
  • canonical SHA-256 audit chaining
  • jury decision rules

The important architectural choice was deciding which tasks should never be delegated to an LLM.

Evidence existence is one of them.

Multi-agent orchestration

We use the Vercel AI SDK with:

  • Groq as the primary provider
  • separate REASONING_MODEL and FAST_MODEL
  • OpenRouter as an automatic fallback

The system coordinates four major agent roles:

  • Red-Team Attacker
  • three-member Defense Jury
  • Surgical Repair Drafter
  • Purpose Ingestion Helper

Each agent receives only the information required for its role.

That isolation became one of the most important parts of the system.

Persistence and streaming

We use Neon Serverless Postgres with Drizzle ORM.

The database stores:

  • immutable source versions
  • SHA-256 hashes
  • finding evidence bundles
  • human rulings
  • audit history

Attack events and jury deliberations stream live to the browser using Server-Sent Events (SSE) with automatic reconnection.

The result feels less like submitting a prompt and waiting for a response, and more like watching an adversarial simulation unfold.

Challenges we ran into

1. A mid-hackathon architectural pivot

Our first architecture used the Z3 SMT solver.

The idea was to compile legal clauses into formal logic and search for states satisfying:

$$ L \land \neg P $$

where the literal law holds but its intended purpose fails.

It was mathematically elegant.

It was also the wrong abstraction.

Real laws depend on concepts such as:

  • "reasonable efforts"
  • "material adverse effect"
  • "good faith"

These concepts do not reduce cleanly to Boolean constraints.

Worse, the formalization process forced users to inspect around 15 mathematical expressions before they could even see an adversarial scenario.

We killed the architecture mid-hackathon.

The replacement was an Adversarial Multi-Agent Jury.

The hard part then became preserving rigor without turning Loophole into another ungrounded chatbot.

That constraint shaped nearly every architectural decision that followed.

2. Eliminating the echo chamber

When one LLM generates an argument and another evaluates it, the evaluator can become anchored by the first model's framing.

The attacker says:

"This loophole clearly works because..."

And the judge begins reasoning from that premise.

We solved this structurally.

The Attack Agent can generate persuasive reasoning for the UI, but the server strips it before jury evaluation.

The judges receive only the scenario, verified statutory text, and policy intent.

The attacker never gets to argue its own case to the jury.

3. Exact statutory quote alignment

Legal text is messy.

Real documents contain:

  • curly quotation marks
  • inconsistent whitespace
  • line breaks
  • indentation
  • section numbering
  • formatting artifacts

Our Grounding Gate still needed to determine, deterministically, whether a cited passage actually existed in the source.

We built a normalization-aware matching system that handles formatting differences while refusing to invent or fuzzy-match legal language.

That allowed us to keep the evidence gate deterministic without making it unusably brittle.

Accomplishments that we're proud of

Historical retrodiction

We tested Loophole against California's original 2018 CCPA language.

The system independently rediscovered the uncompensated data-sharing loophole and proposed the same fundamental repair California later introduced by expanding the law beyond "selling" data to also cover "sharing."

That was the moment the idea stopped feeling theoretical.

Loophole had identified, from the original text alone, a failure mode that policymakers later had to address in the real world.

Under 10 seconds to understand the product

Our first version required users to move through multiple pages before anything interesting happened.

We rebuilt the experience around a live War Room.

Now a judge can land on the product, watch an adversarial attack begin, see it connect to the exact statutory language, and understand the core idea in roughly 10 seconds.

Human-in-the-loop by architecture

We deliberately avoided building a black-box "AI judge."

Instead:

  • AI attacks the rule
  • deterministic code verifies its evidence
  • three independent judges evaluate the scenario
  • humans resolve disagreement
  • humans approve amendments

The system is designed around human authority rather than treating it as an afterthought.

What we learned

Law behaves more like an adversarial system than a static document

A policy may look complete until someone is financially motivated to find the one interpretation its authors missed.

That makes rule-making surprisingly similar to security engineering.

You do not discover vulnerabilities by reading your own code more carefully.

You attack it.

Loophole applies the same idea to laws and policies.

Transparency beats pretending to be perfect

An AI system claiming to be "100% legally correct" is not trustworthy.

A system becomes easier to trust when users can inspect:

  • the exact cited clause
  • the exact scenario
  • the judges' reasoning
  • where the judges disagree
  • the proposed amendment
  • the results of the re-attack

We learned that showing uncertainty is often more valuable than hiding it.

What's next for Loophole AI

Cross-statute interaction fuzzing

Today, Loophole primarily attacks a rule in isolation.

The next step is testing how proposed laws interact with:

  • existing federal and state statutes
  • regulatory definitions
  • overlapping jurisdictions
  • established case law

Many real loopholes exist not inside one law, but between multiple laws.

Enterprise policy and governance fuzzing

The same engine can stress-test rules outside government.

We want to apply it to:

  • platform Terms of Service
  • Acceptable Use Policies
  • compliance controls
  • internal governance rules
  • DAO governance proposals

Any system governed by written rules can potentially be adversarially tested.

Native drafting integration

Ultimately, Loophole should not live beside the drafting workflow.

It should live inside it.

We want to integrate adversarial testing directly into legislative and regulatory drafting tools so that, while a rule is being written, its authors can immediately see:

Here's how someone may exploit this.

Then fix it before deployment.

Built With

Share this project:

Updates

Submission history