Faultline

Inspiration

AI agents are rapidly entering production environments, but most companies still rely on manual prompt reviews and basic testing to check compliance. We built Faultline to solve that problem by making policy enforcement automated, measurable, and repeatable.

What it does

Faultline is a policy enforcement layer for AI agents.

Users upload an agent file and a company policy document. Faultline:

generates adversarial attacks
tests the agent against company policies
detects compliance violations
automatically patches the system prompt
re-runs the tests to verify improvements
exports a redeployable patched agent file

Think of it as unit testing for agent behaviour instead of application code.

How we built it

We built Faultline using React, Vite, and Vercel with LLMs accessed through the Clod API.

The workflow:

Analyse uploaded files and generate 7 domain-specific attack cases
Simulate adversarial interactions against the agent
Detect and classify violations
Patch the system prompt automatically
Re-run tests and generate a downloadable fixed .py file

Challenges we ran into

One challenge was ensuring the generated attacks were both realistic and reliable for demos.

Another was preventing the patched prompt from becoming overly restrictive after fixing violations. We solved this by automatically re-running the same test suite after patching.

We also had to handle secure API proxying and real-time workflow state management.

Accomplishments that we're proud of

We're proud that Faultline completes the full remediation loop:

detect violations
patch prompts
verify fixes
redeploy updated agents

We also built:

structured policy diffs
reproducible adversarial tests
real-time compliance visualisation
downloadable patched agent files

What we learned

We learned that testing AI agent behaviour is very different from testing traditional software. Structured outputs, evaluation loops, and transparent reasoning are critical for building trustworthy AI tooling.