gateAudit

Inspiration

People now ask AI instead of Google — and AI reads websites to answer. But many sites quietly block the AI crawlers that would surface them, while robots.txt hides what they actually do at the network layer. We wanted an agent that exposes how a site really treats AI.

What it does

gateAudit is an autonomous agent that audits how websites treat AI agents. It probes each site as a browser, a generic crawler, and each AI bot to catch "allowed in robots, 403 at the edge" blocking, scores agent-readiness, publishes a cited report, gates it behind an x402 paywall, and keeps monitoring for changes.

How we built it

A behavioral probe layer + ClickHouse for evidence, an LLM (via TrueFoundry) to write the assessment, Guild.ai to govern the agent, Composio to publish, Langfuse to trace, x402/Coinbase CDP for the agent-payable report, and Render for the scheduled monitor. Runs end-to-end locally with zero keys.

Challenges we ran into

Telling AI-targeted blocking apart from generic anti-bot (solved with a three-way comparison), the fact that robots.txt is ignored so we had to probe real behavior, and wiring fast-moving payment SDKs without breaking the demo — every integration has a mock fallback.

Accomplishments that we're proud of

A real end-to-end autonomous agent — it investigates, reports with cited evidence, monitors, and gets paid agent-to-agent — surfacing genuine AI-targeted blocking on well-known sites that robots.txt hides.

What we learned

The web is fragmenting by AI identity, declared policy (robots.txt) and enforced policy (WAF) routinely disagree, and x402 lets a service charge and get paid with no human at the moment of sale.

What's next for gateAudit

Deeper probing and continuous monitoring at scale — and turning gateAudit into the diagnostic wedge for gateX, the protection and policy layer that lets sites act on it: admit the right agents, block the wrong ones, or charge them.

Built With

• clickhouse
• javascript
• openui
• render
• truefoundery