Qrawl

• 

  Make any website AI agent ready. Qrawl audits and generates agents.json automatically.

• 

  Live crawl in progress — Playwright visits each page and streams screenshots in real-time via SSE.

• 

  Score breakdown across 10 categories with circular grade indicator and detailed bar chart.

• 

  Every score comes with Claude's exact reasoning — click to expand per category.

• 

  Syntax-highlighted agents.json output — download directly or copy the embed script.

Inspiration

The web was built for humans. Every page, every form, every nav menu — designed for eyes and hands, not agents. When I started building AI agents that needed to browse real websites, they kept failing — not because the AI wasn't smart enough, but because there was no standard way for a website to say "here's how to navigate me."

robots.txt told crawlers what to avoid. sitemap.xml told search engines what exists. Nothing told AI agents how to act. So I built Qrawl to fix that.

What it does

Qrawl audits any website and auto-generates agents.json — a structured spec that tells AI agents exactly how to navigate it: where to enter, what actions exist, how auth works, where captchas appear, and what to avoid.

It scores sites across 10 categories (semantic HTML, navigation clarity, auth friction, captcha presence, API parity, etc.) and generates a downloadable agents.json file agents can consume directly.

Live crawl progress streams in real-time via SSE — you watch Playwright visit each page and screenshot it as it scores.

How I built it

• Playwright (stealth mode, headless Chromium) crawls up to 5 pages per site, capturing screenshots streamed live via Server-Sent Events
• Claude API runs two passes: Pass 1 scores 10 categories with detailed reasoning, Pass 2 generates the full agents.json spec
• Next.js 14 + TypeScript for the full-stack app
• Supabase for storing audit results
• Deployed on Render

Challenges

Bot detection was the hardest part. Sites like IRCTC block everything — they won't even let an audit run. Amazon scores ~49/100 despite being one of the most visited sites on earth. Making the crawler resilient while staying ethical took most of the time.

Prompt determinism was another challenge — getting Claude to produce consistent, valid agents.json output across wildly different site structures required careful prompt engineering across both passes.

What I learned

Most of the web is genuinely hostile to agents — not intentionally, just by default. The gap between a human-friendly site and an agent-friendly site is enormous, and almost no one is thinking about it yet. agents.json could be to the agentic web what robots.txt was to search — a simple open standard that becomes universal infrastructure.

Built With

• api
• claude
• nextjs
• playwright
• render
• server-sent-events
• shadcn/ui
• supabase
• tailwindcss
• typescript