SEO Scrubbox

See what search engines and AI crawlers actually see. Audit rendering, indexability, freshness, hreflang and internal links, then explore websites through a live visual crawl.

Comment

Inspiration

Technical SEO tools often show either the raw HTML returned by a server or the final page displayed in a browser. Modern search engines and AI crawlers may encounter something between those two states, depending on rendering, robots directives, HTTP headers, JavaScript and crawler identity.

I built SEO Scrubbox to make those differences easier to investigate without switching between numerous tools. It is a Chrome extension for technical SEOs, developers and web teams who need to understand what crawlers can access, extract and interpret.

What it does

SEO Scrubbox combines technical page auditing with live visual crawling.

It can:

  • Compare source HTML with the rendered DOM
  • Diagnose redirects, canonicals, robots directives and indexability
  • Check access rules for search and AI crawlers
  • Validate hreflang implementation and reciprocal links
  • Inspect sitemaps, structured data and internal links
  • Scope internal-link audits using XPath
  • Identify bot-visible publication and modification signals
  • Highlight nofollow and data-nosnippet elements
  • Crawl up to 500 URLs and visualise architecture by directory or crawl depth
  • Test pages using different user agents
  • Process audit data locally within the browser

What I extended during Build Week

SEO Scrubbox existed before the Build Week submission period. During Build Week, I meaningfully extended it using GPT-5.6 and Codex in four focused areas.

Bot-visible Freshness Signals

I added a freshness audit that brings together dates exposed through JSON-LD, meta tags, microdata, HTTP Last-Modified headers, sitemaps and elements.

The extension identifies published and updated dates, shows where each signal came from and flags potentially conflicting values. Generic elements remain visible but are excluded from conflict scoring because they do not always represent publication or modification dates.

I also deliberately removed document.lastModified after determining that it is browser/load metadata rather than a reliable crawler-visible freshness signal.

JavaScript-aware hreflang validation

Reciprocal hreflang checks previously depended on source HTML and HTTP headers. They now use those sources as the fast path, but when no return link is found, SEO Scrubbox renders the target page in a temporary background tab and inspects its rendered hreflang tags.

Source and rendered results are merged and cached, while temporary renderer tabs are cleaned up after completion or cancellation.

XPath-scoped internal-link auditing

Users can provide an optional XPath expression to audit links from a specific area, such as the main navigation.

The XPath is applied independently to source HTML and the rendered DOM. SEO Scrubbox combines the resulting URLs, reports how many elements and links matched, and validates invalid, unmatched or silently ignored expressions.

Reliability and security

I hardened cancellation handling, URL normalisation, redirect processing, robots and X-Robots-Tag logic, crawler persistence, cache isolation, incognito storage and local-network request protection.

Cancelled audits now distinguish between requests that received no response and URLs that were never checked. Partially completed audits are labelled as partial data instead of incorrectly reporting that all links are healthy.

How I used Codex and GPT-5.6

I worked with Codex across four focused development sessions covering freshness analysis, rendered hreflang validation, XPath auditing, and broader reliability and security hardening.

Codex helped investigate the existing codebase, trace asynchronous browser behaviour, implement changes across content scripts, the popup and service worker, review security boundaries, and perform syntax and regression verification.

I retained responsibility for the product and engineering decisions, including keeping source HTML as the fast path, rendering only when necessary, restricting freshness conflict scoring to meaningful crawler-visible signals, and preserving a coherent interface rather than exposing unnecessary implementation detail.

Challenges

The most difficult problems involved asynchronous Chrome extension behaviour: temporary renderer tabs, stale content scripts, cached results from earlier audit scopes, cancellation races and separating genuinely failed requests from work that never started.

Another challenge was avoiding false confidence. A date displayed by the browser is not necessarily a date available to a crawler, and a hreflang tag visible after JavaScript execution may be absent from the original response. The extension needed to expose those distinctions accurately without overwhelming the user.

What I learned

The project reinforced that crawler analysis cannot rely on a single representation of a page. Source HTML, HTTP headers, rendered content, structured data, sitemaps and robots rules all contribute different evidence.

It also demonstrated how Codex can help work across a large browser-extension codebase while still leaving important product and technical judgement with the developer.

What's next

I plan to expand crawler comparison capabilities, improve exportable reports and use SEO Scrubbox alongside Scrubnet's independent crawler research to study how search and AI systems access changing web content.

Built With

Share this project:

Updates