PatchSignal

Why I built it

Developer incident searches often mix a confident AI answer with stale or unofficial evidence. I wanted a small tool that makes the answer useful without asking developers to trust it blindly.

What it does

PatchSignal accepts a bounded incident description and rejects text that looks like identity data, local paths, or credentials. Each analysis makes two server-side SerpApi requests.

The first, on the Google AI Mode engine, answers the incident. The app shows a plain-text synopsis, then ranks only the HTTPS sources cited in that response, favoring official documentation, exact version matches, incident-term overlap, dates, and source diversity.

The second request is the point of the tool. It runs the same sanitized incident terms as a plain Google search, so the citations get checked against evidence AI Mode had no hand in producing. Each cited source is labelled by whether that independent search returns the exact page, the domain, or neither. Authoritative pages the independent search found and AI Mode never cited are listed separately, because a confident answer that skips the official docs is worth knowing about.

PatchSignal also displays a query hash and a safety note. It never scrapes cited pages, runs commands, or applies a fix.

How I built it

The app uses Python's standard-library HTTP server with vanilla HTML, CSS, and JavaScript. The SerpApi key stays in a server environment variable. The parser reads reconstructed_markdown when it is present and falls back to the documented text_blocks structure. It removes code blocks, links, raw URLs, and unsafe text before rendering the synopsis as plain text. The ranker and the corroboration pass are separate from the synopsis, and the corroboration pass reads only the organic results of the second search.

Challenges I ran into

My first live AI Mode response did not contain a usable reconstructed_markdown field, so the app failed closed. I added a bounded text_blocks fallback and tested it with synthetic fixtures before trying a different untouched live incident.

The harder problem showed up the first time I ran the corroboration pass against a real incident. I had written it to subtract points from any citation the independent search did not return, which sounds reasonable until you watch it demote docs.python.org to seventh place, below four GitHub issue threads, on a Python 3.12 question. The official documentation was the right answer. The organic query was simply noisy.

Authority and popularity are not the same axis, and I had collapsed them into one. An uncorroborated verdict now lowers the score only for community sources. Official documentation and source repositories keep their standing and carry the verdict as information instead of a penalty. On a Node 20 incident the effect is easy to see: nodejs.org is labelled uncorroborated and still ranks first, while an uncorroborated community post sits last.

Accomplishments

The app passes 33 network-free unit tests, up from 24, with the new ones covering corroborated, domain-corroborated, and uncorroborated citations, the fail-closed path when no independent results come back, and the rule that keeps an uncited forum thread out of the missed-sources list.

On a live Python 3.12 importlib_metadata incident, two free-tier requests produced eight HTTPS citations, six of them corroborated against nine independent results. Ruff, Gitleaks, and Semgrep complete with zero findings. The whole project runs inside the SerpApi free tier.

What I learned

Google AI Mode responses can vary even when they contain useful evidence, and structured text blocks make a safer fallback than assuming one generated field will always exist.

The bigger lesson was that an audit built entirely from the thing it audits is not an audit. Every signal in the original ranking came out of the AI Mode response, so a citation that had nothing to do with the incident scored exactly as well as one that did. One independent request changed that, and it only cost a second search.

What's next

I would add a third read for release notes and changelogs, so a version-specific incident can be checked against the actual change that caused it. PatchSignal would still stop at evidence review rather than changing a developer's system.

Built With

• ai
• css
• google
• html
• javascript
• python
• serpapi