AI-slop-check

• 

Inspiration

I vibe-code a lot. And a lot of what comes back is slop.

Not broken code. Code that looks finished. A catch block that swallows the error so nothing ever turns red. A comment that says // In a real implementation, validate this sitting exactly where the validation should be. A default that no caller has ever overridden, because the model invented a knob nobody asked for. It runs. It passes. It is not done.

My usual fix for this was to paste the file into a better model and ask it to look. Which works, and costs tokens, and takes a minute, and I have done it for mistakes that a grep would have caught. That felt stupid enough to build around.

So slop-check has no AI in it. Twelve rules, plain text matching, one pass over the source. It scans a repo, a diff, or a pull request and hands back a 0-100 score, a grade, and every hit quoted as file:line. Instant, offline, deterministic, and the same input gives the same output forever.

The scoring is deliberately boring so you can argue with it:

$$\text{score} = \min\left(100,\ \frac{100 \times \sum_{r} w_r \cdot d(h_r)}{1000 \times \max(L_{\text{eff}},\ 420)}\right)$$

where $w_r$ is a rule's weight, $h_r$ its hit count, $d$ a damping function so one repeated house idiom cannot fail a whole codebase, and $L_{\text{eff}}$ the effective scanned lines. --explain prints that arithmetic per run. No black box to trust.

why it is a card game

A wall of 200 findings is a linter, and linters get disabled. So findings come back as a deck you swipe. Left says the finding is fair. Right says it is a false positive, and disputed cards come back out as slop-disable directives you paste into your code, which the CLI and the GitHub Action then honour on every later run. Your disagreement becomes configuration instead of a shrug.

There is also a confession mode on the share card. It stamps YES, I WROTE THIS WITH AI on the image. Everyone is doing it. The pretending is the annoying part.

It has already earned its keep on repos much larger than mine. Pointed at openai/openai-python (1,542 files) it surfaced real findings, and the same run surfaced six defects in my own rules.

what I actually learned

A scanner's most important property is staying quiet. I spent most of my time not adding rules. I read every finding on a large, well-written repo by hand and asked whether a maintainer would act on it. 61 findings became 12. None of the six bugs I found this way would have surfaced from unit tests, because every rule passed its own fixtures.

Some of them were embarrassing in a useful way. dead-defaults split parameter lists on bare commas, so a comma inside dict[str, Any] invented parameters that did not exist. happy-path-only matched \bopen(, which caught tarfile.open() and wave.open() and read every with open(...) as unhandled I/O, when a context manager is the failure path. duplicate-helper flagged every f / async_f pair in an async library at 95% similarity, which is a Python constraint and not a mistake. swallowed-error flagged __del__, where suppressing exceptions is correct and the tool was asking for the code to be made worse.

And the one I liked most: every rule read prose as code. A rule that matches source text will happily match a sentence describing the pattern it looks for. This repo scored itself a grade worse because two of its own explanatory comments contain .catch(() => {}). Comments are masked before matching now.

Each of those is pinned in test/false-positives.test.ts, which exists for one job: keep fixed rules fixed.

Then WebMCP. I liked it immediately, and for a reason that is more about how I work than about the spec. I am usually looking at a page and I want my agent looking at the same page. Not a copy of it, not a scrape of it, the actual thing in front of me. And half the time I am lazy and would rather ask than click, but I still want to watch it happen. Every other integration shape makes you pick one: either the agent works somewhere you cannot see, or you do it yourself. WebMCP is the first one where the agent presses the buttons you can see it pressing.

how I built it

One TypeScript source ships three things: the slop-check CLI, a GitHub Action, and the web app. The rules run in a Web Worker in the tab, so a scan never uploads your code anywhere.

The WebMCP layer is web/webmcp.ts. Seven tools registered with document.modelContext.registerTool(): list_rules, explain_rule, scan_repository, scan_diff, list_findings, judge_finding, export_suppressions.

The rule I gave myself was that the tool layer owns no state at all. It gets a small handle assembled in main.ts out of the exact functions the click handlers call. There is no second code path, so there is no way for an agent to reach a result the page is not showing, or to record a verdict the deck does not reflect. A tool-driven scan fills the visible input, flips the visible tab, and runs the same progress panel a click does.

Three tools are meaningless before a scan exists, so they are not registered until one lands and are unregistered when results clear, using the AbortSignal that registerTool takes as its second argument. The page emits its own state event, the WebMCP layer syncs registration to it, the browser fires toolchange. An agent reading the tool list is reading the page's real state rather than a manifest.

Cancellation is threaded all the way down. execute receives a signal, it reaches githubFiles and every fetch under it, composed with the existing per-request timeout through AbortSignal.any.

challenges

False positives, by a distance. Everything above about the openai-python pass was the hard part of the project, not a footnote to it. It is still not fully solved. hardcoded-config is my noisiest rule and I will tell you exactly how: it flags process.env.SITE_URL || 'https://example.com', which is a literal inside an environment-variable fallback, which is the configuration it is demanding. It flags a project's own canonical URL, which belongs in a sitemap. swallowed-error still cannot tell recovery from silence, so .catch(() => null) whose value gets branched on reads the same to it as a bare .catch(() => {}). Both get reported. One is a fallback and one is a shrug.

I also know the rule set came from my intuition rather than data. There is empirical work on LLM code smells finding magic numbers and missing documentation most over-represented, which barely overlaps with my twelve rules, and their method is better than mine: compare a codebase's smell profile against a human-written baseline instead of scoring it alone.

Deciding how far the agent gets to reach. My first version let judge_finding walk the whole deck and decide everything. Technically better tool use. Completely wrong product. An agent that judges every card has replaced the user, and the verdict was the only part I wanted a human doing. The shipped version can move one card the user can see, one verdict at a time, and the tool's own description tells the agent to ask first.

Judging out of order. The deck is a stack, but an agent asked about one specific rule wants that card now, not in fourteen swipes. Re-rendering the stack broke the animation on cards already leaving, so swapToFront swaps the rendered contents of two positions instead. The deck's order was a sampling artefact, never meaning, so nothing is lost.

what it does not claim

slop-check cannot tell you code was written by an AI and does not try. There is no reliable authorship signal in source text and anything claiming otherwise is guessing. It measures whether code carries the patterns that AI-assisted code carries more often. A human can write all of them, and I regularly do.

Read the score as a description of the code, never as an accusation about who wrote it.

What it does

How we built it

Challenges we ran into

Accomplishments that we're proud of

What we learned

What's next for AI-slop-check

Built With

• claude
• codex
• github
• typescript