SlopGuard

AI invents packages. Attackers register them. SlopGuard checks every dependency before you install, with plain-English verdicts, a one-click fix, a CLI and a GitHub Action.

Comment

Inspiration

AI has made it incredibly easy to generate code, but it has also changed the way developers discover dependencies. A developer can ask an AI assistant to build a feature, copy the generated code, and run the suggested pip install or npm install command without questioning whether every package name actually exists or is trustworthy.

This creates a supply-chain security problem known as slopsquatting: an AI-generated dependency name may be incorrect, unavailable, or potentially become an opportunity for an attacker to register a malicious package under that name.

The idea for SlopGuard came from a simple question:

What if I could check AI-generated dependencies before they ever reach the installation step?

Instead of expecting developers to manually verify every package, I wanted to build a lightweight security layer that sits between AI-generated code and package installation.

My goal was not to claim that every AI-generated dependency is malicious. The goal was to make uncertainty visible and give developers a practical way to verify dependencies before installing them.

That led me to build SlopGuard — a pre-install dependency security scanner designed specifically for AI-generated code and dependency lists.

What it does

SlopGuard analyzes dependency references before installation and provides a security decision without installing or executing the submitted packages or code.

The scanner accepts several common forms of input:

  • requirements.txt
  • package.json
  • Python imports
  • JavaScript imports
  • pip install / npm install commands
  • General dependency text

The default Auto Detect mode automatically identifies the input format. Developers can also manually select a format when they already know what they are submitting.

For Python imports, SlopGuard maps import names to their actual package names. For example:

  • cv2 → opencv-python
  • yaml → pyyaml
  • PIL → pillow
  • sklearn → scikit-learn
  • bs4 → beautifulsoup4

It also recognizes standard-library modules such as os and json and does not treat them as external dependencies.

For each external dependency, SlopGuard performs read-only checks against public package-registry metadata and evaluates security-related signals.

The scanner can produce five security outcomes:

  • OK
  • REVIEW
  • RISKY
  • BLOCK
  • UNKNOWN

Some of the signals considered include:

  • Unregistered or non-existent packages
  • Malicious security advisories
  • Typosquat/lookalike indicators
  • Very new packages
  • Low download activity
  • Missing repository information
  • Single-release packages
  • Install-time scripts
  • Registry lookup failures

A key design decision was how to handle uncertainty.

If a registry lookup fails because of a network problem or rate limit, SlopGuard does not pretend the package is safe and does not automatically call it malicious. Instead, it returns UNKNOWN and tells the user that verification was inconclusive.

I also built a "Fix it for me" feature. When a suspicious dependency is detected, SlopGuard can create a safer version of the submitted dependency file while clearly showing what changed.

For example, in my test case:

slopguard-demo-phantom-pkg-93817

was identified as a phantom package because it did not exist in the checked public registry. SlopGuard classified it as BLOCK with a 100/100 risk score and removed it from the generated fixed version while preserving the legitimate dependency and its version pin.

The user can then copy or download the fixed file and independently verify any suggested changes before installation.

SlopGuard also provides downloadable scan reports, including JSON output, and is designed to fit into developer workflows through its CLI and GitHub Action.

I also integrated SlopGuard into a real GitHub pull-request workflow. In my demo PR, adding the phantom dependency caused the SlopGuard GitHub check to fail, allowing the issue to be identified during code review before merging.

How I built it

I built SlopGuard as a full-stack security tool with a focus on transparency, deterministic decisions, and safe pre-install analysis.

The overall pipeline is:

AI-generated code → Dependency extraction → Read-only registry metadata → Security signals → Deterministic scoring → Verdict + action

The frontend provides the interactive scanner and educational interface. It includes the Scanner, Learn, Benchmark, and About sections so that users can understand both the product and the problem it addresses.

The backend exposes the scanning API and performs the dependency extraction, package metadata checks, security-signal analysis, scoring, and report generation.

For dependency extraction, I added parsing for different input styles instead of assuming that every user provides a traditional requirements file. This allows the same scanner to work with raw AI-generated code, import statements, package manifests, and installation commands.

For registry verification, SlopGuard uses public PyPI and npm metadata through read-only lookups. It does not install packages and does not execute submitted code.

The risk engine uses deterministic scoring rather than allowing an LLM to directly decide whether a package is safe. This makes the security decision explainable and reproducible.

The result is converted into transparent risk points and mapped to the final verdict.

I also added explicit handling for UNKNOWN results. This was important because real-world registry APIs can fail, rate-limit requests, or return incomplete information. Rather than hiding that uncertainty, SlopGuard surfaces it to the developer.

For the AI-related evaluation, I created a benchmark that requests dependency-only responses from configured models, extracts the package names, checks them against public registries, and evaluates them using the SlopGuard scorer.

I also created a deterministic scorer evaluation against an author-labelled corpus of 70 cases.

Finally, I added an automated test suite covering the application. The final test run passed 125 tests.

The application was deployed as:

  • Frontend: Vercel
  • Backend/API: Vercel
  • Source: GitHub
  • CI/CD integration: GitHub Actions

The deployed architecture allows the live website to communicate with the deployed scanning API rather than relying on a local development server.

Challenges I ran into

One of the biggest challenges was making the scanner reliable enough to distinguish between a genuinely suspicious dependency and a dependency that simply could not be verified.

During testing, legitimate packages such as Flask and Requests encountered registry lookup failures in one run. Instead of incorrectly marking them as safe or unsafe, SlopGuard returned UNKNOWN.

This forced me to think carefully about the difference between:

"the package is safe"

and

"we could not verify the package."

That distinction became an important part of the product.

Another challenge was testing phantom dependencies in a live serverless environment. Registry access can behave differently locally and on deployed infrastructure because of network conditions, rate limits, and execution constraints.

This meant I could not simply assume that a package that produced BLOCK locally would always produce the same result from a deployed API call. Rather than hiding this limitation, I designed the product to expose UNKNOWN when verification is inconclusive.

I also encountered a deployment issue where the frontend generated API URLs containing a double slash:

https://slopguard-api.vercel.app//api/...

This caused redirect and CORS-related problems in the browser.

I fixed this by normalizing the API base URL in the frontend so that trailing slashes could not create malformed endpoint URLs.

Another challenge was making dependency extraction work across different formats. An import such as cv2 does not correspond directly to the PyPI package name opencv-python, and sklearn maps to scikit-learn.

I therefore had to distinguish between import names and actual package names rather than treating the text literally.

The GitHub workflow also presented a practical challenge. I wanted to demonstrate that SlopGuard was not only a standalone website but could actually participate in a development workflow.

I created a test pull request containing a deliberately non-existent dependency and verified that the SlopGuard GitHub Action failed the check. This gave me a concrete demonstration of pre-merge protection.

Finally, I had to be careful with my benchmark claims. A 100% score on a small author-labelled evaluation corpus does not mean SlopGuard has 100% accuracy in the real world. I therefore separated my deterministic scorer evaluation from the model-output benchmark and documented the limitations directly in the Benchmark tab.

Accomplishments that I'm proud of

One of my biggest accomplishments is that SlopGuard became a working end-to-end product rather than just a concept.

I built the scanner, backend, risk engine, reports, educational interface, benchmark interface, automated tests, deployment, and GitHub workflow integration.

The live application can take real dependency input and return a security decision before installation.

I am particularly proud of the phantom-package test because it demonstrates the core idea clearly.

Given:

flask==3.0.3

requests

slopguard-demo-phantom-pkg-93817

SlopGuard identified the phantom dependency as BLOCK with a 100/100 risk score because the declared package name did not exist in the public registry.

It then used the Fix it for me workflow to remove the suspicious dependency while keeping the legitimate Flask version pin unchanged.

I am also proud of the Python import handling.

When I tested:

cv2

yaml

PIL

sklearn

bs4

requests

along with standard-library imports such as os and json, SlopGuard correctly mapped the external imports to their package names and produced an OK result for all six external dependencies in that run.

Another accomplishment is that I did not build the product around a black-box "AI says safe/unsafe" decision.

The scanner uses explicit security signals, deterministic scoring, and transparent explanations. When it cannot verify something, it says UNKNOWN.

My deterministic scorer evaluation contained 70 labelled cases and matched every label in that evaluation, producing 100% precision, recall, F1, and accuracy for that corpus. I clearly present this as an evaluation of my labelled test corpus rather than a claim about real-world accuracy.

I also added 125 automated tests and successfully passed the complete test suite.

Beyond the website, I integrated SlopGuard into GitHub Actions and demonstrated it on a real pull request. A PR containing the phantom dependency triggered the SlopGuard check and failed the workflow before the dependency could be merged through the normal workflow.

I am also proud of the transparency built into the product. The Benchmark and About sections explain how the system works, what was tested, what the limitations are, and why UNKNOWN is treated differently from a security verdict.

What I learned

I learned that supply-chain security is not only about detecting malware. It is also about handling uncertainty correctly.

A package that cannot be verified should not automatically be treated as safe simply because there is no evidence of malicious behavior. At the same time, lack of verification should not automatically be presented as proof that a package is malicious.

That led me to make UNKNOWN a first-class result.

I also learned that AI-generated code needs to be treated differently from traditional developer-written dependency files. AI assistants can produce syntactically convincing installation commands while still producing package references that need verification.

Another important lesson was that package names are not always the same as import names. Building a useful scanner therefore requires understanding the programming language and package ecosystem rather than simply searching for strings.

I learned the importance of deterministic security decisions as well. Using an LLM to generate an explanation can be useful, but the core security verdict should be based on observable signals and reproducible logic.

Testing also taught me that a successful local result does not automatically guarantee identical behavior in production. Network access, registry rate limits, serverless environments, and external APIs all introduce real-world uncertainty.

My benchmark work taught me another important lesson: evaluation numbers need context. A perfect score on a controlled labelled corpus can demonstrate that an implementation behaves correctly on that corpus, but it cannot by itself establish real-world security accuracy.

Most importantly, I learned that a security tool becomes much more useful when it fits into the developer's existing workflow. That is why I extended SlopGuard from a website scanner to CLI and GitHub workflow concepts rather than keeping it as an isolated demo.

What's next for SlopGuard

The next step for SlopGuard is to move from a hackathon prototype toward a more comprehensive developer security tool.

First, I want to strengthen registry verification and make the system more resilient to temporary lookup failures and rate limits. Better caching and more robust retry strategies would reduce unnecessary UNKNOWN results while still preserving honest uncertainty.

I also want to expand the security-signal layer. Future versions could incorporate additional package-health, maintainer, repository, release-history, dependency, and security-advisory signals while keeping the final scoring transparent.

Another direction is stronger typosquat and lookalike detection. Instead of only checking whether a package exists, SlopGuard could compare suspicious names against known popular packages and explain why a name looks unusual.

I also want to expand ecosystem support beyond the current PyPI and npm focus, making the same pre-install verification concept applicable to additional package managers and programming languages.

On the developer-experience side, I want to make the CLI and GitHub Action production-ready so teams can automatically scan dependency changes in pull requests and optionally configure policies such as failing a workflow when a dependency reaches a selected risk threshold.

I also see an opportunity to integrate SlopGuard directly into AI-assisted development environments. The long-term vision is for dependency verification to happen automatically at the moment an AI assistant suggests a package, rather than only after a developer copies the generated code into a terminal.

Finally, I want to continue expanding the benchmark with larger, independently labelled datasets and more model providers. The goal is to measure the problem and the effectiveness of SlopGuard more rigorously over time, while continuing to clearly separate experimental results from real-world security claims.

The core idea will remain the same:

AI can generate code quickly. SlopGuard adds a verification step before that code turns into an installed dependency.

Built With

Share this project:

Updates

Submission history