Prompt Ranger

In a world full of AI, only one hero stands against it - Prompt Ranger!

Comment

Inspiration

Open source software has been one of the biggest drivers in expanding the fronts of technology for years now. With the advent of LLMs and AI-generated code, open source repositories on websites such as GitHub are now prone to "slop" pull requests - PRs that contain way too much information for maintainers to review while also containing critical bugs.

See this document for sources on the AI-driven impact on open source projects.

What it does

Prompt Ranger is a combination of software solutions designed to support repository maintainers in filtering out PRs. Whenever a new PR is opened, Prompt Ranger will detect the PR for possible AI-generated indicators, such as:

  • High frequency of commits
  • Large number of changes within one PR
  • AI-chat history tagged by the VSCode extension

Repository maintainers are free to configure what they want filtered from the repository. If Prompt Ranger detects too many commits in a PR, that PR may be automatically closed. If Prompt Ranger is told to exclude all AI-chat history, PRs tagged with AI will be automatically closed.

How we built it

Prompt Ranger operates in two parts: the Analysis Server and the Extension.

Prompt Ranger Analysis Server

The Prompt Ranger Analysis Server connects to the GitHub API to control the Prompt Ranger Bot. This server receives events from GitHub about new PRs; it's responsible for analyzing them and reporting back to maintainers (via the PR itself) about the status of a PR.

Prompt Ranger Chat Extension

The Prompt Ranger Chat Extension is a contributor-oriented tool critical for tracking AI usage. Contributors may be required by repository owners to use the tool; this way, Prompt Ranger can keep track of AI-chat history and flag possible events when needed.

Challenges we ran into

To protect Prompt Ranger from unauthorized requests, we verified signatures using HMAC with SHA-256 and signed short-lived JWTs. This added complexity when testing across different development environments.

Beyond security, we faced latency during our early testing phases due to building and running containers and troubleshooting new libraries with limited documentation. Logging was critical to confirm our business logic worked as expected.

Accomplishments that we're proud of

Prompt Ranger directly interacts with PRs by labeling them and providing direct feedback when potential issues are detected after the PR is created in real time.

What's next for Prompt Ranger

In the future, we see Prompt Ranger enabling GitHub repo maintainers more control over configuring the detection settings for the potential usage of AI. We would also like to expand the extension to other IDEs.

Built With

Share this project:

Updates