DevSheriff

WE (yes WE) hate PRs

Comment

Project Inspiration

Code review is the last line of defense before bugs and vulnerabilities ship to production. But it's also one of the most inconsistent practices in software development. Reviewers are busy, context-switching between features, and security isn't always top of mind when evaluating a 400-line PR at the end of a sprint.

The problem isn't that developers don't care, it's that catching security issues requires a specific mindset and deep knowledge of vulnerability patterns that most developers don't apply consistently under time pressure. Things like SQL injection via f-string interpolation, timing attacks on signature comparison, XXE via unsafe XML parsing, or SSRF via user-controlled URLs, these are well-documented OWASP Top 10 categories, yet they appear in production codebases regularly.

I built DevSheriff because I wanted a reviewer that never gets tired, never skips the security pass, and always knows the OWASP categories. One that works where developers already are, inside GitHub pull requests, with zero workflow change required.

Technology Stack

Languages

  • Python 3.11 (backend)
  • TypeScript (dashboard frontend)

Frameworks and Libraries

  • FastAPI: async webhook server with lifespan management
  • Pydantic / pydantic-settings: configuration and validation
  • PyGithub: GitHub REST API client (PR fetching, review posting, commit status)
  • PyJWT + cryptography: RS256 JWT generation for GitHub App authentication
  • httpx: async HTTP client for GitHub API calls
  • Anthropic Python SDK (anthropic): Claude API integration
  • React 18 + Vite: dashboard frontend
  • TailwindCSS + shadcn/ui: dashboard UI components

Platforms

  • GitHub Apps: webhook delivery, PR review API, commit status API
  • Anthropic API (Claude Sonnet 4.6): AI-powered semantic and security code analysis
  • Google Cloud Platform:
    • Cloud Run: containerized backend hosting
    • Firestore: review audit log persistence
    • Secret Manager: secure credentials storage
    • Cloud Build: CI/CD pipeline

Tools

  • smee.io: webhook proxy tunnel for local development
  • Terraform: GCP infrastructure as code
  • Docker: containerization
  • pip-audit + OSV API: dependency vulnerability scanning

Product Summary

DevSheriff is a GitHub App that performs AI-powered code review on every pull request, automatically.

How it works:

When a developer opens or pushes to a pull request, GitHub sends a signed webhook to DevSheriff's FastAPI backend. The backend verifies the HMAC-SHA256 signature, then asynchronously:

  1. Authenticates with GitHub using RS256 JWT → installation access token exchange
  2. Fetches the PR's changed files and unified diffs
  3. Runs two parallel Claude AI review passes per file:
    • Semantic pass: logic errors, bugs, performance anti-patterns (N+1 queries, blocking I/O), maintainability issues
    • Security pass: OWASP Top 10 vulnerabilities: injection, hardcoded secrets, insecure cryptography, path traversal, XXE, SSRF, ReDoS, insecure deserialization
  4. Maps AI findings back to exact diff line positions
  5. Posts inline review comments directly on the PR via GitHub's review API
  6. Sets a commit status check (pass/fail) that can block merging on critical or high severity findings
  7. Saves an audit log to Firestore for the dashboard

What makes it different:

  • Zero workflow change: developers work in GitHub as normal; DevSheriff's comments appear inline on the diff, indistinguishable from a human reviewer's comments
  • Two-pass architecture: separating semantic and security analysis into parallel Claude calls allows each system prompt to be deeply specialized without compromise
  • OWASP-anchored security findings: every security finding references the specific OWASP Top 10 category, making it immediately actionable for compliance-aware teams
  • Concrete suggestions: Claude is prompted to always include a specific suggested fix in a code block, not just "this is vulnerable" but exactly how to fix it
  • Merge blocking: the commit status integration means teams can enforce that critical findings must be addressed before code ships

Dashboard:

A React + TypeScript dashboard shows recent PR reviews across all connected repos, finding severity breakdowns, and per-review details, giving engineering leads visibility into security trends across their codebase over time.

AI Use

Yes, more than 70% of the code in this project was generated by AI (Claude Code).

The core product itself, the code review engine, is powered by Claude Sonnet 4.6, using carefully engineered system prompts that instruct the model to act as both a senior code reviewer and a security auditor, returning structured JSON findings that map directly to GitHub's review comment API format.

Built With

Share this project:

Updates