Outrider

• 

• 

• 

Inspiration

I started Outrider after seeing AI reviewers like Greptile and wondering what my own take on the idea would look like.

The question I kept coming back to was trust. AI can be useful in code review, but it can also miss context or sound very sure when it’s wrong. I wanted developers to see how each finding was reached and to have the final say when a review found something serious.

That idea shaped the whole project. The model reviews the code and suggests findings. Regular code handles severity, evidence checks, comment placement, and approval rules.

What it does

Outrider is a self-hosted AI code reviewer for GitHub pull requests. It reads the changes, looks for bugs and other issues, and leaves inline comments with suggested fixes when it can.

Before asking the model to review a file, Outrider uses tree-sitter to understand its structure. It finds the function or class around the changed lines and sends that complete section for review. This gives the model enough surrounding code to understand what’s happening without sending unrelated parts of the file.

I also added a few guardrails around what the model can do.

Severity comes from a fixed, versioned table. The model identifies the type of issue, and Outrider uses that type to decide how serious it is. Old reviews keep the same rules they originally ran with.

Every finding also explains where its evidence came from:

• Observed means a code check found it directly.
• Inferred means Outrider followed it through related code.
• Judged means it came from the model’s reasoning.

Outrider checks those labels before accepting a finding. The model can’t give itself a stronger evidence label without proof.

When a review contains a critical or high-severity finding, it pauses before anything is posted. A person can approve, reject, suppress, or change each finding from the dashboard. If nobody responds before the timeout, the review expires quietly.

Each step is saved in Postgres, so a paused review can survive a restart and continue later. Outrider also keeps an append-only record of what happened. Finished production reviews are replayed in the background to confirm that the saved result still matches the recorded events.

Everything runs on your own infrastructure using your own model-provider keys. Review data stays in your Postgres database, and source context only goes to the provider you configure.

How I built it

Outrider is a Python application built with LangGraph. A pull request moves through seven stages: intake, triage, analysis, cross-file tracing, synthesis, human approval, and publishing.

The dashboard uses React and Vite. Postgres stores review data and the saved workflow state.

I tried to give every complicated part of the system one clear home. Model-provider code lives in one area, GitHub code in another, and source parsing in another. Severity rules also have their own module.

Comment placement has the strictest setup. Source files, parsed code, diffs, and GitHub all count lines a little differently. Every conversion between those formats goes through the same piece of code. That keeps the line-number math consistent and makes it easier to test.

CI checks these boundaries automatically. If an SDK gets imported from the wrong area or sensitive logic starts spreading across the project, the build fails.

Files are reviewed in parallel, so a pull request usually takes about as long as its slowest file rather than the combined time of every file. Prompt caching also worked well in the benchmark. Sonnet served 95.2% of its input tokens from cache during that test. Production performance still needs to be measured.

Challenges I ran into

Getting comments onto the correct GitHub line was surprisingly painful.

Tree-sitter, source files, diffs, and GitHub each describe positions differently. An off-by-one error can attach a warning to the wrong line and still look completely believable. I solved that by moving all coordinate conversion into one place and testing it heavily.

Pausing for human approval was another challenge. A review might wait for hours, and the process that started it may be gone by the time someone responds. The saved state has to be plain data that a new process can load safely. Resuming also has to avoid posting the same finding or comment twice.

I also ran into a race with Slack notifications. Two parts of the workflow could update the same message at nearly the same time, each with different text. I spent too long trying to protect the update itself. The fix was much simpler once I made the message come from one shared source.

The last challenge was describing the project honestly. A successful test run can easily sound like a general promise if the wording is careless. I started separating published benchmarks, internal experiments, architectural guarantees, and things I haven’t measured yet. That made it much easier to explain what Outrider can currently prove.

Accomplishments I’m proud of

Outrider can complete a full review on real pull requests in sandbox repositories. That includes receiving the webhook, reading the code, reviewing files in parallel, tracing related code, waiting for human approval, and posting the final review to GitHub.

In the tracked benchmark, the stronger model found all 22 planted vulnerable cases. Across the main benchmark and a separate provider test, all 408 recorded model responses passed schema validation.

These results come from controlled fixtures with planted examples. Real-world accuracy will need to be measured with a much wider range of pull requests.

I’m especially proud of the safety work around the model:

• Severity comes from project rules.
• Evidence labels have to match real proof.
• High-risk findings wait for a person.
• The application can’t quietly rewrite its audit history.
• Finished reviews can be replayed and checked.
• CI makes sure the code stays within those boundaries.

A lot of that work sits behind the scenes, but it’s what makes me comfortable putting an AI reviewer on a real pull request.

What I learned

One of the best prompt improvements was a single sentence telling the model that returning no findings is completely normal.

Earlier versions seemed to feel pressured to find something in every file. Fixing one false alarm often caused another type of finding to appear in the same clean code. Allowing an empty result reduced false positives from 28 to 5 in an internal test without losing any of the planted bugs.

I also learned to look closely at the assumptions behind a guarantee. A safety check near the end of a workflow doesn’t help much if an earlier step can quietly break the information it depends on. Finding those weak assumptions early led to simpler and stronger fixes.

Keeping a decision log helped more than I expected. The code can show that severity comes from a table, but it can’t explain why I chose that design. Writing down those reasons has saved me from reopening the same questions later.

What’s next for Outrider

The next step is measuring real review time and full-review cost. I have rough numbers from sandbox runs, but some files were skipped by the token budget, so I need better data before publishing those numbers.

I also want to close two review gaps I already found.

Changes that only touch Python imports or module-level constants may miss model review unless a built-in code check catches them. Deletion-only changes can also be missed because Outrider currently chooses review scopes using added lines. Removing an authorization check is an obvious example of why deletion review matters.

After that, I plan to:

• Test the two cost-saving features that currently run in observation mode.
• Tune and evaluate prompts for more model families.
• Support more languages beyond Python, JavaScript, TypeScript, and TSX.
• Measure quality on a wider range of real pull requests.
• Expand replay so it can rerun code checks against saved evidence.
• Add a GitHub Checks integration that can block a merge.

I’ve held off on merge blocking because it raises the reliability bar. Before Outrider can stop a pull request from merging, it needs to recover cleanly from crashes, timeouts, and interrupted approvals without leaving developers stuck.

Built With

• anthropic
• fastapi
• github
• langgraph
• openai
• postgresql
• python
• slack
• tree-sitter
• typescript