precedent

Inspiration

Large projects answer the same contributor questions forever. The answers exist, buried in review threads going back a decade, but they are not retrievable. A maintainer explains why the whatsnew note goes in a particular file, and three weeks later explains it again to someone else.

Worse, when a maintainer corrects a wrong answer today, that correction helps exactly one person and then evaporates.

I wanted memory that behaves like an experienced reviewer: it knows what the project has already decided, it can point at where each decision was made, and when someone senior tells it that it is wrong, it stays corrected.

What it does

Precedent ingests a repository's entire code review history and distils the durable conventions out of it. On pandas that is 86,321 comments across 21,763 pull requests, from February 2011 to August 2026, reduced to 298 conventions.

Then it does three things.

It comments on new pull requests without being asked. When a pull request opens, it reads the changed paths, finds the conventions anchored to those files, and posts them. Each line names the file that raised it and links to the pandas pull request where that convention was actually settled. If nothing specific applies, it says nothing, which is the harder half.

It answers contributor questions with citations, and refuses when it cannot. Every claim is tied to a real review thread. Citations are verified against the retrieved evidence before an answer is released, and an answer whose citations do not check out is discarded rather than shown.

It takes a correction from a maintainer, in place. Replying @precedent followed by the convention as it actually is turns that sentence into a rule. Identity comes from GitHub's signed webhook and author_association, so it knows who you are and whether you have write access without ever handling a password, running an OAuth flow, or holding a session. If the correction contradicts an existing rule, the old one is retired rather than left sitting in memory disagreeing with the new one, and it tells you which rule it retired.

How we built it

CockroachDB holds everything: the comments, the distilled rules, the evidence linking each rule to the threads it came from, and the embeddings, in VECTOR(1536) columns beside the rows they describe. A distributed vector index, rules@idx_rules_embedding, serves the agent's retrieval path, confirmed with EXPLAIN. There is no second vector store to keep consistent with the source of truth, which is the whole argument for putting it here.

The query shapes are built around one constraint learned the hard way: any metadata predicate in the same SELECT as the vector ordering defeats the index, so filtering happens in an outer query over an over-fetched candidate set.

AWS Lambda serves the entire application behind a Function URL: the API, the static page, and the GitHub webhook. S3 holds the raw ingested history, 3,801 gzipped pages, staged before transform. Both run in ap-south-1, the same region as the cluster, so a query round trip is about 200ms.

A GitHub App is how it reaches the place work actually happens. Webhook deliveries are signed with HMAC SHA-256, and writing back uses an RS256 JWT exchanged for a short-lived installation token, so the grant is revocable by uninstalling rather than by rotating a key.

Challenges we ran into

Vector distance could not decide what applies to a pull request. The first version searched rules by similarity and commented on 38 of 40 real pull requests. The cause was not a badly chosen threshold: measured across 25 pull requests, the nearest and tenth-nearest rule differ by a median of 0.10 under four different query phrasings. Every rule is a general statement about the same codebase, so every rule is similar to any description of a change to it. Retrieval now runs on path anchors, which are facts, and similarity only orders what the paths have already admitted.

The path patterns in memory were SQL LIKE, not globs. Nothing asked the extraction prompt for pandas/tests/% and nothing documented it; it is simply what the model wrote. Matching them with fnmatch meant 68 of 79 anchored rules matched nothing at all, so the agent ran entirely on its two least specific rules while appearing to work.

A null in an optional field silently discarded valid corrections. The prompt asks for a rationale "only if they gave a reason", so a maintainer stating a convention flatly produces rationale: null. Pydantic rejects that for a plain string, validation is all-or-nothing, and a real maintainer correction became "no convention stated" six times out of six. The module written to stop models failing silently was itself failing silently.

A maintainer's correction lost to five inferred rules. Asking which whatsnew file a note belongs in answered from patterns and never mentioned the correction, which sat inside the distance threshold but ranked thirteenth. Retrieval now reserves slots for what a maintainer actually said. A correction that does not surface is the one thing this system promises cannot happen.

Mangum runs the ASGI lifespan on every Lambda invocation, not once per execution environment, which rebuilt the connection pool per request and reset the spend counter so the budget ceiling capped nothing. The budget now lives in the database as a reservation taken in the same statement that checks for room, because a read-then-spend check let ten concurrent requests through a $1.00 ceiling at $0.90 spent.

Accomplishments that we're proud of

An evaluation that reports the column it loses. The same 29 questions went to Precedent and to the same model with no memory at all:

	correct	refused when it should	citations that resolve
Precedent	8/24	4/5	43/43
gpt-4o-mini alone	9/24	0/5	0/24

Memory did not improve accuracy. The baseline scored one question higher, which on 24 questions is noise. What changed is whether the answer can be trusted. Five questions are deliberately unanswerable from review history: the bare model answered all five, confidently inventing project policy on release schedules, governance and credentials. Of the 24 pull requests it cited, none resolve to a real discussion in the corpus. Precedent refused four of five, and every one of its 43 citations resolves.

Memory does not make the model smarter. It makes it accountable, which is the difference that matters when a contributor cannot tell a confident right answer from a confident invented one.

Maintainer identity with no login anywhere. No password, no OAuth flow, no session, no account to create. GitHub already knows who has write access, and a signed webhook is how you ask.

It runs on about $1.40 of API credit, including the evaluation.

What we learned

That the interesting failures are the invisible ones. A 502 gets fixed in ten minutes. A schema that silently discards a maintainer's correction, an index that was dropped for a backfill and never rebuilt, a citation whose label and target disagree: those keep working well enough to look fine, and each one was found by measuring something rather than by reading the code.

Also that abstention is a feature you have to design for and then measure. It does not emerge from a better prompt.

What's next for Precedent

Extraction currently runs as a batch over the whole history. It should run incrementally as review happens, so a convention settled this morning is citable this afternoon.

Beyond that, the interesting direction is telling a maintainer when the project has quietly stopped following one of its own conventions. The evidence for that is already in the schema: every rule carries when it was last evidenced.

Built With

• amazon-web-services
• asyncpg
• aws-lambda
• cockroachdb
• fastapi
• github-api
• openai
• pydantic
• python
• sqlalchemy
• vector-search