Repoherald

You found it already — let's make sure you don't have to find it again.

Comment

You already found it. Find it again.

You scroll past a repository on Instagram, tap save, and the next day it is gone. Not deleted — buried. The saved folder is a grid of unlabelled thumbnails: no titles, no notes, no search. You know you found the thing. You cannot find it twice.

repoherald rescues them. Send the post to repoherald instead of saving it, and it comes back identified — the actual owner/repo, its stars, its language, its description — catalogued, and filed into a private list with a note on why you kept it. Claim your handle and everything you ever sent is waiting under it, searchable.

The step in the middle used to be a person

That middle step — which repository is this post about? — was a human being. The landing page says so out loud, in the subhead, today: "A person looks up the repository behind it."

That person was me. It is why the catalogue could never grow faster than somebody's evening. Automating that one step is the whole of this submission: NVIDIA Nemotron on Nebius Token Factory does the lookup now, and you can run it yourself at https://repoherald.com/identify.

Stated plainly, because a judge who reads our landing page will notice: that sentence is still on the page, and it is still true of the production ingest path. resolveRepo has exactly one caller today — the public /api/identify endpoint. The swap into ingest is the next step, and was deliberately not done in the final week of this hackathon: repoherald.com is a live product that is also being judged elsewhere into mid-November, and migrating a running pipeline to hit a submission deadline is how you take a product down. What is submitted is the identifier itself — public, rate-limited, measured, and degrading rather than failing when the balance runs out.

Why it is harder than it sounds

A post names its repository in one of three places, in descending fidelity — and each one damages the name differently:

Where the name hides How it arrives
The caption Clean, when it is there at all
Text on screen (OCR) Glyphs mangled: HKUDS/ViMax → HKUD5/ViMax, 5 for S, 1 for l
The narration (speech-to-text) Punctuation is spoken: yt-dlp/yt-dlp → "yt dlp slash yt dlp"

The narration case is the interesting one. The slash was said aloud; the hyphens were never said at all. No regex recovers that. A reasoning model does.

The ladder

Five rungs, cheapest first, stopping at the first candidate GitHub confirms:

  1. A literal github.com link — regex, zero model calls. A quarter of the corpus resolves here for nothing.
  2. Nemotron fast tier — nvidia/nemotron-3-super-120b-a12b, over all three texts at once.
  3. Nemotron deep tier — nvidia/Nemotron-3-Ultra-550b-a55b, told which candidates GitHub already rejected, so it repairs instead of repeating itself. This is the rung that undoes spoken punctuation.
  4. Tavily search — only when the post named nothing at all. That gate is itself a measured decision: with search running whenever the earlier rungs merely failed, the OCR mode reached 89.5% correct and 10.5% wrong, because a half-read name sent to search comes back as a confident near-miss. If a grounded candidate was proposed, the post does contain a name, and the honest answer is "I don't know".
  5. Nemotron verification — the only rung that can say no. It compares the post against the candidate's real GitHub description and vetoes a near-miss. Search never abstains on its own — its measured unresolved rate is 0%, because search always returns something and the ranking always picks one — so this rung exists to supply the abstention search cannot. It took the describes-but-never-names mode from 34.1% wrong to 13.3%. An absent or unparseable verdict counts as rejection, never approval. But a verification call that throws degrades rather than rejects: the candidate survives at low confidence carrying needsReview, because an outage must not read as "no such project".

Everything the search rung returns carries needsReview: true and is never auto-catalogued. It is a pre-filled candidate for a human review queue, not a result — and /identify says so on screen.

The two rules that make it safe to automate

Rule one: Nemotron proposes, GitHub decides. Every candidate is checked against the real GitHub API. One that does not resolve is discarded however confident the model sounded. The model never names the output; it only narrows the search.

Measuring it taught us that rule was not enough. Given a post that describes a tool and never names it, the model does not abstain — it free-associates to whatever well-known project owns the niche. A description of chrisgreg/boop came back as binwiederhier/ntfy. Both are real, so the existence check confirmed both. Validation proves a repository exists; it cannot prove it is the one the post was about. On that corpus the system was confidently wrong 14.6% of the time.

Rule two, now sitting in front of rule one: a name the model proposes must survive as a run of characters in the post itself, once case, spacing and punctuation are thrown away. vimax survives the OCR dump HKUD5/ViMax; ytdlp survives the spoken "yt dlp slash yt dlp" — so genuine damage still repairs. But a name that appears nowhere is refused before GitHub is ever asked. That took the mode to 0% wrong — it abstains instead — and left the rest of the corpus untouched. Read that figure against a search-less ladder: switching search on reintroduces wrong answers from the other direction, which is exactly what rung 5 is there to clamp.

Both rules are enforced, not promised. One test feeds the ladder a model that fabricates repositories at confidence 1.0 and asserts nothing comes back. Another feeds it a real repository the post never mentions, with a lookup that confirms everything, and asserts the same.

What measuring it changed

We ran the no-model baseline first, so the model's score would mean something.

mode                no model   with Nemotron
caption_url            98.2%       98.2%      (regex; no model needed)
caption_plain           0.0%       98.2%
screen      (OCR)       0.0%       70.2%
narration   (ASR)       0.0%       93.0%
─────────────────────────────────────────────
228 cases              24.6%       89.9%      1 wrong (0.4%)

The describes-but-never-names figures quoted above — 14.6%, 34.1%, 13.3% — come from a separate 41-case corpus whose repository names were scrubbed out of their own descriptions. It sits outside the 228 deliberately: a post that never names its repository has no answer a regex could ever reach, so folding it in would flatter the model by lowering the baseline it is measured against.

Two of our four input classes were not "worse" before. They were at zero. Completely unreachable.

The single wrong answer is arguably ours: our catalogue stores open-slide/open-slide, the model returned openslide/openslide, and the model is right about which one exists.

The eval reports three outcomes, deliberately not two — correct, wrong (resolved to a real repository that is not the right one), and unresolved. Only wrong is harmful: an unresolved post falls back to the review queue, which is where every post went before any of this existed, while a wrong one is filed silently against somebody's saved post. So wrong is reported on its own line and never folded into an accuracy percentage.

On the inputs, stated plainly: the 57 repository names are real, pulled from repoherald's production database, each originally identified by a person reading a post. The mangled text those people read was never stored — so the OCR and narration inputs are synthetically generated, from a seeded deterministic generator, each mode reproducing a failure the ingest pipeline hit in production (README.md documents each one). That is said here, in the README, and in the video's description. It also means a judge running the same command gets the same number rather than one that moves each time it is quoted.

Two further honest limits. This measures reconstruction from damaged text, not end-to-end production accuracy. And because the ground truth is a closed set of 57, a plausible-but-wrong real repository scores as wrong where production would accept it — so the real-world confidently-wrong rate is higher than the eval's, not lower.

The tiering surprise

We went looking for a small model for the cheap pass. We measured instead of assuming:

NVIDIA-Nemotron-3-Nano-30B-A3B    25-34s    (4 measurements, no cold-start effect)
nemotron-3-super-120b-a12b        1-2s      -> fast tier
Nemotron-3-Ultra-550b-a55b        0.8-1.2s  -> deep tier

The 30B model is reproducibly ~20× slower than the 550B one on our account. So the tiers are split by measured latency and output shape, not parameter count — which is not the lesson we expected to learn, and is not discoverable from /models, the pricing page or the model card.

What is new during the Submission Period

repoherald existed before 26 August. Every line of model inference in it is new.

  • src/lib/nemotron.ts — Token Factory client: two tiers, retry on 429 and 5xx, 401 never retried, live /models verification, defensive JSON extraction
  • src/lib/resolve-repo.ts — the five-rung ladder and the two measured guards
  • src/lib/tavily.ts — search for a project a post describes but never names
  • /api/identify and /identify — the public demo, with hard request ceilings
  • scripts/nemotron-doctor.ts — validates configured model ids against the live catalogue, so a renamed id fails at deploy instead of 404ing a real user
  • A test suite of 143 assertions, none of which need an API key, plus a scored eval with a --baseline mode so the 24.6% figure is reproducible

Roughly 1.1M tokens went into measuring it properly — including the runs that told us two of our own ideas were wrong.

Try it

Built With

Share this project:

Updates

Submission history