Cite 3

Does the source exist, match, and actually say that?

Comment

Inspiration

AI assistants write essays, reports and legal briefs full of citations. Some of those sources do not exist. Others exist but say nothing like what the sentence claims. The tools that check citations mostly ask one question: does this source exist? We wanted to know how much that single check really catches, and what a better check would look like.

What it does

Cite 3 checks every citation in a text three ways:

  • Does it exist? The paper is looked up in Crossref and OpenAlex.
  • Are the details right? Title, year and first author are compared with the real record.
  • Does it say that? An AI model reads the paper's abstract and the sentence that cites it, and says whether the abstract supports, partly supports, contradicts, or is unrelated to the sentence.

Paste a text with a reference list and each citation comes back green, yellow or red, with the reason. When a publisher shares no abstract, Cite 3 says it cannot verify instead of guessing.

How we built it

  • Plain JavaScript with no dependencies. The same engine runs in the browser (levels 1 and 2) and in Node.
  • Free, open data: Crossref and OpenAlex, with Semantic Scholar as a fallback for abstracts.
  • Gemini 3.1 and 3.5 Flash-Lite (free tier) as the level 3 judge, with a fixed prompt that treats the abstract as untrusted text.
  • A measured test set: we sample real papers, then generate cases whose correct answer is known (correct, invented, wrong details, real but unrelated, real but contradicted). Every case is checked once, and we compare what each level would have flagged.
  • A GitHub Actions workflow runs the whole benchmark in the cloud. 25 unit tests cover the parser, the three levels, the model fallback and the test generator.

Challenges we ran into

  • Building from a phone. The whole project was written, tested and deployed from a smartphone, with GitHub's web interface and no laptop.
  • Messy references. Titles with question marks, quotation marks, soft hyphens and two papers sharing one title all broke the first version. We found each of these by reading every error in the first benchmark run, fixed them, and then re-ran on a fresh sample so the final numbers are not tuned to the test set.
  • Free-tier limits. Rate limits and busy models forced retries, a model fallback and a slow, careful run.

Accomplishments that we're proud of

On 200 test cases (40 per type) from a fresh sample of real papers:

  • A check that only looks at whether a source exists catches 0 of 80 real papers cited for something they do not say. Cite 3 catches 76 of 80.
  • Wrong year, author or title: 0% with the existence check, 100% with level 2.
  • Correct references flagged by mistake: 0 of 40.
  • Invented references: 78% caught by the existence check alone, and all 40 raise at least a warning.

What we learned

  • Whether a source exists tells you very little about whether it is cited correctly.
  • Reading the errors of a first run is worth more than a bigger run. Four real bugs hid behind the 5% false alarms.
  • Reporting a fresh sample after fixing the code keeps the numbers honest.
  • Small honest limits (abstract only, synthetic test cases, 40 cases per type) are easier to defend than big claims.

What's next for Cite 3

  • Test on citations from real AI answers, not only generated ones.
  • Read full papers when they are open access, instead of only abstracts.
  • Browser extension and a copy-paste mode for Word and Google Docs.
  • A larger test set with harder cases, such as sentences that are only partly supported.

Built With

Share this project:

Updates

Submission history