CiteProof — Cross-Field Citation Verification
The Problem
Academic research depends on trustworthy references, yet citations generated by AI tools can contain subtle errors that are difficult to detect. A reference may look convincing and even point to a real paper while combining the wrong title, authors, publication year, venue, or DOI.
Most basic citation checkers focus on whether a paper exists or whether its title can be found. This can allow chimera citations—references assembled from mismatched details of real publications—to pass unnoticed.
CiteProof was built to address this gap: verifying whether a single scholarly record actually supports the information presented in a citation.
The Solution
CiteProof is a deterministic citation-auditing tool that cross-checks academic references against live records from Crossref and OpenAlex. It evaluates multiple bibliographic fields together rather than relying on existence alone.
Each reference receives one of three results:
- VERIFIED: One indexed record consistently supports the citation's key fields.
- MISMATCH: A real record exists, but one or more citation fields do not match. CiteProof identifies inconsistencies and offers a corrected reference.
- NOT_FOUND: Neither index provides a sufficiently close match. This result indicates that manual verification is needed, not necessarily that the reference is fabricated.
The goal is to make citation verification more explainable, reproducible, and useful for researchers, students, and anyone working with AI-generated bibliographies.
Inspiration
The central idea behind CiteProof is that a reference should not be considered reliable simply because a paper with a similar title exists.
When an AI system produces a plausible-looking citation, checking only its title can miss incorrect authors, swapped publication years, or DOIs belonging to different papers. These errors can undermine academic credibility even when the reference appears legitimate at first glance.
CiteProof approaches the problem as one of cross-field consistency: can one real scholarly record explain the citation as a whole?
How It Works
- Input: Users paste a list of references or upload a PDF containing citations.
- Extraction: CiteProof extracts reference text and normalizes bibliographic information.
- Candidate retrieval: It gathers candidate publication records from Crossref and OpenAlex.
- Field-level evaluation: The system evaluates evidence across the title, first author, year, DOI, and remaining citation text.
- Classification: Each reference is classified as VERIFIED, MISMATCH, or NOT_FOUND.
- Correction and reporting: When a mismatch is detected, CiteProof links to the closest real record and offers a corrected reference where possible.
Users can also compare CiteProof against two baseline checkers on the same input, making differences in verification behavior visible.
What Makes It Different?
Cross-field verification instead of existence checking
CiteProof distinguishes between finding a similar publication and verifying that the publication supports the actual citation. This is particularly important when AI-generated references combine genuine bibliographic details incorrectly.
Explainable, deterministic decisions
CiteProof does not use an LLM to make its verification decisions. Its scoring and classification pipeline uses bibliographic evidence and calibrated thresholds, making its decisions easier to inspect and reproduce.
Evidence-backed corrections
When a citation is inconsistent, CiteProof provides a link to the closest real scholarly record and offers a corrected reference rather than merely flagging an error.
Benchmark-driven evaluation
The project includes a synthetic benchmark containing 400 Crossref records, divided into valid, corrupted, and fabricated references. Corruptions include swapped years, authors, titles, and venues, while fabricated references are constructed by combining details from multiple real records.
Thresholds are tuned on a training split, with a false-alarm-rate constraint of at most 5%. Performance is evaluated against a weak existence-based baseline and a stronger baseline that also checks the first author.
A separate manually reviewed collection of real chatbot-generated references is used to evaluate performance beyond synthetic examples. This wild set is not used for threshold tuning.
How I Built It
CiteProof is implemented in Python, with a web interface, command-line interface, and containerized deployment option.
The core system is organized into modules for configuration, text normalization, HTTP requests, Crossref and OpenAlex integration, baseline checking, scoring, verification, extraction, and reporting.
The web application accepts pasted references and PDF uploads. The CLI supports individual citation checks, PDF processing, offline extraction, and JSON report generation. Docker provides a reproducible deployment path, while the supplied Render configuration supports hosted deployment.
The verification workflow is designed to be auditable. A single script runs the benchmark and checks documented project claims against measured outputs. Results are written to JSON files so that reported performance can be traced back to actual runs.
Challenges and Lessons Learned
One of the main challenges was distinguishing a citation that is genuinely correct from one that merely resembles a real publication. This required evaluating several fields together rather than treating a successful title lookup as sufficient evidence.
Another challenge was designing an evaluation process that avoids overstating performance. Thresholds must be calibrated on training data rather than on the references used for final evaluation. Synthetic tests are useful for controlled experiments, but they cannot fully represent the diversity of citations produced by real AI systems. That is why CiteProof also includes a separate wild evaluation set.
The project also highlights the importance of uncertainty. Crossref and OpenAlex do not cover every book, publication, or piece of grey literature. A missing match must therefore trigger manual review rather than an automatic accusation of fabrication.
Building CiteProof reinforced the value of transparent decision rules, independent evaluation data, reproducible benchmarks, and explicit limitations in research-oriented software.
Technology Stack
- Language: Python
- Bibliographic data sources: Crossref API and OpenAlex API
- Web application and API: FastAPI
- Command-line interface: Python CLI
- PDF processing: Citation extraction pipeline
- Benchmarking: Synthetic dataset, manually labelled wild dataset, and calibrated baselines
- Deployment: Docker and Render
Limitations
CiteProof currently supports English-language references and single-column PDFs. Its verification coverage depends on Crossref and OpenAlex, so incomplete indexing can lead to NOT_FOUND results for legitimate publications. Corrected references also omit author initials because the underlying citation records store family names only.
The wild evaluation set is relatively small and covers three chatbot models: Gemini Flash, Gemini Pro, and ChatGPT. Two models come from the same vendor, and the labels were produced with AI-assisted lookups and checked against cited records. A separate AI agent labelled 30 rows independently, so the reported agreement is between two AI labellers rather than an AI labeller and a human.
These limitations are documented so users can interpret the results appropriately.
Future Scope
Potential next steps include broader scholarly database coverage, support for additional document layouts and languages, improved author-name handling, and evaluation on larger independently labelled datasets. Further work could also explore integrations with reference managers and academic writing workflows.
Impact
Incorrect citations can waste researchers' time, weaken literature reviews, and make it harder to trace evidence back to its original source. CiteProof aims to make reference verification more accessible by turning a manual, error-prone task into a structured process with explicit evidence and actionable results.
CiteProof does not attempt to determine whether an entire paper is true or whether AI-generated writing is detectable. It focuses on a narrower, testable problem: whether the bibliographic details of a citation are supported by a real scholarly record.
By making citation inconsistencies easier to identify and investigate, CiteProof contributes to more reliable research workflows and more responsible use of AI in academic work.
Log in or sign up for Devpost to join the conversation.