-
-
Fact-verification engine: checks a claim against a source, returns confirmed, contradicted, or clouded — never a bare yes/no.
-
Self-test: fed it a claim about AI hallucination with a swapped entity baked in. It caught the mismatch and backed off to clouded.
-
Every stage is swappable and inspectable — nothing here is a black box the answer just falls out of
Inspiration
A confident wrong answer can be more damaging than an honest "I don't know."
Most LLM-based fact-checking systems are designed to produce an answer, even when the available evidence is insufficient. I built Anti-Cassiopeia to challenge that behaviour.
Its name comes from the Greek myth of Cassiopeia, who was punished for boasting of a beauty she could not substantiate. The project takes inspiration from that very failure: making claims without evidence to support them.
Give Anti-Cassiopeia a claim, and it returns one of three verdicts: confirmed, contradicted, or undetermined, accompanied by the quotation, source, and exact location used to reach its conclusion.
When the evidence is insufficient, it says so instead of guessing. Because uncertainty should be an acceptable result, not something an AI system needs to hide.
What it does
Anti-Cassiopeia uses 4 domain adapters and 1 shared verification engine to check claims across different sources of evidence.
- substrate: Verifies hardware specifications against stated conditions. For example, whether a Bar30 pressure sensor is rated to survive a depth of 500 metres. It checks real manufacturer datasheets or user-uploaded documents, using NVIDIA Nemotron Ultra for evidence extraction.
- quant: Verifies statistical claims about datasets by recomputing the requested values directly from the actual CSV files, rather than estimating or relying on a model's calculations.
- silicon: Checks FPGA timing and register claims against real Vivado timing reports and CMSIS-SVD files.
- searchlight: Verifies whether live web sources actually support a claim. It uses Tavily search, with DuckDuckGo and Wikipedia fallbacks, followed by hybrid BM25 and vector retrieval.
Every verdict includes a quotation, source, and locator, making the result straightforward to inspect. Reviewing the evidence becomes a lookup rather than another research task.
Most importantly, undetermined is a first-class verdict, not an error or failure state.
How I built it
The architecture centres on a single VerificationEngine and a set of pluggable DomainAdapters. Each domain follows the same verification contract while using the methods best suited to its evidence.
Rather than relying on one model for every task, I evaluated models independently for each pipeline stage through live A/B testing.
Model selection: measured, not assumed
I selected NVIDIA Nemotron Ultra (nvidia/Nemotron-3-Ultra-550b-a55b) for two specific stages:
- Substrate evidence extraction: It was the only model tier to maintain a clean record of zero dangerous misses across
3independent runs (n=15,40, and40). Nemotron Super and Nano both produced dangerous misses atn=40that had not appeared in their initialn=15evaluations. - Searchlight stance judgment: It handles the hardest cases, particularly determining whether a passage actually supports a causal or relational claim. On one genuinely difficult evaluation set, Nemotron Ultra scored
5/5, compared with2/6for GPT-OSS-120B.
I selected GPT-OSS-120B for Searchlight's main evidence-extraction stage. No Nemotron tier outperformed it at an adequate sample size, and GPT-OSS-120B tied 4/4 with Tavily active while costing approximately one-seventh as much.
I also integrated:
- Qwen/Qwen3-Embedding-8B: Generates embeddings for the vector component of hybrid retrieval.
- Tavily: Primary live web-search backend, with DuckDuckGo and Wikipedia fallbacks.
- BM25 + vector retrieval: Combines keyword-based and semantic search, with results fused using Reciprocal Rank Fusion (RRF).
All model calls run through Nebius Token Factory, providing one account and one API surface across the pipeline. This made it practical to compare models at individual stages and select them using measured performance rather than assumptions.
The final verification step is deliberately independent of model judgment: ordinary code checks every proposed quotation against its original source.
I didn't choose the biggest model. I chose the model that performed best for each job.
Challenges I ran into
One of my biggest challenges was model selection. The most obvious model choice was not always the best one.
GPT-OSS-120B outperformed Nemotron Ultra on cost-adjusted accuracy in one pipeline stage, but performed significantly worse in another. I discovered this through live A/B experiments at increasing sample sizes, not intuition.
I also learned that small benchmarks can be misleading. The Super and Nano models both appeared promising at n=15, but produced dangerous misses when evaluated at n=40.
The benchmarking process itself uncovered and helped me fix 4 genuine bugs:
- Faith-claim calibration bug: The system incorrectly confirmed a faith-based claim because its source merely said "according to tradition." It confused evidence of a traditional belief with evidence that the underlying event had occurred.
- Stance-detection limitation: A rule was too strict to contradict a false claim unless a single source directly rebutted it. This prevented the system from combining relevant evidence correctly.
- Compound-claim decomposition bug: The decomposition model silently rewrote a sub-claim into its own negation, potentially causing the system to verify the opposite of what was originally asked.
- Environment-loading bug: A
load_dotenv()issue silently disabled Tavily in every stress-test run before2026-10-25. Production was unaffected. After correcting the issue,3 of 4round-1 failures were resolved.
These were not merely theoretical weaknesses. I discovered them through experiments, investigated their causes, implemented corrections, and incorporated regression tests to prevent their return.
Accomplishments I'm proud of
I evaluated Anti-Cassiopeia against competing approaches using the same evidence and claims, with independently verified ground truth.
1. Open-web claim verification
Across 88 independently verified open-web claims, Anti-Cassiopeia achieved:
- 96.9% precision, with
2incorrect answers out of64answered claims. - Tavily + Gemini:
95.3%. - Generic RAG:
93.2%. - The same pipeline using a single stance model:
92.6%.
The precision confidence intervals overlap: 89.3–99.1% for Anti-Cassiopeia and 88.5–98.2% for Tavily + Gemini. I therefore report this as a promising lead, not proof of statistical superiority.
2. Handling claims that cannot be settled by web evidence
Across 14 faith-based claims spanning 7 traditions, Anti-Cassiopeia returned undetermined for all 14.
In comparison, generic RAG confidently confirmed 9 of those claims, including the Fátima apparition and Muhammad's Night Journey.
The distinction is important: evidence that a tradition describes or believes in an event does not automatically establish that the event itself occurred.
3. FPGA timing and register verification
Across 116 real FPGA timing and register claims, Anti-Cassiopeia produced 0 incorrect confident answers.
Every competing system tested produced between 3 and 15 incorrect answers.
4. Exact computation over datasets
Across 216 dataset-statistic claims, Anti-Cassiopeia produced 0 incorrect answers.
GPT-OSS-120B produced up to 59 incorrect answers when the dataset file was pasted directly into its prompt.
These results demonstrate the value of performing exact computations using code rather than asking a language model to reproduce statistical calculations from a large table.
I deliberately report the limitations alongside the achievements. Overstating performance would reproduce exactly the failure mode this project was created to prevent.
What I learned
I learned that no single model is the best choice for every task. Model selection must happen at the individual pipeline-stage level, and the decision must be supported by experiments.
I also learned that benchmark size matters. A model comparison that looks successful at n=15 can conceal a dangerous failure that only appears at n=40.
Perhaps my most important lesson was that a benchmark is only as trustworthy as the instrumentation behind it.
My own load_dotenv() bug silently disabled Tavily in every pre-2026-10-25 stress-test run. I did not consider the problem resolved merely because the code had been changed. I trusted the correction only after confirming that Tavily was working live and that 3 of the affected 4 round-1 failures were actually resolved.
Building a verification system also means verifying the experiments used to evaluate that system.
What's next for Anti-Cassiopeia
My next priority is to extend the live A/B testing methodology to smaller Nemotron tiers, including Nemotron Nano and Nemotron 3.5 Lightning.
Cost-sensitive teams should not have to choose between affordability and safety without reliable comparative evidence. I want to evaluate these models at sufficiently large sample sizes to understand their actual trade-offs.
On the retrieval side, I plan to expand hybrid BM25 + vector retrieval beyond its current 13-question evaluation set.
I also want to address the remaining causal-claim limitation in searchlight. The current judge_comparative_stance() implementation recovers only 2 of 12 such misses. I intend to investigate this with a larger evaluation sample before generalising or deploying another fix.
My goal remains the same: a verification system that can produce a defensible answer when evidence supports one, identify contradictions when they exist, and recognise when the available information simply isn't enough.
Anti-Cassiopeia is built to make evidence matter more than confidence.
Log in or sign up for Devpost to join the conversation.