Mizan ميزان
Source-available for review only. This repository is published so its author's technical work can be evaluated. It is NOT open source. All rights reserved: no copying, cloning, use, or derivative work is permitted without the author's prior written permission. See LICENSE.
Claim-level groundedness verification for Arabic and cross-lingual RAG.
Mizan decomposes a RAG answer into atomic claims, verifies each claim against the retrieved evidence with a multilingual NLI cross-encoder, calibrates the scores, separates hallucination from retrieval failure as two measured signals, and suppresses unsupported claims before they ever reach the user.
question ──> RAG answer ──> decompose ──> verify (NLI) ──> triage ──> aggregate ──> report
│ │ │ │
atomic claims 3-way verdict hallucination weighted
(ar / en) per claim vs retrieval groundedness
failure score + ECE
Why
Most factuality tooling is English-centric (MiniCheck, AlignScore, RAGAS, SAFE). The Arabic side of 2025-2026 (AraHalluEval, HalluTruthQA) provides detection benchmarks over finished outputs. What does not exist is a calibrated, interactive claim-level verification layer for Arabic and cross-lingual RAG that:
- issues three-way verdicts (supported / contradicted / no-evidence) instead of a binary,
- re-verifies every no-evidence claim under maximal retrieval before calling it a hallucination,
- reports calibration (Expected Calibration Error) as a first-class metric, and
- measures selective suppression as a risk-coverage tradeoff.
Design decisions stated plainly
- The answer-level score is a groundedness score, not a truth probability. It measures how well the retrieved corpus supports the claims. A wrong corpus yields a confident wrong score. This limitation is documented, not hidden.
- Aggregation is a weighted mean of claim scores (length x centrality by default). The weights are part of the report payload, never implicit.
- NLI cross-encoders operate at sentence granularity, so atomic decomposition is a correctness precondition, not an optimization.
Quickstart
pip install -r requirements.txt
python -m unittest discover -s tests -v
python demo/run_demo.py # uses the real mDeBERTa NLI model when transformers is installed
from mizan import MizanPipeline, TfidfRetriever
from mizan.verify import NLIVerifier
retriever = TfidfRetriever.from_jsonl("demo/sample_corpus.jsonl")
verifier = NLIVerifier() # MoritzLaurer/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7
report = MizanPipeline(retriever, verifier).check_answer(answer_text)
print(report.to_dict())
Evaluation design
| System | Balanced Acc. | ECE | Risk-Coverage AUC | Latency / claim |
|---|---|---|---|---|
| Mizan (mDeBERTa cross-lingual) | ||||
| MiniCheck (cross-lingual) | ||||
| Translate-then-Verify | ||||
| LLM-as-Judge |
Gold set: 200-300 hand-annotated claim-evidence pairs (3-way labels) drawn from the target dataset. Protocol follows the AVeriTeC 2025 shared-task posture: reproducible, open-weights, under one minute per claim.
Status
Seed stage. The pipeline, calibration, triage, and aggregation logic are implemented and unit-tested. Gold-set annotation and the baseline comparison table are the active work items.
License
Proprietary. All rights reserved. Source-available for review only, no license to copy, use, or redistribute. See LICENSE.
ميزان
طبقة تحقق من الإسناد على مستوى الادعاء للأنظمة العربية ومتقاطعة اللغة.
يفكك ميزان إجابة نظام الاسترجاع والتوليد إلى ادعاءات ذرية، ويتحقق من كل ادعاء مقابل الأدلة المسترجعة عبر نموذج استدلال لغوي متعدد اللغات، ويعاير الدرجات، ويفصل الهلوسة عن فشل الاسترجاع كإشارتين مقاستين، ويحجب الادعاء غير المسند قبل وصوله للمستخدم.
المبدأ الصريح: الدرجة المعروضة هي درجة إسناد للمصادر المسترجعة وليست احتمالاً لصحة المعلومة في العالم، وهذا القيد موثق لا مخفي. الحكم ثلاثي: مدعوم، متناقض، بلا دليل. وأي ادعاء بلا دليل يعاد فحصه بشروط استرجاع قصوى قبل تصنيفه هلوسة.
Built With
- ai-safety
- arabic
- llm
- machine-learning
- natural-language-processing
- nli
- python
- rag