PHI — Synthetic Data Quality & Hallucination Risk Benchmark

Inspiration

Synthetic datasets power the next wave of Small Language Models (SLMs), but training on model-generated outputs introduces severe risks: synthetic echo chambers, vocabulary decay, and model collapse. Standard benchmarks measure model accuracy after training, but fail to evaluate whether the underlying datasets are degraded. We built PHI-Bench to audit, score, and sanitize synthetic datasets before they poison training pipelines.

What it does

PHI-Bench is an interactive dataset quality and benchmarking suite hosted on Hugging Face Spaces. It audits synthetic datasets through three core layers:

• Multi-Signal Feature Engine: Extracts 8 key structural metrics per sample, including embedding cosine drift, causal LM perplexity, token entropy, and lexical diversity.
• Density & Cluster Mapping: Projects semantic distributions into 2D space using UMAP and HDBSCAN to isolate outlier clusters and narrow vocabulary loops.
• Calibrated Scoring & Pruning: Trains an XGBoost model to output a Predicted Hallucination Index ($\text{PHI} \in [0, 100]$), allowing developers to set absolute thresholds or relative percentiles to filter out poisonous samples before fine-tuning.

How we built it

• UI & Hosting: Built with Gradio and Plotly for real-time visualization, deployed live on Hugging Face Spaces.
• Feature Pipeline: Python-backed multi-signal engine measuring token distribution, embedding drift via sentence transformers, and causal perplexity.
• Clustering & Analytics: UMAP manifold reduction paired with HDBSCAN density clustering to detect structural dataset failures.
• Predictive Risk Model: Trained an XGBoost pipeline calibrated to quantify hallucination risk and output the normalized PHI metric.

Challenges we ran into

• Real-Time Feature Extraction: Calculating embeddings, perplexity, and token entropy over large text streams caused latency spikes. Solved by vectorizing text operations and optimizing batch processing.
• Noise Reduction in Clustering: Tuning HDBSCAN parameters so the system accurately flags synthetic text loops without falsely penalizing complex, valid prompts.

Accomplishments that we're proud of

• Developed an end-to-end framework that turns abstract dataset degradation into measurable, actionable metrics.
• Created an intuitive UMAP/HDBSCAN visual suite that immediately exposes data drift patterns.
• Built a fully working, interactive demo hosted live on Hugging Face Spaces for the Reverie Hacks Datathon.

What we learned

• Single-metric evaluation (such as perplexity alone) fails to catch subtle synthetic loops; multi-signal verification is critical.
• Manifold learning combined with gradient boosting offers an extremely efficient, interpretable safety gate for dataset curation.

What's next for PHI — Synthetic Data Quality & Hallucination Risk Benchmark

• PyPI Package: Releasing phi-bench as a Python library to automatically filter datasets directly inside PyTorch and Hugging Face training scripts.
• Domain-Specific Scorer: Expanding metric heads specifically tuned for synthetic code, mathematical proofs, and JSON structures.
• Multimodal Auditing: Extending PHI scoring to synthetic visual-language datasets and image-prompt pairs.

Built With

• gradio
• hdbscan
• huggingface
• numpy
• pandas
• plotly
• python
• pytorch
• scikit-learn
• sentence-transformers
• umap
• xgboost