Fairvis-Model

Inspiration

Modern vision–language models often describe people in ways that unintentionally leak sensitive attributes, reinforce stereotypes, or make subjective judgments. While these issues are usually “fixed” using post-hoc filters, that approach is brittle, opaque, and hard to audit. We were inspired by a simple question: what if fairness was enforced inside the model itself instead of being patched afterward? FairVis-Model was built from this idea—making bias-aware, responsible AI practical even for small teams and low-resource environments.

What it does

FairVis-Model is a lightweight, fairness-aware vision–language model that generates neutral, factual, and deterministic descriptions of people in images. Specifically, it: Extracts non-identity visual cues from images. Generates descriptions without: Sensitive attribute inference, Stereotype amplification, Subjective judgments (e.g., attractiveness or professionalism). Produces repeatable outputs for the same input, enabling auditability. Runs efficiently as a Small Language Model (SLM) suitable for edge or regulated deployments.

How we built it

🧠 System Architecture Image ↓ BLIP (Vision Encoder) ↓ Neutral Visual Cues ↓ ERNIE-4.5 (0.3B) + LoRA ↓ Bias-Aware, Deterministic Text

We designed FairVis-Model as a modular, fairness-first pipeline: Vision Encoding: We used BLIP to extract neutral semantic visual cues (e.g., clothing or accessories) without direct identity reasoning. Language Generation: We selected ERNIE-4.5 (0.3B) as a compact language backbone to ensure low latency and deterministic generation.

Embedded Fairness via LoRA: Instead of retraining the full model, we fine-tuned ERNIE using LoRA adapters on fairness-constrained data, modifying less than 0.3% of the parameters.

Training Data: A balanced subset of the FairFace dataset was converted into supervised instruction–response pairs with strictly neutral and factual language. Deterministic Inference: We disabled sampling during generation to guarantee consistent outputs for the same input image.

Challenges we ran into

Finding perfect dataset Avoiding implicit bias: Even neutral prompts can trigger demographic or emotional assumptions. Designing prompts and training data that truly constrained the model required multiple iterations. Balancing fairness and descriptiveness: Removing bias without producing empty or overly vague descriptions was a key challenge.

Reproducibility:

Mixed-precision training and non-deterministic decoding caused inconsistencies early on, which we resolved by enforcing strict float32 and deterministic inference. Model size vs. capability: Ensuring the model remained small while still producing coherent, useful descriptions required careful architecture and parameter choices. Accomplishments that we're proud of Built a fairness-first vision–language model, not just a filtered one. Achieved deterministic, auditable outputs suitable for regulated use cases. Reduced training cost by over 99% using LoRA instead of full fine-tuning. Delivered an end-to-end, reproducible pipeline with open documentation. Demonstrated that responsible AI does not require massive models.

What we learned

Fairness is far more effective when embedded into model behavior rather than enforced after generation. Small, well-designed models can outperform larger systems in trustworthiness and deployability. Determinism is critical for auditing and governance, yet often overlooked in modern AI systems. Responsible AI is not just an ethics problem—it is a systems design problem.

🔗 Resources

Hugging Face Model: https://huggingface.co/dishana11/ernie-fairvis-sft GitHub Repository: https://github.com/dishana11/fairvis Training Data set: https://github.com/joojs/fairface

What's next for Fairvis-Model

Expand training to larger and more diverse datasets with stricter fairness constraints. Add quantitative bias evaluation metrics and benchmark against existing VLMs. Release a lightweight API for easy integration into accessibility and compliance tools. Explore on-device deployment for mobile and edge environments. Extend the framework beyond faces to other sensitive vision tasks.

Built With

• ernie
• fairfacedataset
• googlecolab
• huggingfacedataset
• python
• unsloth