Fairvis-SLM

• 

  what we propose

• 

  gender biasness as future agi depends highly on current datasets and this could be very dangerous

• 

  fair vis slm could be very useful at large scale for computer vision in agi

• 

  gender shades and how current datasets have biased data

Inspiration

Our work began with a simple but urgent question: Why do state-of-the-art AI systems still fail for the people who need fairness the most? While surfing through different research papers on gender shades and stochastic learning, set out to examine how modern LLMs and VLMs behave across intersectional groups. Very quickly, I uncovered a troubling pattern-models consistently made more errors for darker-skinned individuals, especially Black and Brown women. Even worse, hallucinations disproportionately targeted underrepresented demographics, revealing an overlooked fairness gap in generative AI. I realized that these failures weren’t isolated glitches; they were symptoms of deeper architectural issues in non-deterministic inference and probabilistic sampling. That insight became the foundation for FairVis-SLM.

What it does

FairVis-SLM is a hybrid Small Language Model–Vision Language Model designed to reduce bias amplification at the architectural level. It does this by: Using modular manifolds to enforce structured, interpretable reasoning pathways. Employing deterministic inference and inference stabilization to remove the randomness that often magnifies hallucinations and fairness gaps. Integrating fairness-aware mechanisms to reduce accuracy disparities across race, skin tone, and gender intersections. The result is a more consistent, more transparent, and more equitable AI system.

How we built it

FairVis-SLM was built through a multi-stage research and engineering pipeline: Empirical audit: Firstly I conducted a large-scale analysis of bias amplification in LLMs and VLMs, measuring error rates and hallucination patterns across intersectional demographic groups. Quantification of nondeterminism impacts: We systematically tested stochastic sampling methods and compared them to deterministic decoding to observe fairness distortions caused by inference randomness. Model design: We combined: Modular manifolds for structured reasoning Deterministic inference + stabilization to control variability Fairness-aware constraints integrated directly into training and inference to create the FairVis-SLM architecture. Prototype implementation: Initially we would build an SLM-VLM hybrid that operates on predictable, stable inference paths, enabling more reliable behavior across demographic groups.

Challenges we ran into

Quantifying intersectional bias required building custom evaluation pipelines because standard benchmarks lacked demographic coverage. Measuring hallucination disparities across groups was difficult, as hallucinations are inherently unbounded. Controlling nondeterminism in generative models pushed us to rethink decoding, sampling, and model architecture from the ground up. Integrating modular manifolds into a hybrid LLM–VLM system required new structural designs not present in existing frameworks.

Accomplishments that we're proud of

We uncovered clear causal links between stochastic sampling and fairness degradation-something rarely quantified explicitly. We built one of the first SLM–VLM hybrids with deterministic inference specifically optimized for equitable behavior. We demonstrated that scaling alone does not fix fairness-a crucial insight for AGI safety discourse. We created the foundation for FairVis-SLM, showing meaningful reductions in variability and bias amplification.

What we learned

Fairness cannot be achieved through data scaling alone; architectural changes are essential. Nondeterminism in inference is not just a reliability issue;it is a fairness amplifier. Successful bias mitigation requires interventions in training data, reasoning modules, and inference mechanics simultaneously. Structured reasoning via modular manifolds can constrain unwanted behavior and reduce hallucinations. Deterministic models produce more stable and equitable outputs across demographic groups.

What's next for Fairvis-SLM

Full model training: Scaling FairVis-SLM to a production-grade SLM–VLM architecture. Benchmark creation: Designing a public intersectional fairness benchmark for generative models. Real-world deployment: Integrating FairVis-SLM into safety-critical applications that demand consistent and equitable model behavior. Open-source toolkit: Releasing our inference stabilization techniques for researchers and practitioners. Expansion to multimodal safety: Extending fairness constraints across audio, video, and multimodal reasoning pathways.

Built With

• huggingfaceparameters
• python
• pytorch
• qlora