FairStress benchmark reveals hidden LLM fairness asymmetries

Hugging Face has released FairStress, a benchmark designed to measure demographic fairness in LLMs by testing decision stability against adversarial prefixes. The study evaluated open-weight and frontier models across 300 scenarios in domains like hiring and healthcare, finding that models frequently switch from the objectively stronger candidate if the persuasive argument favors a minority-coded individual. This 'anti-stereotypical over-correction' was observed even with implied demographic cues, revealing asymmetries that standard static benchmarks miss. The release includes both a full 13-million-case dataset and a streamlined FairStress-Core subset for efficient evaluation.