Bias, fairness and honesty
Models reflect their training data, and a system built on them can treat people unfairly or mislead them. Both can be tested for.
Words on this page (2)
- training
- : Adjusting a model's parameters on data so its predictions improve.
- prompt
- : The input text you give a model, including instructions, examples and data.
Bias and fairness
A model trained on human text picks up human patterns, including stereotypes. Outputs can treat groups differently, for example in resume screening, loan explanations or medical advice. The way to find out is to test on purpose: swap names, genders, ages or locations in otherwise identical inputs and compare the results. Document the limits you find. Keep a person responsible for any decision that changes someone's life, such as hiring, credit or health.
A worked example: a counterfactual test
Take 100 resumes. Make two copies of each, differing only in the applicant's name or pronouns. Run both through the screener. If the scores differ on average, or the pass rates differ by group, you have found a bias that no overall accuracy number would show. Fixes include changing the , removing the field, auditing the examples, or keeping a human decision-maker.
Honesty
Tell users when they are talking to an AI. Show sources and uncertainty. Do not treat a model's confident tone as evidence that it is right, and do not let the interface imply more certainty than the system has.
Reading counterfactual results
Take 100 resume pairs. Count the pairs where copy A scored higher, where copy B did, and ties. Invented result: A higher in 38, B higher in 21, tied in 41. Ties say nothing, so look at the 59 others. With no bias you would expect about 29 or 30 each way.
A gap of 38 against 21 is bigger than chance usually makes. For fair coin flips, a split this lopsided in either direction happens roughly 4 times in 100. That is a warning to investigate, not proof. Run more pairs to be sure.
Overall accuracy hides groups
Say a screener is right on 94 of 100 cases. Split by group. Group A has 80 cases and 78 are correct, or 97.5 percent. Group B has 20 cases and 16 are correct, or 80 percent. The average hides a 17.5 point gap. Report results per group, and remember that 20 cases is a small sample, so the 80 percent could be off by a lot.
What the swap test cannot do
Removing names does not remove bias. A postcode, a school or a hobby can stand in for a protected attribute. Test several signals, not just the name.
Also, fair has more than one definition. Equal pass rates and equal error rates can conflict. Decide with the people affected which one your product needs.
Common mistake
Treating fairness as a final checklist. Decide at the start what unfair treatment would look like for your users, and test for it while you build.
A tool that screens resumes scores identical resumes differently when the name changes from one that sounds male to one that sounds female. Nobody programmed that. The model learned patterns from the past. The team found it only because they ran the same resume with different names and compared. Testing for harm has to be done on purpose.
Check yourself
You want to check a resume screener for bias. What is the most direct test?
Not quite. Counterfactual tests change one attribute at a time, so any difference in the score is caused by it.
Overall accuracy is 94 percent. Why can the screener still be unfair?
Not quite. A good average can hide large gaps between groups, so results must be compared by group.
Who should own a high-stakes decision such as hiring?
Not quite. A model can inform a decision, but a person should stay accountable for outcomes that change lives.
You got 0 of 3 right.
Go deeper
- PaperGender ShadesPaper
Buolamwini and Gebru · A landmark study of bias in deployed systems.
- ArticleAI Risk Management FrameworkArticle
NIST · A widely used process for identifying and managing AI risk.
- Article
Barocas, Hardt and Narayanan · A free online textbook on how fairness is defined and measured.
Spotted a mistake or something unclear? Report it on GitHub.