AI bias refers to situations where an AI system's outputs are systematically less accurate or less favourable for certain groups of people, typically as an unintended side effect of how the system was built or trained.
Where Bias Comes From
- Training data: if historical data reflects past human bias, a model trained on it can learn and repeat that same bias.
- Underrepresentation: if certain groups are underrepresented in the training data, the model may simply perform worse for them due to insufficient examples.
- Labeling choices: the specific way humans label training data can introduce subjective bias into the dataset.
- Proxy variables: a model can indirectly learn to discriminate through variables that correlate with a protected characteristic, even if that characteristic itself was never directly included.
A Well-Documented Example
Facial recognition systems have, in independent studies, shown notably lower accuracy rates for certain demographic groups compared to others, largely traced back to imbalances in the datasets used to train early versions of these systems, prompting significant changes in how such systems are now evaluated before deployment.
Addressing AI bias requires deliberate effort throughout the entire development process, including auditing training data for representation gaps, testing model performance across different subgroups, and maintaining ongoing monitoring after a system is deployed, since bias can resurface even after initial fixes.