Bias in BIOS: A case study of semantic representation bias in a high-stakes setting

Maria De-Arteaga; Alexey Romanov; Hanna Wallach; Jennifer Chayes; Christian Borgs; Alexandra Chouldechova; Sahin Geyik; Krishnaram Kenthapadi; Adam Tauman Kalai

doi:10.1145/3287560.3287572

Bias in BIOS: A case study of semantic representation bias in a high-stakes setting

Maria De-Arteaga
, Alexey Romanov
, Hanna Wallach
, Jennifer Chayes
, Christian Borgs
, Alexandra Chouldechova
, Sahin Geyik
, Krishnaram Kenthapadi
, Adam Tauman Kalai

Research output: Book chapter › Conference contribution › peer-review

392 Citations (Scopus)

Abstract

We present a large-scale study of gender bias in occupation classification, a task where the use of machine learning may lead to negative outcomes on peoples' lives. We analyze the potential allocation harms that can result from semantic representation bias. To do so, we study the impact on occupation classification of including explicit gender indicators-such as first names and pronouns-in different semantic representations of online biographies. Additionally, we quantify the bias that remains when these indicators are “scrubbed,” and describe proxy behavior that occurs in the absence of explicit gender indicators. As we demonstrate, differences in true positive rates between genders are correlated with existing gender imbalances in occupations, which may compound these imbalances.

Original language	English
Title of host publication	FAT* 2019 - Proceedings of the 2019 Conference on Fairness, Accountability, and Transparency
Publisher	Association for Computing Machinery, Inc
Pages	120-128
Number of pages	9
ISBN (Electronic)	9781450361255
DOIs	https://doi.org/10.1145/3287560.3287572
Publication status	Published - 29 Jan 2019
Externally published	Yes
Event	2019 ACM Conference on Fairness, Accountability, and Transparency, FAT* 2019 - Atlanta, United States Duration: 29 Jan 2019 → 31 Jan 2019

Publication series

Name	FAT* 2019 - Proceedings of the 2019 Conference on Fairness, Accountability, and Transparency

Conference

Conference	2019 ACM Conference on Fairness, Accountability, and Transparency, FAT* 2019
Country/Territory	United States
City	Atlanta
Period	29/01/19 → 31/01/19

Keywords

Algorithmic fairness
Automated hiring
Compounding injustices
Gender bias
Online recruiting
Supervised learning

Access to Document

10.1145/3287560.3287572

Cite this

De-Arteaga, M., Romanov, A., Wallach, H., Chayes, J., Borgs, C., Chouldechova, A., Geyik, S., Kenthapadi, K., & Kalai, A. T. (2019). Bias in BIOS: A case study of semantic representation bias in a high-stakes setting. In FAT* 2019 - Proceedings of the 2019 Conference on Fairness, Accountability, and Transparency (pp. 120-128). (FAT* 2019 - Proceedings of the 2019 Conference on Fairness, Accountability, and Transparency). Association for Computing Machinery, Inc. https://doi.org/10.1145/3287560.3287572

De-Arteaga, Maria ; Romanov, Alexey ; Wallach, Hanna et al. / Bias in BIOS : A case study of semantic representation bias in a high-stakes setting. FAT* 2019 - Proceedings of the 2019 Conference on Fairness, Accountability, and Transparency. Association for Computing Machinery, Inc, 2019. pp. 120-128 (FAT* 2019 - Proceedings of the 2019 Conference on Fairness, Accountability, and Transparency).

@inproceedings{a66172f8281a47939b89538ad3b05ab3,

title = "Bias in BIOS: A case study of semantic representation bias in a high-stakes setting",

abstract = "We present a large-scale study of gender bias in occupation classification, a task where the use of machine learning may lead to negative outcomes on peoples' lives. We analyze the potential allocation harms that can result from semantic representation bias. To do so, we study the impact on occupation classification of including explicit gender indicators-such as first names and pronouns-in different semantic representations of online biographies. Additionally, we quantify the bias that remains when these indicators are “scrubbed,” and describe proxy behavior that occurs in the absence of explicit gender indicators. As we demonstrate, differences in true positive rates between genders are correlated with existing gender imbalances in occupations, which may compound these imbalances.",

keywords = "Algorithmic fairness, Automated hiring, Compounding injustices, Gender bias, Online recruiting, Supervised learning",

author = "Maria De-Arteaga and Alexey Romanov and Hanna Wallach and Jennifer Chayes and Christian Borgs and Alexandra Chouldechova and Sahin Geyik and Krishnaram Kenthapadi and Kalai, \{Adam Tauman\}",

note = "Publisher Copyright: {\textcopyright} 2019 Association for Computing Machinery.; 2019 ACM Conference on Fairness, Accountability, and Transparency, FAT* 2019 ; Conference date: 29-01-2019 Through 31-01-2019",

year = "2019",

month = jan,

day = "29",

doi = "10.1145/3287560.3287572",

language = "English",

series = "FAT* 2019 - Proceedings of the 2019 Conference on Fairness, Accountability, and Transparency",

publisher = "Association for Computing Machinery, Inc",

pages = "120--128",

booktitle = "FAT* 2019 - Proceedings of the 2019 Conference on Fairness, Accountability, and Transparency",

}

De-Arteaga, M, Romanov, A, Wallach, H, Chayes, J, Borgs, C, Chouldechova, A, Geyik, S, Kenthapadi, K & Kalai, AT 2019, Bias in BIOS: A case study of semantic representation bias in a high-stakes setting. in FAT* 2019 - Proceedings of the 2019 Conference on Fairness, Accountability, and Transparency. FAT* 2019 - Proceedings of the 2019 Conference on Fairness, Accountability, and Transparency, Association for Computing Machinery, Inc, pp. 120-128, 2019 ACM Conference on Fairness, Accountability, and Transparency, FAT* 2019, Atlanta, United States, 29/01/19. https://doi.org/10.1145/3287560.3287572

Bias in BIOS: A case study of semantic representation bias in a high-stakes setting. / De-Arteaga, Maria; Romanov, Alexey; Wallach, Hanna et al.
FAT* 2019 - Proceedings of the 2019 Conference on Fairness, Accountability, and Transparency. Association for Computing Machinery, Inc, 2019. p. 120-128 (FAT* 2019 - Proceedings of the 2019 Conference on Fairness, Accountability, and Transparency).

Research output: Book chapter › Conference contribution › peer-review

TY - GEN

T1 - Bias in BIOS

T2 - 2019 ACM Conference on Fairness, Accountability, and Transparency, FAT* 2019

AU - De-Arteaga, Maria

AU - Romanov, Alexey

AU - Wallach, Hanna

AU - Chayes, Jennifer

AU - Borgs, Christian

AU - Chouldechova, Alexandra

AU - Geyik, Sahin

AU - Kenthapadi, Krishnaram

AU - Kalai, Adam Tauman

PY - 2019/1/29

Y1 - 2019/1/29

N2 - We present a large-scale study of gender bias in occupation classification, a task where the use of machine learning may lead to negative outcomes on peoples' lives. We analyze the potential allocation harms that can result from semantic representation bias. To do so, we study the impact on occupation classification of including explicit gender indicators-such as first names and pronouns-in different semantic representations of online biographies. Additionally, we quantify the bias that remains when these indicators are “scrubbed,” and describe proxy behavior that occurs in the absence of explicit gender indicators. As we demonstrate, differences in true positive rates between genders are correlated with existing gender imbalances in occupations, which may compound these imbalances.

AB - We present a large-scale study of gender bias in occupation classification, a task where the use of machine learning may lead to negative outcomes on peoples' lives. We analyze the potential allocation harms that can result from semantic representation bias. To do so, we study the impact on occupation classification of including explicit gender indicators-such as first names and pronouns-in different semantic representations of online biographies. Additionally, we quantify the bias that remains when these indicators are “scrubbed,” and describe proxy behavior that occurs in the absence of explicit gender indicators. As we demonstrate, differences in true positive rates between genders are correlated with existing gender imbalances in occupations, which may compound these imbalances.

KW - Algorithmic fairness

KW - Automated hiring

KW - Compounding injustices

KW - Gender bias

KW - Online recruiting

KW - Supervised learning

UR - https://www.scopus.com/pages/publications/85061785505

U2 - 10.1145/3287560.3287572

DO - 10.1145/3287560.3287572

M3 - Conference contribution

AN - SCOPUS:85061785505

T3 - FAT* 2019 - Proceedings of the 2019 Conference on Fairness, Accountability, and Transparency

SP - 120

EP - 128

BT - FAT* 2019 - Proceedings of the 2019 Conference on Fairness, Accountability, and Transparency

PB - Association for Computing Machinery, Inc

Y2 - 29 January 2019 through 31 January 2019

ER -

De-Arteaga M, Romanov A, Wallach H, Chayes J, Borgs C, Chouldechova A et al. Bias in BIOS: A case study of semantic representation bias in a high-stakes setting. In FAT* 2019 - Proceedings of the 2019 Conference on Fairness, Accountability, and Transparency. Association for Computing Machinery, Inc. 2019. p. 120-128. (FAT* 2019 - Proceedings of the 2019 Conference on Fairness, Accountability, and Transparency). doi: 10.1145/3287560.3287572

Bias in BIOS: A case study of semantic representation bias in a high-stakes setting

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this