Examining risks of racial biases in NLP tools for child protective services

Although much literature has established the presence of demographic bias in natural language processing (NLP) models, most work relies on curated bias metrics that may not be reflective of real-world applications. At the same time, practitioners are increasingly using algorithmic tools in high-stakes settings, with particular recent interest in NLP. In this work, we focus on one such setting: child protective services (CPS). CPS workers often write copious free-form text notes about families they are working with, and CPS agencies are actively seeking to deploy NLP models to leverage these data. Given well-established racial bias in this setting, we investigate possible ways deployed NLP is liable to increase racial disparities. We specifically examine word statistics within notes and algorithmic fairness in risk prediction, coreference resolution, and named entity recognition (NER). We document consistent algorithmic unfairness in NER models, possible algorithmic unfairness in coreference resolution models, and little evidence of exacerbated racial bias in risk prediction. While there is existing pronounced criticism of risk prediction, our results expose previously undocumented risks of racial bias in realistic information extraction systems, highlighting potential concerns in deploying them, even though they may appear more benign. Our work serves as a rare realistic examination of NLP algorithmic fairness in a potential deployed setting and a timely investigation of a specific risk associated with deploying NLP in CPS settings.

尽管许多文献已经证实自然语言处理（NLP）模型中存在人口统计学偏差，但大多数研究依赖于精心设计的偏差指标，这些指标可能无法反映现实世界的应用情况。与此同时，从业者在高风险环境中越来越多地使用算法工具，近期对NLP尤其感兴趣。在这项工作中，我们关注这样一个环境：儿童保护服务（CPS）。CPS工作人员经常撰写大量关于他们所服务家庭的自由格式文本记录，并且CPS机构正在积极寻求部署NLP模型以利用这些数据。鉴于在这种环境中已确定存在种族偏差，我们研究已部署的NLP可能会增加种族差异的方式。我们专门检查记录中的词汇统计以及风险预测、共指消解和命名实体识别（NER）中的算法公平性。我们记录了NER模型中持续存在的算法不公平性、共指消解模型中可能存在的算法不公平性，以及在风险预测中种族偏差加剧的证据很少。虽然对风险预测已有明显的批评，但我们的结果揭示了现实信息提取系统中以前未记录的种族偏差风险，凸显了部署这些系统时的潜在担忧，尽管它们可能看起来更无害。我们的工作是对潜在部署环境中NLP算法公平性的一次罕见的现实检验，也是对在CPS环境中部署NLP相关特定风险的一次及时调查。