FAI: Quantifying and Mitigating Disparities in Language Technologies

FAI：量化和减轻语言技术方面的差异

• 批准号: 2040926
• 负责人: Graham Neubig
• 金额: $ 37.5万
• 依托单位: Carnegie-Mellon University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-10-01 至 2023-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2040926&HistoricalAwards=false
• 关键词: FAI; Quantifying; Mitigating; Disparities; Language

Advances in natural language processing (NLP) technology now make it possible to perform many tasks through natural language or over natural language data -- automatic systems can answer questions, perform web search, or command our computers to perform specific tasks. However, ``language'' is not monolithic; people vary in the language they speak, the dialect they use, the relative ease with which they produce language, or the words they choose with which to express themselves. In benchmarking of NLP systems however, this linguistic variety is generally unattested. Most commonly tasks are formulated using canonical American English, designed with little regard for whether systems will work on language of any other variety. In this work we ask a simple question: can we measure the extent to which the diversity of language that we use affects the quality of results that we can expect from language technology systems? This will allow for the development and deployment of fair accuracy measures for a variety of tasks regarding language technology, encouraging advances in the state of the art in these technologies to focus on all, not just a select few.Specifically, this work focuses on four aspects of this overall research question. First, we will develop a general-purpose methodology for quantifying how well particular language technologies work across many varieties of language. Measures over multiple speakers or demographics are combined to benchmarks that can drive progress in development of fair metrics for language systems, tailored to the specific needs of design teams. Second, we will move beyond simple accuracy measures, and directly quantify the effect that the accuracy of systems has on users in terms of relative utility derived from using the system. These measures of utility will be incorporated in our metrics for system success. Third, we focus on the language produced by people from varying demographic groups, predicting system accuracies from demographics. Finally, we will examine novel methods for robust learning of NLP systems across language or dialectal boundaries, and examine the effect that these methods have on increasing accuracy for all users.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

自然语言处理（NLP）技术的进步现在使通过自然语言或自然语言数据执行许多任务成为可能 - 自动系统可以回答问题，执行Web搜索或命令我们的计算机执行特定任务。但是，``语言''并不是整体的。人们在说话的语言，使用的方言，产生语言的相对轻松或选择自己表达自己的单词方面有所不同。然而，在NLP系统的基准测试中，这种语言品种通常没有得到调查。最常见的任务是使用Canonical American English制定的，设计不关心系统是否会在任何其他品种的语言上工作。在这项工作中，我们提出了一个简单的问题：我们可以衡量我们使用的语言多样性的程度会影响我们对语言技术系统所期望的结果的质量吗？这将允许针对各种关于语言技术的任务制定和部署公平的准确度措施，鼓励这些技术的最新进步将重点放在所有人上，而不仅仅是精选的几个。特别是，这项工作集中于这一整体研究问题的四个方面。首先，我们将开发一种通用方法，用于量化特定语言技术在许多语言中的工作方式。多个扬声器或人口统计学的措施与基准相结合，可以推动针对设计团队的特定需求量身定制的语言系统公平指标的进展。其次，我们将超越简单的精度度量，并直接量化系统的准确性对用户的效果，从使用系统中得出的相对效用。这些实用程序的度量将纳入我们的系统成功指标中。第三，我们专注于不同人口群体的人们产生的语言，从而预测了人口统计学的系统精度。最后，我们将研究跨语言或方言边界对NLP系统的强大学习的新颖方法，并研究这些方法对所有用户的准确性提高的效果。该奖项反映了NSF的法定任务，并被认为是值得通过基金会的知识分子优点和更广泛的审查标准来通过评估来通过评估来支持的。

项目成果

Gendered Mental Health Stigma in Masked Language Models

蒙面语言模型中的性别心理健康耻辱

• 发表时间: 2022
• 期刊: Conference on Empirical Methods in Natural Language Processing
• 作者: Inna Wanyin Lin;Lucille Njoo;Anjalie Field;Ashish Sharma;Katharina Reinecke;Tim Althoff;Yulia Tsvetkov
• 通讯作者: Yulia Tsvetkov

Examining risks of racial biases in NLP tools for child protective services

• DOI: 10.1145/3593013.3594094
• 发表时间: 2023-05
• 期刊: Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency
• 作者: Anjalie Field;Amanda Coston;Nupoor Gandhi;A. Chouldechova;Emily Putnam-Hornstein;David Steier;Yulia Tsvetkov

SD-QA: Spoken Dialectal Question Answering for the Real World

SD-QA：现实世界的口语方言问答

• DOI: 10.18653/v1/2021.findings-emnlp.281
• 发表时间: 2021
• 期刊: Findings of the Association for Computational Linguistics: EMNLP 2021
• 作者: Faisal, Fahim;Keshava, Sharlina;Alam, Md Mahfuz;Anastasopoulos, Antonios
• 通讯作者: Anastasopoulos, Antonios

Controlled Text Generation as Continuous Optimization with Multiple Constraints

• 发表时间: 2021-08
• 作者: Sachin Kumar;Eric Malmi;Aliaksei Severyn;Yulia Tsvetkov

From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models

• DOI: 10.48550/arxiv.2305.08283
• 发表时间: 2023-05
• 期刊: ArXiv
• 作者: Shangbin Feng;Chan Young Park;Yuhan Liu;Yulia Tsvetkov