RI: Small: Actively Learning From The Crowd

RI：小型：积极向大众学习

基本信息

批准号：
1816986
负责人：
Reinhard Heckel
金额：
$ 47.43万
依托单位：
William Marsh Rice University
依托单位国家：
美国
项目类别：
Standard Grant
财政年份：
2018
资助国家：
美国
起止时间：
2018-09-01 至 2023-08-31
项目状态：
已结题

来源：
https://www.nsf.gov/awardsearch/showAward?AWD_ID=1816986&HistoricalAwards=false
关键词：
RI Small Actively Learning Crowd

项目摘要

How can a large number of questions be answered from few and only partially correct responses? This problem lies at the heart of labeling large collections of unlabeled data, which are common in machine learning and data science. It also lies at the heart of learning about preferences of people and quality of items by carrying out surveys. A popular approach to address this problem is to crowdsource the labeling or learning task by paying a large number of people small amounts of money to answer questions on the internet through a crowdsourcing platform. However, the quality of the workers responses varies significantly due to different abilities of the people and difficulties of the questions. To account for the uncertainty of the responses, each question is assigned to multiple people and their responses are aggregated. However, the assignment process is often agnostic to the peoples abilities and questions difficulties, since both are unknown a priori. This project will develop algorithms that adapt to the people and questions and thereby significantly reduces the number of responses required to enable machine learning algorithms to perform well and surveys to be informative. Besides the research objectives, the researchers will pursue educational objectives by integrating parts of this project into a graduate class, promoting undergraduate research, and fostering exchange across disciplines by running an interdisciplinary machine learning seminar. The core optimization problem in crowdsourcing is to achieve confidence in the final answers at minimal cost, by assigning only few tasks to the people (or workers). Intuitively, that can be accomplished by only posing a question to the workers best qualified to answer that question. This project will develop efficient and practical active schemes for crowdlabeling and crowdsourcing that adaptively choose which question to pose to which worker. For each algorithm, the project will prove corresponding rigorous computational and statistical problem-instance dependent performance guarantees, as opposed to worst-case performance guarantees. The theoretical results will be complemented with practical open-source implementations and experiments on real-world data.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

如何从少数且仅部分正确的答案中回答大量问题？这个问题是对大量未标记数据进行标记的核心，这在机器学习和数据科学中很常见。它也是通过开展调查了解人们的偏好和物品质量的核心。解决这个问题的一种流行方法是众包标记或学习任务，通过众包平台向大量人员支付少量费用来回答互联网上的问题。然而，由于人们的能力不同以及问题的难度不同，工人的回答质量差异很大。为了考虑到答案的不确定性，每个问题都会分配给多个人，并且汇总他们的答案。然而，分配过程往往与人们的能力和问题困难无关，因为两者都是先验未知的。该项目将开发适应人和问题的算法，从而显着减少机器学习算法良好执行和调查提供信息所需的响应数量。除了研究目标之外，研究人员还将通过将该项目的部分内容整合到研究生课程中、促进本科生研究以及通过举办跨学科机器学习研讨会来促进跨学科交流来实现教育目标。众包的核心优化问题是通过只向人们（或工人）分配很少的任务，以最小的成本获得对最终答案的信心。直观地说，只需向最有资格回答该问题的工人提出问题即可实现这一目标。该项目将为众标签和众包开发高效实用的主动方案，自适应地选择向哪个工人提出哪个问题。对于每种算法，该项目将证明相应的严格计算和统计问题实例相关的性能保证，而不是最坏情况的性能保证。理论结果将与实际的开源实现和真实世界数据的实验相补充。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。