CCF: CIF: Small: Interactive Learning from Noisy, Heterogeneous Feedback

CCF：CIF：小型：从嘈杂、异构的反馈中进行交互式学习

• 批准号: 1719133
• 负责人: Kamalika Chaudhuri
• 金额: $ 50万
• 依托单位: University of California-San Diego
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2017
• 资助国家: 美国
• 起止时间: 2017-07-15 至 2021-06-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1719133&HistoricalAwards=false
• 关键词: CCF; CIF; Small; Interactive; Learning

The goal of this project is to develop interactive learning frameworks and methods that can learn predictors based on complex, imperfect feedback adaptively solicited in an on-line fashion from human annotators. Such predictors can significantly benefit the practice of machine learning by making it more accessible in domains where annotations are expensive. Currently, beyond a handful of heuristic studies, the only well-understood interactive learning setting is active binary classification, where a single annotator interactively provides labels to a learning algorithm. The main challenge in exploiting richer feedback is that human responses are inherently inconsistent and imperfect. This project will overcome this challenge by assuming that the responses come from unknown probability distributions with some mild yet realistic properties, which will be exploited to provide methods that can learn reliably from complex feedback.Specifically, this project will introduce a general framework for interactive learning from imperfect, complex feedback, and develop methods for three common cases: (1) Active Learning with Abstention Feedback, where annotators can either provide a label or declare I Don't Know (2) Active Learning for Multiclass Classification, where the goal is to learn a classifier for a large number of classes and (3) Active Learning with Feedback from Multiple Annotators, where the goal is to combine feedback from many labelers with varying amounts of expertise subject to a budget. These problems will be approached through two main tools -- adaptive hypothesis testing and surrogate loss minimization. Combining these approaches will lead to principled algorithms for building accurate machine learning predictors with low annotation cost, which in turn, will benefit the practice of machine learning in domains where annotated data is expensive.

该项目的目标是开发交互式学习框架和方法，可以根据以在线方式自适应地从人类注释者处获取的复杂、不完美的反馈来学习预测器。此类预测器可以使机器学习实践在注释昂贵的领域中更容易使用，从而极大地有益于机器学习实践。目前，除了少数启发式研究之外，唯一易于理解的交互式学习设置是主动二元分类，其中单个注释器交互式地为学习算法提供标签。利用更丰富的反馈的主要挑战是人类的反应本质上是不一致和不完美的。该项目将通过假设响应来自具有一些温和但现实的属性的未知概率分布来克服这一挑战，这将被用来提供可以从复杂反馈中可靠学习的方法。具体来说，该项目将引入交互式学习的通用框架来自不完美、复杂的反馈，并为三种常见情况开发方法：（1）带有弃权反馈的主动学习，其中注释者可以提供标签或声明“我不知道”（2）多类分类的主动学习，其目标是学习一个分类器大量的类和（3）利用多个注释者的反馈进行主动学习，其目标是将许多标记者的反馈与预算范围内不同数量的专业知识相结合。这些问题将通过两个主要工具来解决——自适应假设检验和替代损失最小化。结合这些方法将产生以低注释成本构建准确的机器学习预测器的原则性算法，这反过来又将有利于注释数据昂贵的领域中的机器学习实践。

项目成果

Active Learning for Classification With Abstention

主动学习弃权分类

• DOI: 10.1109/jsait.2021.3081433
• 发表时间: 2021-06
• 期刊: IEEE Journal on Selected Areas in Information Theory
• 作者: Shekhar, Shubhanshu;Ghavamzadeh, Mohammad;Javidi, Tara
• 通讯作者: Javidi, Tara

Gaussian process bandits with adaptive discretization

具有自适应离散化的高斯过程老虎机

• 发表时间: 2018-12
• 期刊: Electronic journal of statistics
• 作者: Shekhar, S;Javidi, T
• 通讯作者: Javidi, T

Active Learning for Classification with Abstention

主动学习弃权分类

• DOI: 10.1109/isit44484.2020.9174242
• 发表时间: 2020-06
• 期刊: IEEE International Symposium on Information Theory (ISIT
• 作者: Shekhar, Shubhanshu;Ghavamzadeh, Mohammad;Javidi, Tara
• 通讯作者: Javidi, Tara

Active Learning with Logged Data

使用记录数据进行主动学习

• 发表时间: 2018-02-25
• 期刊: ArXiv
• 作者: Songbai Yan;Kamalika Chaudhuri;T. Javidi
• 通讯作者: T. Javidi

Exploring Connections Between Active Learning and Model Extraction

探索主动学习和模型提取之间的联系

• 发表时间: 2018-11-05
• 作者: Varun Ch;rasekaran;rasekaran;Kamalika Chaudhuri;Irene Giacomelli;S. Jha;Songbai Yan
• 通讯作者: Songbai Yan