CAREER: Coding Theory for Robust Large-Scale Machine Learning

职业：鲁棒大规模机器学习的编码理论

基本信息

批准号：
1844951
负责人：
Dimitrios Papailiopoulos
金额：
$ 50.83万
依托单位：
University of Wisconsin-Madison
依托单位国家：
美国
项目类别：
Continuing Grant
财政年份：
2019
资助国家：
美国
起止时间：
2019-05-01 至 2024-04-30
项目状态：
已结题

来源：
https://www.nsf.gov/awardsearch/showAward?AWD_ID=1844951&HistoricalAwards=false
关键词：
CAREER Coding Theory Robust Large

项目摘要

Coding theory has played a critical role in modern information technology by supporting robustness of information against a backdrop of multifaceted uncertainty. Following recent successes in machine learning, robustness has emerged as a desired principle, but now in the context of large-scale computation. Challenges related to robustness are prevalent when deploying machine learning solutions in real applications and non-curated settings, which are often non-ideal environments. This project aims to address these challenges by developing novel solutions based on coding theory for computation. These solutions offer provable robustness guarantees, can outperform more traditional solutions in practice, and extend to machine learning systems the gains that have transformed communication and storage systems. Existing and new collaborations of the investigator will facilitate industry cooperation and increase the transition to practice for the frameworks and algorithms generated from this project. The research will be strongly coupled with educational developments guided by recent advances in education science, alongside an outreach program within the Wisconsin Institute for Discovery. This project aims to develop novel coding-theoretic solutions and fundamental trade-offs for robust large-scale machine learning. The research program is centered around three thrusts. The first thrust focuses on robustness during distributed optimization in the presence of delays and straggler nodes, where the speed of convergence is affected by nodes in the system that are significantly slower than average. The second thrust focuses on robustness during distributed optimization in the presence of Byzantine nodes and worst-case failures. Recent studies proposed robust aggregation rules to filter out the effect of worst-case or adversarial failures. This project develops coding-theoretic solutions that can be orders of magnitude faster, and give rise to unexplored trade-offs between computation and Byzantine tolerance. The third thrust focuses on adversarial perturbations during prediction that can force state-of-the-art models to consistently mis-classify events/data. The coding-theoretic approach of this project pursues provable defense mechanisms against adversarial attacks through ensembles of models with inherent redundancy and through data augmentation. The proposed theoretical and algorithmic solutions are afforded by an interdisciplinary mix of tools from information and coding theory, distributed optimization, and machine learning.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

编码理论在多方面不确定性的背景下支持信息的鲁棒性，在现代信息技术中发挥了关键作用。随着机器学习最近取得的成功，鲁棒性已成为一项理想的原则，但现在是在大规模计算的背景下。在实际应用程序和非策划设置（通常是非理想环境）中部署机器学习解决方案时，与鲁棒性相关的挑战非常普遍。该项目旨在通过开发基于计算编码理论的新颖解决方案来应对这些挑战。这些解决方案提供了可证明的稳健性保证，在实践中可以超越更传统的解决方案，并将改变通信和存储系统的优势扩展到机器学习系统。研究人员现有的和新的合作将促进行业合作，并促进该项目生成的框架和算法向实践的过渡。该研究将与以教育科学最新进展为指导的教育发展以及威斯康星州发现研究所的外展计划紧密结合。该项目旨在开发新颖的编码理论解决方案和稳健的大规模机器学习的基本权衡。该研究计划围绕三个主旨展开。第一个重点是在存在延迟和落后节点的情况下分布式优化过程中的鲁棒性，其中收敛速度受到系统中明显慢于平均水平的节点的影响。第二个重点是在存在拜占庭节点和最坏情况故障的情况下分布式优化期间的鲁棒性。最近的研究提出了强大的聚合规则来过滤最坏情况或对抗性失败的影响。该项目开发了编码理论解决方案，速度可以提高几个数量级，并在计算和拜占庭容错之间产生未经探索的权衡。第三个重点是预测过程中的对抗性扰动，这些扰动可能迫使最先进的模型持续对事件/数据进行错误分类。该项目的编码理论方法通过具有固有冗余的模型集合和数据增强来追求针对对抗性攻击的可证明的防御机制。所提出的理论和算法解决方案由信息和编码理论、分布式优化和机器学习等跨学科工具组合提供。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值和更广泛的影响进行评估，被认为值得支持审查标准。

项目成果

期刊论文数量（9）

专著数量（0）

科研奖励数量（0）

会议论文数量（0）

专利数量（0）

Optimal Lottery Tickets via SubsetSum: Logarithmic Over-Parameterization is Sufficient

DOI：
发表时间：
2020-06
期刊：
ArXiv
影响因子：
0
作者：
Ankit Pensia;Shashank Rajput;Alliot Nagle;Harit Vishwakarma;Dimitris Papailiopoulos
通讯作者：
Ankit Pensia;Shashank Rajput;Alliot Nagle;Harit Vishwakarma;Dimitris Papailiopoulos

Attack of the Tails: Yes, You Really Can Backdoor Federated Learning

DOI：
发表时间：
2020-07
期刊：
ArXiv
影响因子：
0
作者：
Hongyi Wang;Kartik K. Sreenivasan;Shashank Rajput;Harit Vishwakarma;Saurabh Agarwal;Jy-yong Sohn;
通讯作者：
Hongyi Wang;Kartik K. Sreenivasan;Shashank Rajput;Harit Vishwakarma;Saurabh Agarwal;Jy-yong Sohn;

Bad Global Minima Exist and SGD Can Reach Them

DOI：
发表时间：
2019-05
期刊：
ArXiv
影响因子：
0
作者：
Shengchao Liu;Dimitris Papailiopoulos;D. Achlioptas
通讯作者：
Shengchao Liu;Dimitris Papailiopoulos;D. Achlioptas

DETOX: A Redundancy-based Framework for Faster and More Robust Gradient Aggregation

DOI：
发表时间：
2019-07
期刊：
ArXiv
影响因子：
0
作者：
Shashank Rajput;Hongyi Wang;Zachary B. Charles;Dimitris Papailiopoulos
通讯作者：
Shashank Rajput;Hongyi Wang;Zachary B. Charles;Dimitris Papailiopoulos

Finding Nearly Everything within Random Binary Networks

DOI：
发表时间：
2022
期刊：
影响因子：
0
作者：
Kartik K. Sreenivasan;Shashank Rajput;Jy-yong Sohn;Dimitris Papailiopoulos
通讯作者：
Kartik K. Sreenivasan;Shashank Rajput;Jy-yong Sohn;Dimitris Papailiopoulos