Collaborative Research: Foundations of Deep Learning: Theory, Robustness, and the Brain​

协作研究：深度学习的基础：理论、稳健性和大脑 —

• 批准号: 2134059
• 负责人: Christos Papadimitriou
• 金额: $ 30万
• 依托单位: Columbia University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-12-01 至 2024-11-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2134059&HistoricalAwards=false
• 关键词: Collaborative; Research; Foundations; Deep; Learning

A truly comprehensive theory of machine learning has the potential of informing science and engineering in the same profound way Maxwell’s equations did. It was the development of that theory by Maxwell that truly unleashed the potential of electricity, leading to radio, radars, computers, and the Internet. In an analogy, deep learning (DL) has found over the past decade many applications, so far without a comprehensive theory. An eventual theory of learning that explains why and how deep networks work and what their limitations are may thus enable the development of even more powerful learning approaches – especially if the goal of reconnecting DL to brain research bears fruit. In the long term, the ability to develop and build better intelligent machines will be essential to any technology-based economy. After all, even in its current – still highly imperfect –state, DL is impacting or about to impact just about every aspect of our society and life. The investigators also plan to complement their theoretical research with the educational goal of training a diverse population of young researchers from mathematics, computer science, statistics, electrical engineering, and computational neuroscience in the field of machine learning and of its theoretical underpinnings.The investigators propose to join forces in a multi-pronged and collaborative assault on the profound mysteries of DL, informed by the sum of their experience, expertise, ideas, and insight. The research goals are threefold: to develop a sound foundational/mathematical understanding of DL; in doing so to advance the foundational understanding of learning more generally; and to advance the practice of DL by addressing its above-mentioned weaknesses. Of six foundational thrusts, the first two focus on the standard decomposition of the prediction error in approximation and sample (or estimation) error. Their goal is to extend classical results in approximation theory and theory of learnability to DL. These two are then supported by a research project that is specific to deep learning: analysis of the dynamics of gradient descent in training a network. The fourth theme is about robustness against adversaries and shifts, a powerful test for theories which is also important for practical deployment of learning systems. The fifth thrust is about developing the theory of control through DL, as well as exploring dynamical systems aspects of deep reinforcement learning. The final topic connects research on DL to its origins - and possibly its future: networks of neurons in the brain. The proposed research also promises to advance the foundations of learning theory. Success in this project will result in sharper mathematical techniques for machine learning and comprehensive foundations of machine learning robustness, broadly construed. It will also ultimately enable development of learning algorithms that transcend deep learning and guide the way towards creating more intelligent machines, and shed new light on our own intelligence.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

真正全面的机器学习理论具有以与麦克斯韦方程相同的深刻方式为科学和工程提供信息的潜力。麦克斯韦对该理论的发展真正释放了电力的潜力，导致了无线电、雷达、计算机和电力的发展。打个比方，深度学习（DL）在过去十年中已经有了许多应用，但迄今为止还没有一个全面的理论来解释深度网络为何、如何工作以及它们的局限性。开发更强大的学习方法 –尤其是如果将深度学习与高度大脑研究重新联系起来的目标从长远来看，开发和建造更好的智能机器的能力对于任何基于技术的经济都至关重要，即使在目前仍然不完善。 –状态，深度学习正在影响或即将影响我们社会和生活的几乎各个方面。研究人员还计划通过培训数学、计算机科学、统计学、电气等领域的不同年轻研究人员的教育目标来补充他们的理论研究。机器学习及其领域的工程和计算神经科学研究人员建议结合他们的经验、专业知识、想法和洞察力，联合起来对深度学习的奥秘进行多管齐下的协作攻击。研究目标有三个：建立健全的基础。 /对深度学习的数学理解；这样做是为了更普遍地促进对学习的基本理解；并通过解决深度学习的上述弱点，其中前两个重点是标准。他们的目标是将近似理论和可学习性理论的经典结果扩展到深度学习，然后由一个特定于深度学习的研究项目支持：分析。第四个主题是关于对抗对手和变化的鲁棒性，这是对理论的有力检验，这对于学习系统的实际部署也很重要。第五个主题是关于通过深度学习发展控制理论。还有最后一个主题将深度强化学习的研究与它的起源联系起来——可能还有它的未来：大脑中的神经元网络。这项研究也有望为该项目的成功奠定基础。将为机器学习带来更敏锐的数学技术和机器学习鲁棒性的全面基础，从广义上讲，它还将最终实现超越深度学习的学习算法的开发，并指导创建更智能的机器的道路，并为我们自己带来新的启示。这个奖项体现了通过使用基金会的智力价值和更广泛的影响审查标准进行评估，NSF 的法定使命被认为值得支持。

项目成果

Conformal Sensitivity Analysis for Individual Treatment Effects

个体治疗效果的适形敏感性分析

• DOI: 10.1080/01621459.2022.2102503
• 发表时间: 2021-12-07
• 期刊: Journal of the American Statistical Association
• 影响因子: 3.7
• 作者: Mingzhang Yin;Claudia Shi;Yixin Wang;D. Blei
• 通讯作者: D. Blei

A biologically plausible parser

生物学上合理的解析器

• 发表时间: 2021-12
• 期刊: Transactions of the Association for Computational Linguistics
• 影响因子: 10.9
• 作者: Daniel Mitropolsky; Michael Collins
• 通讯作者: Michael Collins

Conformal Sensitivity Analysis for Individual Treatment Effects

个体治疗效果的适形敏感性分析

• 发表时间: 2022-09
• 期刊: Journal of the American Statistical Association
• 影响因子: 3.7
• 作者: Mingzhang Yin; Claudia Shi
• 通讯作者: Claudia Shi

Self-Attention Networks Can Process Bounded Hierarchical Languages

自注意力网络可以处理有界分层语言

• DOI: 10.18653/v1/2021.acl-long.292
• 发表时间: 2021-05-24
• 期刊: International Journal of Intelligent Systems
• 影响因子: 7
• 作者: Shunyu Yao;Binghui Peng;C. Papadimitriou;Karthik Narasimhan
• 通讯作者: Karthik Narasimhan