Collaborative Research: Foundations of Deep Learning: Theory, Robustness, and the Brain​

协作研究：深度学习的基础：理论、稳健性和大脑 —

• 批准号: 2134108
• 负责人: Tomaso Poggio
• 金额: $ 60万
• 依托单位: Massachusetts Institute of Technology
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-12-01 至 2024-11-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2134108&HistoricalAwards=false
• 关键词: Collaborative; Research; Foundations; Deep; Learning

A truly comprehensive theory of machine learning has the potential of informing science and engineering in the same profound way Maxwell’s equations did. It was the development of that theory by Maxwell that truly unleashed the potential of electricity, leading to radio, radars, computers, and the Internet. In an analogy, deep learning (DL) has found over the past decade many applications, so far without a comprehensive theory. An eventual theory of learning that explains why and how deep networks work and what their limitations are may thus enable the development of even more powerful learning approaches – especially if the goal of reconnecting DL to brain research bears fruit. In the long term, the ability to develop and build better intelligent machines will be essential to any technology-based economy. After all, even in its current – still highly imperfect – state, DL is impacting or about to impact just about every aspect of our society and life. The investigators also plan to complement their theoretical research with the educational goal of training a diverse population of young researchers from mathematics, computer science, statistics, electrical engineering, and computational neuroscience in the field of machine learning and of its theoretical underpinnings.The investigators propose to join forces in a multi-pronged and collaborative assault on the profound mysteries of DL, informed by the sum of their experience, expertise, ideas, and insight. The research goals are threefold: to develop a sound foundational/mathematical understanding of DL; in doing so to advance the foundational understanding of learning more generally; and to advance the practice of DL by addressing its above-mentioned weaknesses. Of six foundational thrusts, the first two focus on the standard decomposition of the prediction error in approximation and sample (or estimation) error. Their goal is to extend classical results in approximation theory and theory of learnability to DL. These two are then supported by a research project that is specific to deep learning: analysis of the dynamics of gradient descent in training a network. The fourth theme is about robustness against adversaries and shifts, a powerful test for theories which is also important for practical deployment of learning systems. The fifth thrust is about developing the theory of control through DL, as well as exploring dynamical systems aspects of deep reinforcement learning. The final topic connects research on DL to its origins - and possibly its future: networks of neurons in the brain. The proposed research also promises to advance the foundations of learning theory. Success in this project will result in sharper mathematical techniques for machine learning and comprehensive foundations of machine learning robustness, broadly construed. It will also ultimately enable development of learning algorithms that transcend deep learning and guide the way towards creating more intelligent machines, and shed new light on our own intelligence.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

一种真正的机器学习理论具有与麦克斯韦方程相同的方式来告知科学和工程的潜力。正是麦克斯韦（Maxwell）的这一理论的发展确实释放了电力的潜力，导致无线电，雷达，计算机和互联网。在类比中，深度学习（DL）在过去十年中发现了许多应用程序，到目前为止没有全面的理论。一个学习的事件理论解释了为什么深层网络的工作以及它们的局限性可能会导致更强大的学习方法的发展，尤其是在将DL重新连接到大脑研究的目标时。从长远来看，开发和建造更好的智能机器的能力对于任何基于技术的经济都是必不可少的。毕竟，即使在目前的状态（仍然是高度不完美的状态）中，DL也在影响或即将影响我们社会和生活的各个方面。研究人员还计划以教育目标来完成其理论研究，即培训来自数学，计算机科学，统计，电气工程和计算神经科学领域的年轻研究人员的多样性人群，并在机器学习及其理论基础的领域中进行了计算神经科学。调查人员提议联合起来，对DL的深刻谜团进行多管齐全和合作的攻击，这是由于他们的经验，专业知识，思想和见识的总和。是三重的：开发对DL的基础/数学理解；这样做以提高对学习的基础理解；并通过解决其上述弱点来推进DL的实践。在六个基础推力中，前两个重点是近似和样本（或估计）误差中预测误差的标准分解。他们的目标是将学习理论和学习理论扩展到DL。然后，这两个由特定于深度学习的研究项目支持：分析训练网络中梯度下降的动态。第四个主题是针对对手和转移的鲁棒性，这是对理论的有力测试，这对于学习系统的实际部署也很重要。第五个推力是通过DL发展控制理论，并探讨了深度强化学习的动态系统方面。最后一个主题将对DL的研究与其起源联系起来，并可能未来：大脑中神经元网络。拟议的研究还有望推进学习理论的基础。该项目的成功将为机器学习和机器学习鲁棒性的全面基础提供更清晰的数学技术。它还将最终能够开发学习算法，这些算法超越深度学习并指导创建更聪明的机器，并为我们自己的智能提供新的启示。该奖项反映了NSF的法定任务，并被认为是通过基金会的知识分子优点和更广泛的影响审查标准通过评估来获得的支持。

项目成果

STay-ON-the-Ridge: Guaranteed Convergence to Local Minimax Equilibrium in Nonconvex-Nonconcave Games

• DOI: 10.48550/arxiv.2210.09769
• 发表时间: 2022-10
• 作者: C. Daskalakis;Noah Golowich;Stratis Skoulakis;Manolis Zampetakis

The Complexity of Markov Equilibrium in Stochastic Games

随机博弈中马尔可夫均衡的复杂性

• 发表时间: 2023
• 期刊: Proceedings of the 36th Annual Conference on Learning Theory (COLT
• 作者: Constantinos Daskalakis;Noah Golowich;Kaiqing Zhang
• 通讯作者: Kaiqing Zhang

The Complexity of Markov Equilibrium in Stochastic Games.

随机博弈中马尔可夫均衡的复杂性。

• 发表时间: 2023
• 期刊: 36th Annual Conference on Learning Theory (COLT
• 作者: Constantinos Daskalakis;Noah Golowich;Kaiqing Zhang
• 通讯作者: Kaiqing Zhang

Online Learning and Solving Infinite Games with an ERM Oracle.

使用 ERM Oracle 在线学习和解决无限游戏。

• 发表时间: 2023
• 期刊: Proceedings of the 36th Annual Conference on Learning Theory (COLT
• 作者: Angelos Assos;Idan Attias;Yuval Dagan;Constantinos Daskalakis;Maxwell Fishelson
• 通讯作者: Maxwell Fishelson

Sign and Basis Invariant Networks for Spectral Graph Representation Learning

• 发表时间: 2022-02
• 期刊: ArXiv
• 作者: Derek Lim;Joshua Robinson;Lingxiao Zhao;T. Smidt;S. Sra;Haggai Maron;S. Jegelka