Collaborative Research: Transferable, Hierarchical, Expressive, Optimal, Robust, Interpretable Networks

协作研究：可转移、分层、富有表现力、最优、稳健、可解释的网络

• 批准号: 2031899
• 负责人: Yi Ma
• 金额: $ 100万
• 依托单位: University of California-Berkeley
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-09-01 至 2025-08-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2031899&HistoricalAwards=false
• 关键词: Collaborative; Research; Transferable; Hierarchical; Expressive

Recent advances in deep learning have led to many disruptive technologies: from automatic speech recognition systems, to automated supermarkets, to self-driving cars. However, the complex and large-scale nature of deep networks makes them hard to analyze and, therefore, they are mostly used as black-boxes without formal guarantees on their performance. For example, deep networks provide a self-reported confidence score, but they are frequently inaccurate and uncalibrated, or likely to make large mistakes on rare cases. Moreover, the design of deep networks remains an art and is largely driven by empirical performance on a dataset. As deep learning systems are increasingly employed in our daily lives, it becomes critical to understand if their predictions satisfy certain desired properties. The goal of this NSF-Simons Research Collaboration on the Mathematical and Scientific Foundations of Deep Learning is to develop a mathematical, statistical and computational framework that helps explain the success of current network architectures, understand its pitfalls, and guide the design of novel architectures with guaranteed confidence, robustness, interpretability, optimality, and transferability. This project will train a diverse STEM workforce with data science skills that are essential for the global competitiveness of the US economy by creating new undergraduate and graduate programs in the foundations of data science and organizing a series of collaborative research events, including semester research programs and summer schools on the foundations of deep learning. This project will also impact women and underrepresented minorities by involving undergraduates in the foundations of data science.Deep networks have led to dramatic improvements in the performance of pattern recognition systems. However, the mathematical reasons for this success remain elusive. For instance, it is not clear why deep networks generalize or transfer to new tasks, or why simple optimization strategies can reach a local or global minimum of the associated non-convex optimization problem. Moreover, there is no principled way of designing the architecture of the network so that it satisfies certain desired properties, such as expressivity, transferability, optimality and robustness. This project brings together a multidisciplinary team of mathematicians, statisticians, theoretical computer scientists, and electrical engineers to develop the mathematical and scientific foundations of deep learning. The project is divided in four main thrusts. The analysis thrust will use principles from approximation theory, information theory, statistical inference, and robust control to analyze properties of deep networks such as expressivity, interpretability, confidence, fairness and robustness. The learning thrust will use principles from dynamical systems, non-convex and stochastic optimization, statistical learning theory, adaptive control, and high-dimensional statistics to design and analyze learning algorithms with guaranteed convergence, optimality and generalization properties. The design thrust will use principles from algebra, geometry, topology, graph theory and optimization to design and learn network architectures that capture algebraic, geometric and graph structures in both the data and the task. The transferability thrust will use principles from multiscale analysis and modeling, reinforcement learning, and Markov decision processes to design and study data representations that are suitable for learning from and transferring to multiple tasks.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

深度学习的最新进展导致了许多破坏性技术：从自动语音识别系统到自动化超市，再到自动驾驶汽车。但是，深网的复杂而大规模的性质使它们很难分析，因此，它们主要用作黑盒，而无需正式保证其性能。例如，深层网络提供了自我报告的置信度评分，但是它们经常不准确和未校准，或者很可能在极少数情况下犯了很大的错误。此外，深网的设计仍然是一门艺术，并且主要是由数据集中的经验性能驱动的。随着深度学习系统越来越多地在我们的日常生活中使用，了解它们的预测是否满足某些所需特性变得至关重要。这项NSF-Simons在数学和科学基础上的研究合作的目的是开发一个数学，统计和计算框架，有助于解释当前网络体系结构的成功，了解其陷阱，并以确保自信，鲁棒性，可靠性，最佳性，最佳性和可转移性和可转移性来指导新颖体系结构的设计。该项目将通过在数据科学基础上创建新的本科和研究生计划，并组织一系列的协作研究活动，包括学期研究计划和暑期学校在深度学习的基础上，通过创建新的本科和研究生计划来培训多样化的STEM劳动力，这对美国经济的全球竞争力至关重要。该项目还将通过使本科生参与数据科学的基础来影响妇女和代表性不足的少数群体。深入的网络导致了模式识别系统的表现的巨大改善。但是，这一成功的数学原因仍然难以捉摸。例如，尚不清楚为什么深网络概括或转移到新任务，或者为什么简单的优化策略可以达到相关的非凸优化问题的本地或全局最小值。此外，没有原理设计网络体系结构的方法，以便满足某些所需的属性，例如表达性，可传递性，最佳性和鲁棒性。该项目汇集了数学家，统计学家，理论计算机科学家和电气工程师的跨学科团队，以开发深度学习的数学和科学基础。该项目分为四个主要推力。分析推力将使用近似理论，信息理论，统计推断和鲁棒控制中的原理来分析深层网络的属性，例如表达性，可解释性，信心，公平性和鲁棒性。学习推力将使用动态系统，非凸和随机优化，统计学习理论，自适应控制以及高维统计的原理来设计和分析学习算法，并保证收敛，优化性和概括性。设计推力将使用代数，几何，拓扑，图理论和优化的原理来设计和学习网络体系结构，以捕获数据和任务中的代数，几何和图形结构。可转移性推力将使用多尺度分析和建模，强化学习以及马尔可夫决策过程中的原理来设计和研究数据表示，这些数据表示适合于从学习和转移到多个任务的数据表示。这项奖项反映了NSF的法定任务，并被视为值得通过基金会的知识分子优点和更广泛影响的评估来通过评估来获得支持。

项目成果

Zeroth-Order Methods for Convex-Concave Minmax Problems: Applications to Decision-Dependent Risk Minimization

• 发表时间: 2021-06
• 作者: C. Maheshwari;Chih-Yuan Chiu;Eric V. Mazumdar;S. Sastry;L. Ratliff

Learning Diverse and Discriminative Representations via the Principle of Maximal Coding Rate Reduction

• 发表时间: 2020-06
• 期刊: ArXiv
• 作者: Yaodong Yu;Kwan Ho Ryan Chan;Chong You;Chaobing Song;Yi Ma

Are Larger Pretrained Language Models Uniformly Better? Comparing Performance at the Instance Level

• DOI: 10.18653/v1/2021.findings-acl.334
• 发表时间: 2021-05
• 作者: Ruiqi Zhong;Dhruba Ghosh;D. Klein;J. Steinhardt

Predicting Out-of-Distribution Error with the Projection Norm

• 发表时间: 2022-02
• 期刊: ArXiv
• 作者: Yaodong Yu;Zitong Yang;Alexander Wei;Yi Ma;J. Steinhardt

On the principles of Parsimony and Self-consistency for the emergence of intelligence

• DOI: 10.1631/fitee.2200297
• 发表时间: 2022-07
• 期刊: Frontiers of Information Technology & Electronic Engineering
• 影响因子: 3
• 作者: Y. Ma;Doris Y. Tsao;H. Shum