Collaborative Research: Scalable Linear Algebra and Neural Network Theory

合作研究：可扩展线性代数和神经网络理论

• 批准号: 2134247
• 负责人: Michael Mahoney
• 金额: $ 70万
• 依托单位: International Computer Science Institute
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-09-01 至 2025-08-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2134247&HistoricalAwards=false
• 关键词: Collaborative; Research; Scalable; Linear; Algebra

These projects will use randomized numerical linear algebra building blocks to develop improved methods in stochastic optimization theory and statistical/machine learning theory. The motivation is that, while machine learning and deep learning methodology has transformed certain applications, such as computer vision and natural language processing, its promised impact on many other areas has yet to be seen. The reason for this is the flip side of why it has been successful where it has. In the applications where it has had the most remarkable successes, people have adopted the following strategy: get large quantities of data; train a neural network model using stochastic first order methods; and implement and apply the model in a user-facing industrial application. There are many well-known limitations with this general approach, ranging from the need for large quantities of data and daunting compute resources to interpretability and robustness issues. These limitations are particularly apparent when using neural networks for problems such as high-performance computing, fluid mechanics/dynamics, temporal supply chain forecasting problems, biotechnology, etc., where interpretability is paramount. This work aims to address central technical issues underlying this approach, namely: while linear algebraic techniques are central to the design and use of modern neural network models, current methodology uses linear algebra in relatively superficial ways. If we have stronger control over the linear algebraic methods, the community will have a more practical theory to guide neural network use in a broad range of applications beyond computer vision and natural language processing. These methods will enable qualitatively more refined scalable implementations and applications of neural network models in a range of scientific and engineering domains. Broader impacts of these projects include mentoring of grant-supported graduate students and postdoctoral researchers.Technically, the work will focus on three general directions: optimization theory, including convex optimization based neural network and going beyond optimization; scalable linear algebra theory, including randomized linear algebra for neural networks, and sparse randomized linear algebra; and statistics and machine learning theory, including implicit regularization, and learning with limited non-iid data. More broadly, the goal is to provide a basis for practical theory that can guide practice, in a manner analogous to how linear algebraic and functional analytic methods underlie practical and useful theory in a broad range of scientific/engineering applications. We expect that such a challenging task is possible since many of the recent developments in machine learning theory and neural network practice have parallels in scientific computing, where there is a long history of what may be called scalable linear algebra for physical/engineering theory. Many of the methods to be developed may be viewed as bridging the interdisciplinary gap between these old ideas and the new challenges we face; and principal investigators have a history of developing interdisciplinary classes, summer schools, workshops related to the topics of the proposed work, and they will continue to do so.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

这些项目将使用随机数值线性代数构建块来开发随机优化理论和统计/机器学习理论的改进方法。 其动机是，虽然机器学习和深度学习方法已经改变了某些应用程序，例如计算机视觉和自然语言处理，但其对许多其他领域的预期影响还有待观察。 其原因是它取得成功的另一面。 在它取得最显着成功的应用中，人们采取了以下策略：获取大量数据；使用随机一阶方法训练神经网络模型；并在面向用户的工业应用中实施和应用该模型。 这种通用方法存在许多众所周知的局限性，从需要大量数据和令人畏惧的计算资源到可解释性和鲁棒性问题。 当使用神经网络解决高性能计算、流体力学/动力学、时间供应链预测问题、生物技术等问题时，这些局限性尤其明显，其中可解释性至关重要。 这项工作旨在解决该方法背后的核心技术问题，即：虽然线性代数技术是现代神经网络模型的设计和使用的核心，但当前的方法以相对肤浅的方式使用线性代数。 如果我们对线性代数方法有更强的控制，社区将有更实用的理论来指导神经网络在计算机视觉和自然语言处理之外的广泛应用中的使用。 这些方法将使神经网络模型在一系列科学和工程领域中实现质量上更精细的可扩展实现和应用。这些项目的更广泛影响包括指导受资助的研究生和博士后研究人员。从技术上讲，这项工作将集中在三个大方向上：优化理论，包括基于凸优化的神经网络和超越优化；可扩展的线性代数理论，包括神经网络的随机线性代数和稀疏随机线性代数；统计和机器学习理论，包括隐式正则化和使用有限的非独立同分布数据进行学习。 更广泛地说，目标是为指导实践的实用理论提供基础，类似于线性代数和泛函分析方法如何在广泛的科学/工程应用中成为实用和有用的理论的基础。我们期望这样一项具有挑战性的任务是可能的，因为机器学习理论和神经网络实践的许多最新发展与科学计算有相似之处，在科学计算中，物理/工程理论的可扩展线性代数有着悠久的历史。 许多待开发的方法可能被视为弥合这些旧思想与我们面临的新挑战之间的跨学科差距；和主要研究人员都有开发跨学科课程、暑期学校、与拟议工作主题相关的研讨会的历史，他们将继续这样做。该奖项反映了 NSF 的法定使命，并通过使用基金会的评估进行评估，被认为值得支持。智力价值和更广泛的影响审查标准。