AF: Large: Collaborative Research: Nonconvex Methods and Models for Learning: Towards Algorithms with Provable and Interpretable Guarantees

AF：大型：协作研究：非凸学习方法和模型：走向具有可证明和可解释保证的算法

• 批准号: 1704656
• 负责人: Rong Ge
• 金额: $ 50万
• 依托单位: Duke University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2017
• 资助国家: 美国
• 起止时间: 2017-06-01 至 2022-05-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1704656&HistoricalAwards=false
• 关键词: AF; Large; Collaborative; Research; Nonconvex

Artificial Intelligence along with Machine Learning are perhaps the most dominant research themes of our times - with far reaching implications for society and our current life style. While the possibilities are many, there are also doubts about how far these methods will go - and what new theoretical foundations may be required to take them to the next level overcoming possible hurdles. Recently, machine learning has undergone a paradigm shift with increasing reliance on stochastic optimization to train highly non-convex models -- including but not limited to deep nets. Theoretical understanding has lagged behind, primarily because most problems in question are provably intractable on worst-case instances. Furthermore, traditional machine learning theory is mostly concerned with classification, whereas much practical success is driven by unsupervised learning and representation learning. Most past theory of representation learning was focused on simple models such as k-means clustering and PCA, whereas practical work uses vastly more complicated models like autoencoders, restricted Boltzmann machines and deep generative models. The proposal presents an ambitious agenda for extending theory to embrace and support these practical trends, with hope of influencing practice. Theoretical foundations will be provided for the next generation of machine learning methods and optimization algorithms. The project may end up having significant impact on practical machine learning, and even cause a cultural change in the field -- theory as well as practice -- with long-term ramifications. Given the ubiquity as well as economic and scientific implications of machine learning today, such impact will extend into other disciplines, especially in (ongoing) collaborations with researchers in neuroscience. The project will train a new generation of machine learning researchers, through an active teaching and mentoring plan at all levels, from undergrad to postdoc. This new generation will be at ease combining cutting edge theory and applications. There is a pressing need for such people today, and the senior PIs played a role in training/mentoring several existing ones. Technical contributions will include new theoretical models of knowledge representation and semantics, and also frameworks for proving convergence of nonconvex optimization routines. Theory will be developed to explain and exploit the interplay between representation learning and supervised learning that has proved so empirically successful in deep learning, and seems to underlie new learning paradigms such as domain adaptation, transfer learning, and interactive learning. Attempts will be made to replace neural models with models with more "interpretable" attributes and performance curves. All PIs have a track record of combining theory with practice. They are also devoted to a heterodox research approach, borrowing from all the past phases of machine learning: interpretable representations from the earlier phases (which relied on logical representations, or probabilistic models), provable guarantees from the middle phase (convex optimization, kernels etc.), and an embrace of nonconvex methods from the latest deep net phase. Such eclecticism is uncommon in machine learning, and may give rise to new paradigms and new kinds of science.

人工智能和机器学习也许是我们这个时代最主要的研究主题——对社会和我们当前的生活方式具有深远的影响。虽然可能性有很多，但人们也怀疑这些方法能走多远，以及可能需要哪些新的理论基础才能将它们提升到一个新的水平，克服可能的障碍。最近，机器学习经历了范式转变，越来越依赖随机优化来训练高度非凸模型——包括但不限于深度网络。理论理解已经落后，主要是因为大多数问题在最坏的情况下都被证明是棘手的。此外，传统的机器学习理论主要关注分类，而许多实际成功是由无监督学习和表示学习驱动的。过去的大多数表示学习理论都集中在简单的模型上，例如 k 均值聚类和 PCA，而实际工作则使用更加复杂的模型，例如自动编码器、受限玻尔兹曼机和深度生成模型。该提案提出了一个雄心勃勃的议程，旨在扩展理论以拥抱和支持这些实际趋势，并希望影响实践。将为下一代机器学习方法和优化算法提供理论基础。该项目最终可能会对实际机器学习产生重大影响，甚至引起该领域的文化变革（理论和实践），并产生长期影响。鉴于当今机器学习的普遍性以及经济和科学影响，这种影响将扩展到其他学科，特别是与神经科学研究人员（正在进行的）合作。该项目将通过从本科生到博士后各个级别的积极教学和指导计划，培训新一代机器学习研究人员。新一代将轻松结合尖端理论和应用。如今，迫切需要这样的人才，高级 PI 在培训/指导现有的几名 PI 方面发挥了作用。技术贡献将包括知识表示和语义的新理论模型，以及证明非凸优化例程收敛性的框架。将发展理论来解释和利用表征学习和监督学习之间的相互作用，这种相互作用已被证明在深度学习中非常成功，并且似乎是领域适应、迁移学习和交互式学习等新学习范式的基础。将尝试用具有更多“可解释”属性和性能曲线的模型来替代神经模型。 所有 PI 都有理论与实践相结合的记录。他们还致力于采用非正统的研究方法，借鉴机器学习的所有过去阶段：早期阶段的可解释表示（依赖于逻辑表示或概率模型），中间阶段的可证明保证（凸优化、内核等） .)，并拥抱最新深网阶段的非凸方法。这种折衷主义在机器学习中并不常见，并且可能会催生新的范式和新的科学类型。

项目成果

Understanding Deflation Process in Over-parametrized Tensor Decomposition

了解超参数化张量分解中的紧缩过程

• 发表时间: 2021-06-11
• 作者: Rong Ge;Y. Ren;Xiang Wang;Mo Zhou
• 通讯作者: Mo Zhou

A Local Convergence Theory for Mildly Over-Parameterized Two-Layer Neural Network

轻度超参数两层神经网络的局部收敛理论

• 发表时间: 2021-02-04
• 作者: Mo Zhou;Rong Ge;Chi Jin
• 通讯作者: Chi Jin

Extracting Latent State Representations with Linear Dynamics from Rich Observations

从丰富的观察中提取具有线性动力学的潜在状态表示

• 发表时间: 2020-06-29
• 作者: Abraham Fr;sen;sen;Rong Ge
• 通讯作者: Rong Ge

Optimization landscape of Tucker decomposition

Tucker 分解的优化景观

• DOI: 10.1007/s10107-020-01531-z
• 发表时间: 2020-01
• 期刊: Mathematical Programming
• 影响因子: 2.7
• 作者: Frandsen, Abraham;Ge, Rong
• 通讯作者: Ge, Rong

Beyond Lazy Training for Over-parameterized Tensor Decomposition

超越过度参数化张量分解的惰性训练

• DOI: 10.3389/fpls.2023.1302315
• 发表时间: 2020-10-22
• 期刊: ArXiv
• 作者: Xiang Wang;Chenwei Wu;J. Lee;Tengyu Ma;Rong Ge
• 通讯作者: Rong Ge