AF: Large: Collaborative Research: Nonconvex Methods and Models for Learning: Toward Algorithms with Provable and Interpretable Guarantees

AF：大型：协作研究：非凸学习方法和模型：具有可证明和可解释保证的算法

• 批准号: 1704860
• 负责人: Sanjeev Arora
• 金额: $ 170万
• 依托单位: Princeton University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2017
• 资助国家: 美国
• 起止时间: 2017-06-01 至 2023-05-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1704860&HistoricalAwards=false
• 关键词: AF; Large; Collaborative; Research; Nonconvex

Artificial Intelligence along with Machine Learning are perhaps the most dominant research themes of our times - with far reaching implications for society and our current life style. While the possibilities are many, there are also doubts about how far these methods will go - and what new theoretical foundations may be required to take them to the next level overcoming possible hurdles. Recently, machine learning has undergone a paradigm shift with increasing reliance on stochastic optimization to train highly non-convex models -- including but not limited to deep nets. Theoretical understanding has lagged behind, primarily because most problems in question are provably intractable on worst-case instances. Furthermore, traditional machine learning theory is mostly concerned with classification, whereas much practical success is driven by unsupervised learning and representation learning. Most past theory of representation learning was focused on simple models such as k-means clustering and PCA, whereas practical work uses vastly more complicated models like autoencoders, restricted Boltzmann machines and deep generative models. The proposal presents an ambitious agenda for extending theory to embrace and support these practical trends, with hope of influencing practice. Theoretical foundations will be provided for the next generation of machine learning methods and optimization algorithms. The project may end up having significant impact on practical machine learning, and even cause a cultural change in the field -- theory as well as practice -- with long-term ramifications. Given the ubiquity as well as economic and scientific implications of machine learning today, such impact will extend into other disciplines, especially in (ongoing) collaborations with researchers in neuroscience. The project will train a new generation of machine learning researchers, through an active teaching and mentoring plan at all levels, from undergrad to postdoc. This new generation will be at ease combining cutting edge theory and applications. There is a pressing need for such people today, and the senior PIs played a role in training/mentoring several existing ones. Technical contributions will include new theoretical models of knowledge representation and semantics, and also frameworks for proving convergence of nonconvex optimization routines. Theory will be developed to explain and exploit the interplay between representation learning and supervised learning that has proved so empirically successful in deep learning, and seems to underlie new learning paradigms such as domain adaptation, transfer learning, and interactive learning. Attempts will be made to replace neural models with models with more "interpretable" attributes and performance curves. All PIs have a track record of combining theory with practice. They are also devoted to a heterodox research approach, borrowing from all the past phases of machine learning: interpretable representations from the earlier phases (which relied on logical representations, or probabilistic models), provable guarantees from the middle phase (convex optimization, kernels etc.), and an embrace of nonconvex methods from the latest deep net phase. Such eclecticism is uncommon in machine learning, and may give rise to new paradigms and new kinds of science.

人工智能以及机器学习也许是我们时代最主要的研究主题，对社会和我们当前的生活方式产生了巨大影响。尽管可能性很多，但对这些方法将走多远也有疑问 - 将它们带到一个新的水平，克服可能的障碍可能需要哪些新的理论基础。最近，随着对随机优化的依赖，机器学习发生了范式转变，以训练高度非凸模型 - 包括但不限于深网。理论上的理解远远落后，主要是因为在最坏情况下，所讨论的大多数问题是可悲的。此外，传统的机器学习理论主要与分类有关，而实际的成功是由无监督的学习和代表学习所驱动的。大多数过去的表示学理论都集中在简单的模型上，例如K-Means聚类和PCA，而实际工作使用了更复杂的模型，例如自动编码器，受限的Boltzmann机器和深层生成模型。该提案提出了一个雄心勃勃的议程，以扩展理论，以拥抱和支持这些实际趋势，并希望能影响实践。将为下一代机器学习方法和优化算法提供理论基础。该项目最终可能会对实用的机器学习产生重大影响，甚至会导致该领域的文化变化 - 理论和实践 - 长期影响。鉴于当今机器学习的普遍性以及经济和科学意义，这种影响将扩展到其他学科，尤其是（正在进行的）与神经科学研究人员的合作。该项目将通过从本科到博士学位的各个层面的积极教学和指导计划来培训新一代的机器学习研究人员。这一新一代将放松地结合前沿理论和应用。今天，对此类人的需求紧迫，高级PI在培训/指导几个现有的PI中发挥了作用。技术贡献将包括知识表示和语义的新理论模型，还包括证明非convex优化程序的融合的框架。将开发理论来解释和利用代表性学习与监督学习之间的相互作用，这些学习在深度学习中是如此成功，并且似乎是新的学习范式的基础，例如领域适应，转移学习和互动学习。将尝试用更“可解释的”属性和性能曲线替换模型的神经模型。 所有PI都有将理论与实践相结合的记录。它们还致力于一种异​​端的研究方法，从机器学习的过去所有阶段借用：从早期阶段（依赖逻辑表示或概率模型）中解释的表示形式，中间阶段的可证明的保证（凸优化，内核等），以及从最新的详细信息阶段对非convex方法的包含。这种折衷主义在机器学习中并不常见，并可能引起新的范式和新型科学。