CAREER: Toward a Comprehensive Generalization Theory for Deep Learning

职业：走向深度学习的综合泛化理论

• 批准号: 2045685
• 负责人: Tengyu Ma
• 金额: $ 55万
• 依托单位: Stanford University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-03-01 至 2026-02-28
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2045685&HistoricalAwards=false
• 关键词: CAREER; Toward; Comprehensive; Generalization; Theory

The advancement of deep learning, the technique of training artificial neural networks to make predictions, has led to recent breakthroughs in many areas of artificial intelligence, such as computer vision, natural language understanding, and robotics. A major challenge in deep learning is ensuring accurate predictions on unseen scenarios. This project plans to tackle this challenge via theoretical analysis and its empirical evaluation. The project aims to contribute to the fundamental understanding of deep learning and inform the practical advancement of deep learning, improving its reliability, efficiency, and risk management in data-hungry and risk-sensitive applications. An education plan is integrated into this project --- the investigator will develop new courses, mentor students, organize workshops, and work with high-school teachers on developing high-school AI courses.The project aims to build a comprehensive generalization theory for deep neural networks, which covers the technical question of implicit regularization effect and the broad concepts of out-of-domain generalization and the estimation of generalization errors. This project has three major components. The first thrust is to characterize the optimizers’ implicit regularization effect for complex models. Leveraging the theoretical insights, the investigator will make implicit regularization more explicit, stronger, and customizable to datasets to improve generalization. The second thrust is to theoretically study the out-of-domain generalization in settings with an increasing level of differences between the training and test environments by a growing level of exploitation of unlabeled data and their properties. Finally, the PI will study estimating the generalization errors, which is crucial for quantifying the risk before deploying machine learning models in risk-sensitive applications such as healthcare.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

深度学习（训练人工神经网络进行预测的技术）的进步，最近在计算机视觉、自然语言理解和机器人技术等许多人工智能领域取得了突破。深度学习的一个主要挑战是确保准确。该项目计划通过理论分析和实证评估来应对这一挑战，旨在促进对深度学习的基本理解，并为深度学习的实际进步提供信息，提高其可靠性、效率和风险管理。在数据匮乏和风险敏感的领域该项目融入了教育计划——研究者将开发新课程、指导学生、组织研讨会，并与高中教师合作开发高中人工智能课程。该项目旨在建立一个全面的泛化理论。用于神经深度网络，涵盖隐式正则化效果的技术问题以及域外泛化和泛化误差估计的广泛概念。第一个重点是表征优化器的隐式正则化。利用复杂模型的影响，研究人员将使隐式正则化更加明确、更强并且可针对数据集进行定制，以提高泛化能力。第二个重点是从理论上研究训练和测试环境之间的差异程度不断增加的情况下的域外泛化。最后，PI 将研究估计泛化误差，这对于在医疗保健等风险敏感应用中部署机器学习模型之前量化风险至关重要。该奖项反映了 NSF 的法定使命和通过使用基金会的智力价值和更广泛的影响审查标准进行评估，该项目被认为值得支持。