Principled approaches to deep learning: generalization under distribution shift and predictive uncertainty

深度学习的原则方法：分布变化和预测不确定性下的泛化

• 批准号: RGPIN-2022-03609
• 负责人: Oberman, Adam
• 金额: $ 1.97万
• 依托单位: McGill University
• 依托单位国家: 加拿大
• 项目类别: Discovery Grants Program - Individual
• 财政年份: 2022
• 资助国家: 加拿大
• 起止时间: 2022-01-01 至 2023-12-31
• 项目状态: 已结题
• 来源: https://www.nserc-crsng.gc.ca/ase-oro/Details-Detailles_eng.asp?id=759084
• 关键词: Principled; approaches; deep; learning; generalization

Deep Learning (DL) addresses problems which were previously not possible using traditional Machine Learning (ML) methods. However, unlike ML, which has a solid theoretical foundation, DL has so far relied engineering practices. In particular, while very successful in computer applications, DL has so far been limited in its applications to real world problems, for example autonomous vehicles. In order to continue to broaden the applications, deep learning requires a theoretical foundation, in the form of error estimates, which provide control over the accuracy of models on new inputs. Thirty years ago, Machine Learning (ML) was in a situation similar to the one currently faced by DL: the methods were far ahead of the theory. In a short time, ML theory was able to solve theoretical problems, allowing ML models to be safely deployed in a wide range of applications. The proposed program will do for DL what was done thirty years ago for ML. Error estimation can be addressed in two settings. The first is generalization error bounds, which apply before the inputs are seen by the model. The second is predictive uncertainty, which applies after the inputs are seen by the model (but before a decision is made). For example, suppose we have a decision problem where we can only act if the probability of error is less than 1%, but our model error is 5%, on average. Predictive uncertainty can tell us, on a case by case basis, on which inputs the model accuracy is high enough to act. For a second example, we train computer vision models using databases of images, but we want to deploy them on real world images, which are statistically different from the training set. We propose to extend data transformation techniques, and methods for measuring dataset differences, in order to better estimate the model error on real world images. This proposal will address the problem of applying deep learning in a broader setting by building a mathematical theory for DL out-of-distribution (OOD) generalization. It will 1.Formulate suitable definitions and assumptions which allow us to state the deep learning out-of- distribution generalization problem in mathematical terms. 2.Prove a theorem estimating (with high probability) the OOD generalization gap, in terms of relevant problem inputs. 3.Determine relevant problem inputs through mathematical modelling. Determining the relevant hypotheses requires applying the scientific method, which, in this case corresponds to computer experiments designed to probe the generalization behaviour of deep neural networks. Stating the definitions and assumptions precisely involves mathematical modelling. Proving the theorem described above involves mathematical analysis.

深度学习 (DL) 解决了以前使用传统机器学习 (ML) 方法无法解决的问题。然而，与具有坚实理论基础的机器学习不同，深度学习迄今为止依赖于工程实践。特别是，虽然深度学习在计算机应用方面非常成功，但迄今为止其应用仅限于解决现实世界问题，例如自动驾驶汽车。为了继续扩大应用范围，深度学习需要以误差估计的形式提供理论基础，以控制新输入的模型的准确性。 30 年前，机器学习 (ML) 面临着与目前 DL 类似的情况：方法远远领先于理论。在很短的时间内，机器学习理论就能够解决理论问题，使机器学习模型能够安全地部署在广泛的应用程序中。拟议的计划将为深度学习带来三十年前为机器学习所做的事情。误差估计可以通过两种设置来解决。第一个是泛化误差界限，它在模型看到输入之前应用。第二个是预测不确定性，它适用于模型看到输入之后（但在做出决策之前）。例如，假设我们有一个决策问题，只有错误概率小于 1% 时我们才能采取行动，但我们的模型错误平均为 5%。预测不确定性可以根据具体情况告诉我们，模型的准确性足够高，可以采取行动。对于第二个例子，我们使用图像数据库训练计算机视觉模型，但我们希望将它们部署在现实世界图像上，这些图像在统计上与训练集不同。我们建议扩展数据转换技术和测量数据集差异的方法，以便更好地估计现实世界图像的模型误差。该提案将通过建立深度学习分布外（OOD）泛化的数学理论来解决在更广泛的环境中应用深度学习的问题。它将 1. 制定合适的定义和假设，使我们能够用数学术语表述深度学习分布外泛化问题。 2.根据相关问题输入，证明一个定理（以高概率）估计 OOD 泛化差距。 3.通过数学建模确定相关问题输入。确定相关假设需要应用科学方法，在这种情况下，对应于旨在探测深度神经网络泛化行为的计算机实验。精确地陈述定义和假设涉及数学建模。证明上述定理涉及数学分析。