RI: Medium: Foundations of Self-Supervised Learning Through the Lens of Probabilistic Generative Models

RI：媒介：通过概率生成模型的视角进行自我监督学习的基础

基本信息

批准号：
2211907
负责人：
Pradeep Ravikumar
金额：
$ 112.79万
依托单位：
Carnegie-Mellon University
依托单位国家：
美国
项目类别：
Standard Grant
财政年份：
2022
资助国家：
美国
起止时间：
2022-10-01 至 2026-09-30
项目状态：
未结题

来源：
https://www.nsf.gov/awardsearch/showAward?AWD_ID=2211907&HistoricalAwards=false
关键词：
RI Medium Foundations Self Supervised

项目摘要

Supervised learning of modern machine learning models requires very large high-quality labeled datasets. Labeling data requires very expensive human annotations, which is often too expensive for under-resourced end-users of machine learning. Unsupervised learning of machine learning models from unlabeled data has the promise to vastly increase the accessibility and inclusivity of modern machine learning. An emerging paradigm for such unsupervised learning is self-supervised learning (SSL), wherein a machine learning model is trained on tasks for which labels can be automatically generated. This approach is at the core of high-performing language and image machine learning models like BERT and DALL-E. However, despite its promise on many benchmarks across diverse domains, a lot of current methodology for developing SSL methods is opaque and heuristic, and evaluation relies on ad-hoc choices of performance metrics. The goal of this project is to build scientific and mathematical foundations of SSL, and consequently also improve its practice. In some of the earliest work in this area, SSL was used to speed up tasks involving the learning of probabilistic models. Progressively, via a series of approximations for scalability, the outputs of SSL could no longer be rigorously tied to probabilistic model parameters, and the goal shifted to learning features that are "useful" for downstream tasks, that is representation learning. "Useful" however can often be mathematically difficult to pin down, so it is frequently not clear (even empirically, much less theoretically) what these methods learn about the data. At present, designing a well-performing SSL method entails trying many combinations of tasks and model architectures, until a particular one gives good results on the downstream tasks. This has two downsides: (i) it requires a substantial amount of trial-and-error; (ii) on a scientific level, it doesn't yield any understanding of what makes a particular task/architecture suitable, and what the features learned capture about the data distribution. This project will repair the severed tie between probabilistic models and feature learning via self-supervised models by analyzing the aspects of a deep generative model that can be recovered via self-supervised learning. Moreover, through this lens, we propose to understand the relative advantages---both statistical and algorithmic---of self-supervised learning methods over other methods for learning probabilistic models.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

现代机器学习模型的监督学习需要非常大的高质量标记数据集。标记数据需要非常昂贵的人工注释，这对于资源不足的机器学习最终用户来说通常过于昂贵。从未标记数据中对机器学习模型进行无监督学习有望大大提高现代机器学习的可访问性和包容性。这种无监督学习的新兴范例是自监督学习（SSL），其中机器学习模型针对可以自动生成标签的任务进行训练。这种方法是 BERT 和 DALL-E 等高性能语言和图像机器学习模型的核心。然而，尽管它在跨不同领域的许多基准上做出了承诺，但当前开发 SSL 方法的许多方法都是不透明和启发式的，并且评估依赖于性能指标的临时选择。该项目的目标是建立 SSL 的科学和数学基础，从而改进其实践。在该领域的一些最早的工作中，SSL 用于加速涉及概率模型学习的任务。逐渐地，通过一系列可扩展性的近似，SSL 的输出不再严格依赖于概率模型参数，目标转移到学习对下游任务“有用”的特征，即表示学习。然而，“有用”通常在数学上很难确定，因此通常不清楚（即使是经验上的，更不用说理论上的）这些方法从数据中学到了什么。目前，设计一种性能良好的 SSL 方法需要尝试多种任务和模型架构的组合，直到某一特定组合在下游任务上给出良好的结果。这有两个缺点：（i）需要大量的试错； (ii) 在科学层面上，它无法理解什么使特定任务/架构适合，以及学到的特征捕获了数据分布的哪些内容。该项目将通过分析可通过自监督学习恢复的深度生成模型的各个方面，通过自监督模型修复概率模型和特征学习之间断绝的联系。此外，通过这个视角，我们建议了解自监督学习方法相对于其他概率模型学习方法的相对优势（统计和算法）。该奖项反映了 NSF 的法定使命，并被认为值得支持通过使用基金会的智力优点和更广泛的影响审查标准进行评估。