RI: Small: Taming Massive Pre-trained Models under Label Scarcity via an Optimization Lens

RI：小型：通过优化镜头在标签稀缺的情况下驯服大量预训练模型

• 批准号: 2226152
• 负责人: Tuo Zhao
• 金额: $ 53.99万
• 依托单位: Georgia Tech Research Corporation
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-09-01 至 2025-08-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2226152&HistoricalAwards=false
• 关键词: RI; Small; Taming; Massive; Pre; RI; Small; Taming; Massive; Pre

Deep transfer learning (DTL) has made significant progress in many real-world applications such as image and speech recognition. Training deep learning models in these applications often requires large amounts of labeled data, (e.g., images with annotated objects). Labelling these data by human labor, however, can be very expensive and time-consuming, which significantly limits the broader adoption of deep learning. Such an issue is more pronounced in certain domains (e.g. biomedical domain), where labeled data are scarce. To address the concern of label scarcity, researchers have resorted to deep transfer learning, where a massive deep learning model is first pre-trained only using unlabeled data and then adapted to the downstream task of our interests with only limited labelled data. Due to the gap between the enormous sizes of the pre-trained models and the limited labeled data, however, such a deep transfer learning approach is prone to overfitting and fail to generalize well on the unseen data, especially when there are noisy labels. Moreover, the enormous model sizes make practical deployment very difficult when there are constraints on storage/memory usage, inference latency and energy consumption, especially on edge devices. This project aims to develop an efficient computational framework to improve the generalization of deep transfer learning and reduce the model sizes by leveraging cutting-edge optimization and machine learning techniques.Specifically, this project aims to develop: (I) new adversarial regularization methods, which can regularize the complexity of deep learning models and prevent overfitting of the training data, (II) new self-training methods robust to noisy labels in the training data, and (III) new optimization methods, which can improve the training of compact deep learning models in deep transfer learning. Moreover, we will develop new generalization and approximation theories for understanding the benefits of our proposed methods in transfer learning. The proposed research will also deliver open-source software in the form of easy-to-use libraries, which facilitate researchers and practitioners to apply DTL in related fields.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

深度转移学习（DTL）在许多现实世界应用（例如图像和语音识别）中取得了重大进展。这些应用程序中的培训深度学习模型通常需要大量的标记数据（例如带有带注释的对象的图像）。但是，通过人工劳动将这些数据标记可能非常昂贵且耗时，这显着限制了更广泛的深度学习。这样的问题在某些领域（例如生物医学领域）中更为明显，其中标记的数据很少。为了解决标签稀缺性的关注，研究人员已诉诸深度转移学习，其中首先仅使用未标记的数据对大规模的深度学习模型进行预培训，然后适应只有有限的标记数据的利益的下游任务。但是，由于预先训练的模型的巨大尺寸与有限的标记数据之间的差距，因此，这种深厚的传输学习方法容易过度拟合，并且无法很好地概括看不见的数据，尤其是在有嘈杂的标签时。此外，当在存储/内存使用情况，推理潜伏期和能耗（尤其是在边缘设备上）有限制时，巨大的模型大小使实际部署非常困难。该项目旨在开发一个有效的计算框架，以通过利用尖端优化和机器学习技术来改善深度转移学习的概括，并减少模型大小。特别是，该项目旨在开发：（i新的对抗正则化方法，可以将深度学习模型的复杂性和训练数据的培训（II）培训（ii）训练（II）训练（II）（ii）训练（II）（ii）的新型（II）（II）（II）（II）（II）（II）（II）（II）（II）（II）（II）（II）（II）（II）（II）（II）（II）（II）（II）（II）（II）（II）（II）（II）优化方法，可以改善深度转移学习中紧凑型深度学习模型的训练。此外，我们将开发新的概括和近似理论，以理解我们提出的方法在转移学习中的好处。拟议的研究还将以易于使用的库的形式提供开源软件，这些软件促进了研究人员和从业人员在相关领域中应用DTL。该奖项反映了NSF的法定任务，并被认为是通过基金会的知识分子优点和更广泛的审查标准通过评估来通过评估来获得支持的。