RI: Small: Taming Massive Pre-trained Models under Label Scarcity via an Optimization Lens

RI：小型：通过优化镜头在标签稀缺的情况下驯服大量预训练模型

• 批准号: 2226152
• 负责人: Tuo Zhao
• 金额: $ 53.99万
• 依托单位: Georgia Tech Research Corporation
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-09-01 至 2025-08-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2226152&HistoricalAwards=false
• 关键词: RI; Small; Taming; Massive; Pre

Deep transfer learning (DTL) has made significant progress in many real-world applications such as image and speech recognition. Training deep learning models in these applications often requires large amounts of labeled data, (e.g., images with annotated objects). Labelling these data by human labor, however, can be very expensive and time-consuming, which significantly limits the broader adoption of deep learning. Such an issue is more pronounced in certain domains (e.g. biomedical domain), where labeled data are scarce. To address the concern of label scarcity, researchers have resorted to deep transfer learning, where a massive deep learning model is first pre-trained only using unlabeled data and then adapted to the downstream task of our interests with only limited labelled data. Due to the gap between the enormous sizes of the pre-trained models and the limited labeled data, however, such a deep transfer learning approach is prone to overfitting and fail to generalize well on the unseen data, especially when there are noisy labels. Moreover, the enormous model sizes make practical deployment very difficult when there are constraints on storage/memory usage, inference latency and energy consumption, especially on edge devices. This project aims to develop an efficient computational framework to improve the generalization of deep transfer learning and reduce the model sizes by leveraging cutting-edge optimization and machine learning techniques.Specifically, this project aims to develop: (I) new adversarial regularization methods, which can regularize the complexity of deep learning models and prevent overfitting of the training data, (II) new self-training methods robust to noisy labels in the training data, and (III) new optimization methods, which can improve the training of compact deep learning models in deep transfer learning. Moreover, we will develop new generalization and approximation theories for understanding the benefits of our proposed methods in transfer learning. The proposed research will also deliver open-source software in the form of easy-to-use libraries, which facilitate researchers and practitioners to apply DTL in related fields.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

深度迁移学习（DTL）在图像和语音识别等许多实际应用中取得了重大进展。在这些应用程序中训练深度学习模型通常需要大量标记数据（例如带有注释对象的图像）。然而，通过人工标记这些数据可能非常昂贵且耗时，这极大地限制了深度学习的更广泛采用。这样的问题在标记数据稀缺的某些领域（例如生物医学领域）更为明显。为了解决标签稀缺的问题，研究人员求助于深度迁移学习，其中首先仅使用未标记数据对大规模深度学习模型进行预训练，然后仅使用有限的标记数据适应我们感兴趣的下游任务。然而，由于预训练模型的巨大规模和有限的标记数据之间的差距，这种深度迁移学习方法很容易过度拟合，并且无法很好地概括未见过的数据，特别是当存在噪声标签时。此外，当存储/内存使用、推理延迟和能耗受到限制时，巨大的模型尺寸使得实际部署变得非常困难，尤其是在边缘设备上。该项目旨在开发一种高效的计算框架，通过利用尖端优化和机器学习技术来提高深度迁移学习的泛化性并减小模型大小。具体而言，该项目旨在开发：（I）新的对抗性正则化方法，该方法可以规范深度学习模型的复杂性并防止训练数据的过度拟合，（II）对训练数据中的噪声标签具有鲁棒性的新自训练方法，以及（III）新的优化方法，可以改善紧凑型深度学习的训练深度迁移学习中的模型。此外，我们将开发新的泛化和近似理论，以理解我们提出的迁移学习方法的好处。拟议的研究还将以易于使用的库的形式提供开源软件，以方便研究人员和从业者在相关领域应用 DTL。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准。

项目成果

LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximation

LoSparse：基于低秩和稀疏近似的大型语言模型的结构化压缩

• DOI: 10.48550/arxiv.2306.11222
• 发表时间: 2023-06-20
• 期刊: ArXiv
• 作者: Yixiao Li;Yifan Yu;Qingru Zhang;Chen Liang;Pengcheng He;Weizhu Chen;Tuo Zhao
• 通讯作者: Tuo Zhao

Machine Learning Force Fields with Data Cost Aware Training

具有数据成本意识培训的机器学习力场

• DOI: 10.48550/arxiv.2306.03109
• 发表时间: 2023-06-05
• 期刊: 2012 15th International Conference on Electrical Machines and Systems (ICEMS)
• 作者: Ale;er W. Bukharin;er;Tianyi Liu;Sheng Wang;Simiao Zuo;Weihao Gao;Wen Yan;Tuo Zhao
• 通讯作者: Tuo Zhao

Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning

用于参数高效微调的自适应预算分配

• DOI: 10.48550/arxiv.2303.10512
• 发表时间: 2024-09-13
• 期刊: ArXiv
• 作者: Qingru Zhang;Minshuo Chen;Ale;er Bukharin;er;Pengcheng He;Yu Cheng;Weizhu Chen;Tuo Zhao
• 通讯作者: Tuo Zhao

Robust Multi-Agent Reinforcement Learning via Adversarial Regularization: Theoretical Foundation and Stable Algorithms

通过对抗正则化实现鲁棒多智能体强化学习：理论基础和稳定算法

• DOI: 10.48550/arxiv.2310.10810
• 发表时间: 2023-10-16
• 期刊: ArXiv
• 作者: Ale;er W. Bukharin;er;Yan Li;Yue Yu;Qingru Zhang;Zhehui Chen;Simiao Zuo;Chao Zhang;Songan Zhang;Tuo Zhao
• 通讯作者: Tuo Zhao