CAREER: Multi-Query Optimizations for Deep Learning Systems

职业：深度学习系统的多查询优化

• 批准号: 1942724
• 负责人: Arun Kumar
• 金额: $ 55万
• 依托单位: University of California-San Diego
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-07-01 至 2025-06-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1942724&HistoricalAwards=false
• 关键词: CAREER; Multi; Query; Optimizations; Deep

Large-scale data analytics using predictive models called "deep learning" has revolutionized many digital applications, powering modern speech recognition, language translation, Web search, and more. This success of deep learning, primarily at resource-rich technology companies, has led to high interest in adopting deep learning in domain sciences, enterprise companies, healthcare, and even digital humanities. But a major bottleneck to broader adoption is the high resource cost of training deep learning models, which requires a computationally expensive empirical process with a large number of trials. This slow process raises resource costs, wastes energy, and impedes user productivity. This project tackles this problem by devising new techniques to substantially speedup up this process on deep learning systems. It will reduce resource costs and energy needs, and in turn, help democratize deep learning to more application domains. It will lead to a new open source system integrated with existing popular deep learning tools to make it cheaper, faster, and easier to adopt large-scale deep learning. The system will be used by domain scientists and also integrated into industrial products. The research will be disseminated via publications at top conferences and incorporated into new courses on data analytics systems. This project will support graduate, undergraduate, and high school students, including LGBT+ and female students.This project will improve the resource efficiency of scalable deep learning model selection, an empirical process that typically requires training dozens to hundreds of model configurations with varying data representations, neural architectures, and hyper-parameter values. Existing tools like TensorFlow and PyTorch focus on the efficiency of training one model a time, which wastes resources at scale during model selection. Some systems also sacrifice reproducibility, a showstopper for many users. This project resolves these issues by presenting a fresh database systems-inspired view of deep learning that re-imagines its executions as queries. Targeting small cluster settings, it raises the specification of three common deep learning model selection tasks to a declarative level and runs many related model configurations in one go. It proposes a suite of multi-query optimization and view materialization techniques that reduce communication costs and/or avoid computational redundancy, while not sacrificing reproducibility or prediction accuracy. The techniques combine the mathematical properties of stochastic gradient descent and the computational properties of deep learning queries with careful parallel data system design and implementation. Project website: https://adalabucsd.github.io/cerebrosystem/This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

使用称为“深度学习”的预测模型的大规模数据分析彻底改变了许多数字应用程序，为现代语音识别，语言翻译，Web搜索提供了动力。深度学习的成功，主要是在资源丰富的技术公司中，引起了人们对在领域科学，企业公司，医疗保健甚至数字人文方面采用深度学习的浓厚兴趣。但是，更广泛采用的主要瓶颈是培训深度学习模型的高资源成本，这需要经过大量试验的计算昂贵的经验过程。这个缓慢的过程提高了资源成本，浪费能源并阻碍用户生产力。该项目通过设计新技术来大大加速此过程，从而解决此问题。它将减少资源成本和能源需求，从而帮助将深入学习民主化到更多的应用领域。它将导致一个与现有流行的深度学习工具集成的新开源系统，以使其更便宜，更快，更易于采用大规模的深度学习。该系统将被域科学家使用，还将整合到工业产品中。该研究将通过顶级会议的出版物进行传播，并将其纳入数据分析系统的新课程中。该项目将支持毕业生，本科生和高中生，包括LGBT+和女学生。该项目将提高可扩展深度学习模型选择的资源效率，这是一个经验过程，通常需要对数百种模型配置进行数十种模型配置，并具有不同的数据表示，神经体系结构，神经体系结构和超参数值。诸如Tensorflow和Pytorch之类的现有工具专注于一次训练一个模型的效率，这在模型选择过程中会大规模浪费资源。有些系统还牺牲了可重复性，这是许多用户的震撼力。该项目通过介绍了一个新的数据库系统启发的深度学习观点来解决这些问题，从而将其执行为查询。针对小群集设置，它将三个常见的深度学习模型选择任务的规范提高到声明级别，并在一个GO中运行许多相关的模型配置。它提出了一套多Query优化的套件，并查看了降低通信成本和/或避免计算冗余的实现技术，同时又不牺牲可重复性或预测准确性。这些技术结合了随机梯度下降的数学特性以及深度学习查询的计算特性，并仔细的并行数据系统设计和实现。项目网站：https：//adalabucsd.github.io/cerebrosystem/this Award反映了NSF的法定任务，并且使用基金会的知识分子优点和更广泛的影响审查标准，被认为值得通过评估来获得支持。

项目成果

Cerebro: A Layered Data Platform for Scalable Deep Learning

• 发表时间: 2021
• 作者: Arun Kumar;Advitya Gemawat;Kabir;Nagrecha;Yuhao Zhang;Side Li

Towards an Optimized GROUP BY Abstraction for Large-Scale Machine Learning

• DOI: 10.14778/3476249.3476284
• 发表时间: 2021-07
• 期刊: Proc. VLDB Endow.
• 作者: Side Li;Arun Kumar

Cerebro: a data system for optimized deep learning model selection

Cerebro：用于优化深度学习模型选择的数据系统

• DOI: 10.14778/3407790.3407816
• 发表时间: 2020
• 期刊: Proceedings of the VLDB Endowment
• 影响因子: 2.5
• 作者: Nakandala, Supun;Zhang, Yuhao;Kumar, Arun
• 通讯作者: Kumar, Arun

Distributed deep learning on data systems: a comparative analysis of approaches

数据系统的分布式深度学习：方法的比较分析

• DOI: 10.14778/3467861.3467867
• 发表时间: 2021
• 期刊: Proceedings of the VLDB Endowment
• 影响因子: 2.5
• 作者: Zhang, Yuhao;McQuillan, Frank;Jayaram, Nandish;Kak, Nikhil;Khanna, Ekta;Kislal, Orhan;Valdano, Domino;Kumar, Arun
• 通讯作者: Kumar, Arun

Nautilus: An Optimized System for Deep Transfer Learning over Evolving Training Datasets

• DOI: 10.1145/3514221.3517846
• 发表时间: 2022-06
• 期刊: Proceedings of the 2022 International Conference on Management of Data
• 作者: Supun Nakandala;Arun Kumar