Information Theoretic Coding for Deep Neural Networks: Frameworks, Theory, and Algorithms

深度神经网络的信息论编码：框架、理论和算法

• 批准号: RGPIN-2022-03526
• 负责人: Yang, Enhui
• 金额: $ 4.01万
• 依托单位: University of Waterloo
• 依托单位国家: 加拿大
• 项目类别: Discovery Grants Program - Individual
• 财政年份: 2022
• 资助国家: 加拿大
• 起止时间: 2022-01-01 至 2023-12-31
• 项目状态: 已结题
• 来源: https://www.nserc-crsng.gc.ca/ase-oro/Details-Detailles_eng.asp?id=751675
• 关键词: Information; Theoretic; Coding; Deep; Neural

Deep neural networks (DNNs) are increasingly becoming ubiquitous in many applications including computer vision, speech recognition, and natural language processing. With more data collected in our digital society, DNNs will continue to be a major area of growth in engineering, and change how we live, work, and interact with each other and intelligent machines. Before DNNs can be widely deployed in many parts of our ubiquitous communications networks, however, several key challenges of DNNs have to be addressed. For example, take a look at DNNs for image classification. The first challenge lies in what type of raw data will be fed into DNNs. Indeed, in the context of ubiquitous communications networks which include the whole pipeline of data acquisition, data encoding (i.e., compression), data transmission, and data processing/utilization, the raw data fed into each DNN is not "raw"; instead, it is generally encoded/compressed in a lossy manner. How does lossy coding impact a DNN? According to the conventional wisdom, existing lossy codecs designed for human perception generally degrade the classification accuracy of the DNN. In one of our recent works, however, we showed experimentally the opposite---if one can choose intelligently which compressed version of the raw data is fed into the DNN, the classification accuracy of the DNN can actually be improved significantly while reducing dramatically the number of bits for transmission and storage. The question is, of course, how to encode raw data intelligently for DNNs. The second challenge is the vulnerability of DNNs to adversarial examples, maliciously modified inputs with imperceptible perturbation that lead DNNs to produce incorrect outputs. The existence and easy construction of adversarial examples pose significant security risks to DNNs, especially in safety critical applications. The third challenge lies in the huge number of model parameters in DNNs, which can be as high as a few billions. It is the huge number of model parameters that makes DNNs both computationally intensive and memory intensive, hindering the wide deployment of DNNs in resource limited devices. It also makes it difficult and costly (in terms of bandwidth) to transmit and update model parameters in distributed learning. Based on our early success, in this research program, we will investigate the challenges mentioned above systematically by introducing information theoretic ideas such as soft decision quantization into the domain of DNN, proposing new coding frameworks for DNNs, developing their respective theories, and designing new effective algorithms for new forms of compression for both DNNs and human, protecting DNNs against adversarial attacks, or jointly compressing and training DNN models. Our research results will significantly advance the fields of information theory, image coding, deep learning, and computer vision, and have great impacts on the related industries in Canada and beyond.

深度神经网络 (DNN) 在计算机视觉、语音识别和自然语言处理等许多应用中越来越普遍。随着数字社会中收集到的数据越来越多，DNN 将继续成为工程领域增长的主要领域，并改变我们的生活、工作以及彼此之间以及智能机器之间的交互方式。然而，在 DNN 能够广泛部署到我们无处不在的通信网络的许多部分之前，必须解决 DNN 的几个关键挑战。例如，看一下用于图像分类的 DNN。第一个挑战在于将什么类型的原始数据输入 DNN。事实上，在无处不在的通信网络中，包括数据采集、数据编码（即压缩）、数据传输和数据处理/利用的整个管道，输入到每个 DNN 的原始数据并不是“原始的”；而是“原始的”。相反，它通常以有损方式编码/压缩。有损编码如何影响 DNN？根据传统观点，现有的针对人类感知设计的有损编解码器通常会降低 DNN 的分类精度。然而，在我们最近的一项工作中，我们通过实验证明了相反的情况——如果人们能够智能地选择将原始数据的哪个压缩版本输入到 DNN 中，那么 DNN 的分类精度实际上可以显着提高，同时大大降低用于传输和存储的位数。当然，问题是如何为 DNN 智能编码原始数据。 第二个挑战是 DNN 容易受到对抗性示例的影响，恶意修改输入并产生难以察觉的扰动，导致 DNN 产生错误的输出。对抗性示例的存在和简单构建给 DNN 带来了重大的安全风险，尤其是在安全关键应用中。第三个挑战在于 DNN 中的模型参数数量巨大，可能高达数十亿个。大量的模型参数使得 DNN 既是计算密集型的又是内存密集型的，阻碍了 DNN 在资源有限的设备中的广泛部署。它还使得在分布式学习中传输和更新模型参数变得困难且成本高昂（就带宽而言）。 基于我们早期的成功，在本研究项目中，我们将系统地研究上述挑战，将软决策量化等信息论思想引入DNN领域，提出新的DNN编码框架，发展各自的理论，并设计新的DNN编码框架。为 DNN 和人类提供新形式压缩的有效算法，保护 DNN 免受对抗性攻击，或联合压缩和训练 DNN 模型。 我们的研究成果将显着推进信息论、图像编码、深度学习和计算机视觉领域的发展，并对加拿大及其他地区的相关行业产生巨大影响。