CIF: Small: Information Theory Meets Deep Learning: Universal Probability and Common Information for High-Dimensional Data

CIF：小：信息论遇见深度学习：高维数据的普遍概率和公共信息

• 批准号: 1911238
• 负责人: Young-Han Kim
• 金额: $ 50万
• 依托单位: University of California-San Diego
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-07-15 至 2022-06-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1911238&HistoricalAwards=false
• 关键词: CIF; Small; Information; Theory; Meets

Information theory is a scientific discipline that studies how well we can communicate, compress, and otherwise process various signals and data. Over the past 70 years, the tenets of information theory have driven the digital transformation of our daily lives, with ubiquitous and reliable communication of 0s and 1s over wireless networks as an indispensable operation in diverse public and private enterprises. Beyond such "bits," however, conventional methods in information theory often encounter several limitations when dealing with real-world data of multiple dimensions, large alphabets, and complex spatial dependence. This project aims to develop new mathematical tools and engineering techniques for handling such complicated real-world data by incorporating deep learning into the tenets of information theory. In this synergistic combination, information theory provides performance guarantees and a systematic decomposition of complex problems. Deep learning, in turn, provides working procedures for efficient processing of high-dimensional data.This project explores two concrete research directions in which the combination prompts new data science frameworks. The first direction builds on the notion of universal probability, which is a close proxy to the unknown distribution of the data. Although conventional algorithms based on universal probability converge rather slowly for high-dimensional, large alphabet, long-range dependent data, the proposed framework leverages deep neural networks to efficiently learn the universal probability distribution across multiple data contexts by aggregating them smoothly. The resulting combination of information theory and deep learning provides a new paradigm to address general data science problems in a principled yet pragmatic manner, with theoretical performance guarantees and practical performance improvements. The second direction is inspired by the notion of common information in information theory, and develops a new neural network model for statistical inference among multiple high-dimensional data sets (for example, drawing a picture automatically that describes a given text the best). This model and associated training algorithms can be applied to various data processing tasks such as joint and conditional generation as well as supervised or semisupervised learning of high-dimensional output data.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

信息论是一门科学学科，研究我们如何更好地通信、压缩和处理各种信号和数据。在过去的 70 年里，信息论的原理推动了我们日常生活的数字化转型，通过无线网络进行无处不在且可靠的 0 和 1 通信成为各种公共和私营企业不可或缺的操作。然而，除了这些“位”之外，信息论中的传统方法在处理多维、大字母表和复杂空间依赖性的现实世界数据时常常遇到一些限制。该项目旨在通过将深度学习融入信息论的原则，开发新的数学工具和工程技术来处理如此复杂的现实世界数据。在这种协同组合中，信息论提供了性能保证和复杂问题的系统分解。反过来，深度学习为高效处理高维数据提供了工作程序。该项目探索了两个具体的研究方向，两者的结合催生了新的数据科学框架。第一个方向建立在普遍概率的概念之上，它是数据未知分布的近似代理。尽管基于通用概率的传统算法对于高维、大字母表、长范围相关数据收敛得相当慢，但所提出的框架利用深度神经网络通过平滑聚合来有效地学习跨多个数据上下文的通用概率分布。由此产生的信息论和深度学习的结合提供了一种新的范式，以原则性但务实的方式解决一般数据科学问题，具有理论性能保证和实际性能改进。第二个方向受到信息论中公共信息概念的启发，开发了一种新的神经网络模型，用于多个高维数据集之间的统计推断（例如，自动绘制最能描述给定文本的图片）。该模型和相关的训练算法可应用于各种数据处理任务，例如联合和条件生成以及高维输出数据的监督或半监督学习。该奖项反映了 NSF 的法定使命，并通过使用评估结果被认为值得支持。基金会的智力价值和更广泛的影响审查标准。