CIF: Small: Deep Learning for Information Theory- Tackling Algorithm Deficit

CIF：小型：信息论深度学习 - 解决算法缺陷

• 批准号: 1908003
• 负责人: Sreeram Kannan
• 金额: $ 45.84万
• 依托单位: University of Washington
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-07-01 至 2022-06-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1908003&HistoricalAwards=false
• 关键词: CIF; Small; Deep; Learning; Information

Information theory provides the mathematical underpinning on how to optimally communicate, compress and process information. The underlying algorithms, including codes for communication, have been invented using human ingenuity and proved to be optimal using deep mathematical reasoning. The rise of deep-learning methods, a branch of machine learning, presents an opportunity to revisit the paradigm for discovering new algorithms. This project studies how to utilize deep-learning for accelerating algorithm discovery in long-standing information-theoretic problems. On a broader scale, this project will promote a stronger interface between deep-learning and information theory, benefiting both research communities. The outcomes of this project will also help create a theory explaining the gains derived by deep-learning algorithms in many application domains, thus addressing an important scientific gap in our understanding of machine-learning.This project studies three problems at the interface of deep-learning and information theory. (1) Deep-Learning based Code Design: This thrust studies the problem of code design, where the code is used to tackle the noise in the communication medium. This involves first replicating the previous successes in code design using this new paradigm, as well as inventing novel codes in unsolved problems -- requiring novel network architectures that can have application beyond codes. (2) Statistical Property Testing with Deep-Learning: How deep-learning can help in information estimation (such as mutual-information estimation) and statistical property testing (such as independence testing) will be studied. Solutions to high-dimensional property testing will require novel p-value guarantees, which will be explored. (3) Information-theoretic Underpinnings of Deep-Learning: The fundamental information theoretic principles underlying deep-learning, such as the sample complexity and optimal training algorithms for recurrent neural networks will be studied.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

信息理论提供了有关如何最佳沟通，压缩和过程信息的数学基础。基本算法（包括通信代码）是使用人类创造力发明的，并使用深厚的数学推理被证明是最佳的。深度学习方法的兴起，即机器学习的一个分支，为发现新算法的范式提供了一个机会。该项目研究了如何在长期的信息理论问题中使用深度学习来加速算法。在更广泛的规模上，该项目将促进深度学习与信息理论之间的更强界面，从而使两个研究社区受益。该项目的结果还将有助于建立一个理论，解释了许多应用领域中深入学习算法所获得的收益，从而解决了我们对机器学习的理解时的重要科学差距。该项目研究了深度学习和信息理论的界面上的三个问题。 （1）基于深度学习的代码设计：此推力研究代码设计的问题，该代码设计用于处理通信介质中的噪声。这涉及首先使用这种新范式来复制代码设计的先前成功，并在未解决的问题中发明了新颖的代码 - 需要新的网络体系结构，这些网络体系结构可以超出代码。 （2）具有深入学习的统计属性测试：将研究如何深入学习信息估计（例如相互信息估计）和统计属性测试（例如独立性测试）。高维财产测试的解决方案将需要新颖的P值保证，这将探讨。 （3）深入学习的信息理论基础：将研究深入学习的基本信息理论原理，例如，将研究复发性神经网络的样本复杂性和最佳培训算法。该奖项反映了NSF的法定任务，并通过使用基金会的智能效果和广泛的评估来进行评估，并通过评估值得评估。

项目成果

Deepcode: Feedback Codes via Deep Learning

• DOI: 10.1109/jsait.2020.2986752
• 发表时间: 2018-07
• 期刊: IEEE Journal on Selected Areas in Information Theory
• 作者: Hyeji Kim;Yihan Jiang;Sreeram Kannan;Sewoong Oh;P. Viswanath

Turbo Autoencoder: Deep learning based channel codes for point-to-point communication channels

• 发表时间: 2019-11
• 期刊: ArXiv
• 作者: Yihan Jiang;Hyeji Kim;Himanshu Asnani;Sreeram Kannan;Sewoong Oh;P. Viswanath

Learning in Gated Neural Networks

• 发表时间: 2019-06
• 作者: Ashok Vardhan Makkuva;Sewoong Oh;Sreeram Kannan;P. Viswanath

C-MI-GAN : Estimation of Conditional Mutual Information using MinMax formulation

• 发表时间: 2020-05
• 期刊: ArXiv
• 作者: A. Mondal;A. Bhattacharya;Sudipto Mukherjee;Sreeram Kannan;Himanshu Asnani;A. Prathosh