NSF-BSF: CIF: Small: Self-adapting Code Generation in Rate-distortion Theory, Machine Learning, and Channel Coding

NSF-BSF：CIF：小型：率失真理论、机器学习和信道编码中的自适应代码生成

• 批准号: 1909423
• 负责人: Kenneth Rose
• 金额: $ 49.99万
• 依托单位: University of California-Santa Barbara
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-07-01 至 2023-06-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1909423&HistoricalAwards=false
• 关键词: NSF; BSF; CIF; Small; Self

This project builds on the investigators' early information-theoretic results, establishing a mechanism called 'natural type selection' for source coding, which adapts a randomly generated code and is shown to be asymptotically optimal. Fundamental expansions of this framework will be pursued to develop universally applicable methodologies, and thereby yield contributions to information theory itself alongside powerful learning techniques in other application fields, including wireless communications, content delivery, social media, artificial intelligence, and others. From the educational perspective, the project offers a training opportunity for graduate students to experience, first-hand, an international and interdisciplinary research collaboration, which combines theoretical depth with practical impact. It further offers opportunities for extensive curriculum enrichment, and to produce accomplished researchers and practitioners with capacities and skills that are in high demand.This project will develop novel approaches to learning, which employ universal self-adapting mechanisms for random code generation, designed to asymptotically achieve optimality for unknown source distributions. Research will be pursued, in terms of both theoretical analysis of performance bounds and powerful optimization approaches, along three main thrusts: i) Extension of the natural type selection framework to encompass continuous spaces and sources with memory, leveraging the concept of "parametric type" for continuous alphabets, which would expand applicability to virtually all practical scenarios of interest. ii) Applications in machine learning, where supervised learning (e.g., classification, regression) is reformulated as the rate-distortion problem of seeking the minimal amount of information to be learned from a source such that a desired output at the prescribed fidelity can be read from a random codebook; and, on the unsupervised learning side, where the "information bottleneck" method is reformulated universally in a self-adapting codebook generation setting. Both will leverage the optimization framework of deterministic annealing. iii) Applications in communications where stochastic mechanisms are developed for optimal channel input adaptation, including an important extension to multi-user communications which requires the development of a "distributed natural type selection" framework. This project is a collaborative effort between researchers in the US and Israel, with funding for Israeli researchers provided by the Bi-National Science Foundation (BSF).This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该项目建立在研究人员早期信息论成果的基础上，建立了一种称为“自然类型选择”的源编码机制，该机制采用随机生成的代码，并被证明是渐近最优的。我们将寻求对该框架的根本扩展，以开发普遍适用的方法，从而为信息论本身以及其他应用领域（包括无线通信、内容交付、社交媒体、人工智能等）强大的学习技术做出贡献。从教育角度来看，该项目为研究生提供了亲身体验国际跨学科研究合作的培训机会，将理论深度与实际影响相结合。它还提供了广泛丰富课程的机会，并培养了具有高需求能力和技能的有成就的研究人员和从业者。该项目将开发新颖的学习方法，采用通用的随机代码生成自适应机制，旨在渐近地实现未知源分布的最优。将根据性能界限的理论分析和强大的优化方法，沿着三个主要方向进行研究：i）利用“参数类型”的概念，扩展自然类型选择框架，以涵盖具有记忆的连续空间和源对于连续字母表，这将扩展适用性到几乎所有感兴趣的实际场景。 ii) 机器学习中的应用，其中监督学习（例如分类、回归）被重新表述为速率失真问题，即寻求从源中学习的最小信息量，以便可以读取指定保真度下的所需输出来自随机密码本；并且，在无监督学习方面，“信息瓶颈”方法在自适应码本生成设置中被普遍重新表述。两者都将利用确定性退火的优化框架。 iii）在通信中的应用，其中开发了随机机制以实现最佳信道输入适应，包括对多用户通信的重要扩展，这需要开发“分布式自然类型选择”框架。 该项目是美国和以色列研究人员之间的合作成果，由美国国家科学基金会 (BSF) 为以色列研究人员提供资金。该奖项反映了 NSF 的法定使命，并通过使用该基金会的智力评估进行评估，认为值得支持。优点和更广泛的影响审查标准。

项目成果

A Stochastic Rate-Distortion Approach to Supervised Learning Systems

监督学习系统的随机率失真方法

• DOI: 10.1109/isit54713.2023.10206453
• 发表时间: 2023-06
• 期刊: Proceedings
• 作者: Elshafiy, Ahmed;Namazi, Mahmoud;Rose, Kenneth
• 通讯作者: Rose, Kenneth

On Effective Stochastic Mechanisms for On-The-Fly Codebook Regeneration

动态码本再生的有效随机机制

• DOI: 10.1109/isit44484.2020.9174348
• 发表时间: 2020-06
• 期刊: 2020 IEEE International Symposium on Information Theory (ISIT
• 作者: Elshafiy, Ahmed;Namazi, Mahmoud;Rose, Kenneth
• 通讯作者: Rose, Kenneth

On Stochastic Codebook Generation for Markov Sources

马尔可夫源的随机码本生成

• DOI: 10.1109/dcc55655.2023.00039
• 发表时间: 2023-03
• 期刊: DCC
• 作者: Elshafiy, Ahmed;Rose, Kenneth
• 通讯作者: Rose, Kenneth

Stochastic Codebook Regeneration for Sequential Compression of Continuous Alphabet Sources

连续字母源顺序压缩的随机码本再生

• DOI: 10.1109/isit45174.2021.9517877
• 发表时间: 2021-07
• 期刊: 2021 IEEE International Symposium on Information Theory (ISIT
• 作者: Elshafiy, Ahmed;Namazi, Mahmoud;Zamir, Ram;Rose, Kenneth
• 通讯作者: Rose, Kenneth

On-The-Fly Stochastic Codebook Re-generation for Sources with Memory

具有内存的源的动态随机码本重新生成

• DOI: 10.1109/itw46852.2021.9457666
• 发表时间: 2021-04
• 期刊: 2020 IEEE Information Theory Workshop
• 作者: Elshafiy, Ahmed;Namazi, Mahmoud;Zamir, Ram;Rose, Kenneth
• 通讯作者: Rose, Kenneth