RINGS: l-RIM: Learning based Resilient Immersive Media-Compression, Delivery, and Interaction

RINGS：l-RIM：基于学习的弹性沉浸式媒体压缩、交付和交互

• 批准号: 2148382
• 负责人: Shiwen Mao
• 金额: $ 99.33万
• 依托单位: Auburn University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-05-01 至 2025-04-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2148382&HistoricalAwards=false
• 关键词: RINGS; RIM; Learning; based; Resilient

Augmented Reality/Virtual Reality (AR/VR) has been recognized as a transformative service for the Next Generation (NextG) network systems, while wireless supported AR/VR will offer great flexibility and enhanced immersive experience to users and unleash a plethora of new applications. The primary goal of this project is to explore innovative technologies in NextG and Artificial intelligence (AI) to provide a unified media compression, communication, and computing framework to enable resilient wireless AR/VR. The research agenda comprises five thrusts. Thrusts 1 and 2 are focused on learning-based immersive media compression, i.e., developing high-efficiency Light Field and Point Cloud compression solutions, respectively. The next two thrusts explore the fundamental concepts and techniques that facilitate wireless AR/VR transmission and interaction. Thrust 3 shall develop theoretical models and explore the fundamental performance tradeoffs among compression communication, computation, and caching for wireless AR/VR considering several NextG communications and networking technologies. The work in Thrust 4 shall investigate the application of edge intelligence to develop learning-based algorithms for resilient AR/VR transmissions. The design of resilient AR/VR will be guided by the model-based performance analysis developed in Thrust 3. Thrust 5 will integrate the techniques developed in Thrusts 1-4, and validate their performance with simulation studies using open-source datasets and experimental studies using an AR/VR testbed and publicly available wireless and cloud-related platforms.The proposed research has a high potential to achieve great impacts in our research community and society. The wireless community has been making great efforts on developing learning-based solutions for the NextG wireless and mobile communication, networking, sensing, and computing systems. Findings from this project are expected to facilitate breakthroughs in enabling immersive media applications that require huge amounts of data to train complex deep learning models. The PIs are committed to integration of research and education. This project will offer research and educational opportunities for undergraduate and graduate students at the two collaborating institutions. New training certificate courses will be developed and research outcomes will be incorporated into a forthcoming textbook by PI Mao. The several existing outreach programs at the two institutions will be leveraged to broaden participation from underrepresented groups. Research outcomes will be disseminated through scholarly publications, a project website, open-source repositories, as well as workshops and journal special issues to be organized by the PIs. Industry will also be pursued through the NSF IUCRC Center for Big Learning at The University of Missouri, Kansas City.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

增强现实/虚拟现实 (AR/VR) 已被认为是下一代 (NextG) 网络系统的变革性服务，而无线支持的 AR/VR 将为用户提供极大的灵活性和增强的沉浸式体验，并释放大量新应用。该项目的主要目标是探索 NextG 和人工智能 (AI) 的创新技术，以提供统一的媒体压缩、通信和计算框架，以实现弹性无线 AR/VR。研究议程包括五个重点。主旨 1 和 2 侧重于基于学习的沉浸式媒体压缩，即分别开发高效的光场和点云压缩解决方案。接下来的两个重点探讨了促进无线 AR/VR 传输和交互的基本概念和技术。 Thrust 3 应开发理论模型，并考虑多种 NextG 通信和网络技术，探索无线 AR/VR 的压缩通信、计算和缓存之间的基本性能权衡。 Thrust 4 的工作将研究边缘智能的应用，以开发用于弹性 AR/VR 传输的基于学习的算法。弹性 AR/VR 的设计将以 Thrust 3 中开发的基于模型的性能分析为指导。Thrust 5 将集成 Thrusts 1-4 中开发的技术，并通过使用开源数据集和实验研究的模拟研究来验证其性能使用 AR/VR 测试台以​​及公开可用的无线和云相关平台。拟议的研究很有可能在我们的研究社区和社会中产生巨大影响。无线社区一直致力于为 NextG 无线和移动通信、网络、传感和计算系统开发基于学习的解决方案。该项目的研究结果预计将促进沉浸式媒体应用的突破，这些应用需要大量数据来训练复杂的深度学习模型。 PI 致力于研究和教育的整合。该项目将为两个合作机构的本科生和研究生提供研究和教育机会。新的培训证书课程将被开发，研究成果将被纳入皮毛即将出版的教科书中。将利用两个机构现有的几个外展计划来扩大代表性不足群体的参与。研究成果将通过学术出版物、项目网站、开源存储库以及 PI 组织的研讨会和期刊特刊进行传播。工业界还将通过位于堪萨斯城密苏里大学的 NSF IUCRC 大学习中心进行追求。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Multi-stage Locally and Long-range Correlated Feature Fusion for Learned In-loop Filter in VVC

• DOI: 10.1109/vcip56404.2022.10008834
• 发表时间: 2022-12
• 期刊: 2022 IEEE International Conference on Visual Communications and Image Processing (VCIP)
• 作者: B. Kathariya;Zhu Li;Hongtao Wang;G. V. D. Auwera

Location-Dependent Augmented Reality Services in Wireless Edge-Enabled Metaverse Systems

• DOI: 10.1109/ojcoms.2023.3234254
• 发表时间: 2023
• 期刊: IEEE Open Journal of the Communications Society
• 影响因子: 7.9
• 作者: Hongliang Zhang;S. Mao;D. Niyato;Zhu Han

LoneNeuron: A Highly-Effective Feature-Domain Neural Trojan Using Invisible and Polymorphic Watermarks

• DOI: 10.1145/3548606.3560678
• 发表时间: 2022-11
• 期刊: Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security
• 作者: Zeyan Liu;Fengjun Li;Zhu Li;Bo Luo

Multi-Stage Spatial and Frequency Feature Fusion using Transformer in CNN-Based In-Loop Filter for VVC

• DOI: 10.1109/pcs56426.2022.10017998
• 发表时间: 2022-12
• 期刊: 2022 Picture Coding Symposium (PCS)
• 作者: B. Kathariya;Zhu Li;Hongtao Wang;Mohammad Coban

Enabling Mobile Virtual Reality with Open 5G, Fog Computing and Reinforcement Learning

• DOI: 10.1109/mnet.010.2100481
• 发表时间: 2022-11
• 期刊: IEEE Network
• 影响因子: 9.3
• 作者: Yaohua Sun;Jianmin Chen;Zeyu Wang;M. Peng;S. Mao