Collaborative Research: CNS Core: Medium: Innovating Volumetric Video Streaming with Motion Forecasting, Intelligent Upsampling, and QoE Modeling

合作研究：CNS 核心：中：通过运动预测、智能上采样和 QoE 建模创新体积视频流

• 批准号: 2212297
• 负责人: Aditya Akella
• 金额: $ 40万
• 依托单位: University of Texas at Austin
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-10-01 至 2026-09-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2212297&HistoricalAwards=false
• 关键词: Collaborative; Research; CNS; Core; Medium

Volumetric video is an emerging type of multimedia content. Unlike traditional videos and 360-degree videos that are two-dimensional (2D), every frame in a volumetric video consists of a three-dimensional (3D) scene or object represented by a 3D model such as a point cloud or a polygon mesh. The 3D nature of volumetric video enables viewers to move with six degrees of freedom, leading to a truly immersive viewing experience. However, compared to conventional videos, streaming volumetric videos over the Internet faces unique challenges: a lack of precise motion tracking, extremely high bandwidth consumption, a lack of quality-of-experience (QoE) models, and complex interactions among network, computation, and motion. This project proposes a holistic research agenda addressing the above challenges through four core approaches: multi-dimension, multi-modality motion sensing and prediction; deep-learning-based content upsampling; comprehensive QoE modeling; and resource/motion adaptation. The project aims at demonstrable systems and networking research with a synergy among multimedia systems, networking, wireless sensing, machine learning, computer vision, and graphics. The research team will develop prototypes and evaluate them in real-world settings. Internet video streaming is playing a key role in today's world, especially during the COVID-19 pandemic. As a key enabler of immersive telepresence, volumetric content will become popular in the near future. The techniques developed from this project will significantly reduce network resource usage and improve the user experience for volumetric content delivery over the Internet. This will benefit both producers and consumers of volumetric videos, as well as boost their applications in various domains such as education, telehealth, manufacturing, and entertainment, ultimately leading to a high impact on societies and economies. The project will also offer new education components, and provide a platform for various Broadening Participation in Computing (BPC) activities.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

体积视频是一种多媒体内容的新兴类型。与传统视频和二维视频（2D）不同，体积视频中的每个框架都由三维（3D）场景或对象组成，该场景或对象由3D模型（例如点云或多边形网格）组成。体积视频的3D性质使观众能够以六个自由度移动，从而带来真正的身临其境的观看体验。但是，与传统的视频相比，互联网上流式传输视频面临着独特的挑战：缺乏精确的运动跟踪，极高的带宽消耗，缺乏体验质量（QOE）模型以及网络，计算和运动之间的复杂互动。该项目提出了一个整体研究议程，通过四种核心方法来应对上述挑战：多维，多模式运动感应和预测；基于深度学习的内容提升采样；全面的Qoe建模；和资源/运动改编。该项目旨在通过多媒体系统，网络，无线传感，机器学习，计算机视觉和图形在多媒体系统之间的协同作用来证明系统和网络研究。研究团队将开发原型并在现实世界中进行评估。互联网视频流在当今世界中起着关键作用，尤其是在19日期大流行期间。作为沉浸式触觉的关键推动者，体积内容将在不久的将来流行。该项目开发的技术将大大降低网络资源的使用情况，并通过Internet提高用户体验来交付体积内容。这将使体积视频的生产商和消费者都受益，并在教育，远程医疗，制造业和娱乐等各个领域中提高其应用，最终导致对社会和经济的影响很大。该项目还将提供新的教育组成部分，并为各种扩展计算活动（BPC）活动提供平台。该奖项反映了NSF的法定任务，并使用基金会的知识分子优点和更广泛的影响审查标准，认为值得通过评估来获得支持。