CNS Core: Medium: Collaborative Research: Scalable Dissemination and Navigation of Video 360 Content for Personalized Viewing

CNS 核心：媒介：协作研究：视频 360 内容的可扩展传播和导航以实现个性化观看

• 批准号: 1900875
• 负责人: Klara Nahrstedt
• 金额: $ 40万
• 依托单位: University of Illinois at Urbana-Champaign
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-10-01 至 2024-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1900875&HistoricalAwards=false
• 关键词: CNS; Core; Medium; Collaborative; Research

360 video is a form of virtual reality (VR) that allows the viewer to experience media content in an immersive fashion. In contrast to traditional video, 360 video is recorded with a special camera that captures the complete surroundings from almost all directions. Viewers consuming such a video can select the direction they are looking at by using a pointing device on a regular display or through head movement using a head-mounted device. This new format allows a viewer to change their viewing direction when watching the video, e.g., a viewer can watch a sporting event from multiple perspectives on the field. However, creating, storing, and disseminating 360 videos at a large scale over the Internet poses significant challenges. These challenges are the focus of this project, which will develop a new system and framework, called mi360World, to enable smooth delivery of and interaction with 360 video by any user on the Internet. This project, if successful, will significantly improve 360 video delivery, and will enable new and much richer educational, training and entertainment experiences. It will also help train a new class of multimedia systems researchers and practitioners. The mechanisms for delivering a high-quality, personalized 360 video over the Internet to a globally distributed set of users is an unsolved scientific problem that entails the following challenges: 1) Ultra-high Bandwidth; 2) Ultra-low Delay; 3) View Adaptation (to user head movement); 4) Complex video metadata and delivery; 5) Video Quality of Experience (QoE). Traditional video QoE has seen extensive research over the years; however, what contributes to 360 video QoE is much less understood and will require conceiving of and measuring new metrics. The proposed mi360World system incorporates three major research thrusts to address the above challenges: A video creation thrust that enables personalized viewing by generating navigation graphs and cinematographic rules, while maintaining a high QoE and reducing cybersickness. The construction of navigation graphs and inclusion of cinematographic rules represent the main innovations of this project, and are encapsulated in a three-layered metadata representation of the 360 video: a transport layer, a semantic layer, and an interactive story-telling layer. The second thrust focuses on scalable distribution of 360 videos to a global set of diverse viewers, utilizing navigation graphs and cinematographic rules for highly efficient transition-predictive prefetching and caching. The third thrust focuses on QoE and has the goal of devising novel QoE metrics and evaluation methods to assess cybersickness. System architectures and algorithms will be extensively evaluated through simulation, emulation, and benchmarking using testbeds to assess the success of the proposed research.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

360视频是虚拟现实（VR）的一种形式，可让观众以沉浸式的方式体验媒体内容。与传统视频相反，360视频是通过特殊的相机录制的，该视频几乎从所有方向捕获了完整的环境。消费这种视频的观众可以通过在常规显示器上使用指向设备或使用头部安装的设备来选择他们正在寻找的方向。这种新格式使观众可以在观看视频时改变观看方向，例如，观众可以从该领域的多个角度观看体育事件。 但是，在互联网上大规模创建，存储和传播360个视频构成了重大挑战。 这些挑战是该项目的重点，该项目将开发一个称为MI360World的新系统和框架，以使Internet上任何用户的360个视频平稳交付并与360个视频进行交互。 如果成功的话，该项目将大大改善360个视频交付，并将启用新的，更丰富的教育，培训和娱乐体验。 它还将帮助培训新的多媒体系统研究人员和从业人员。通过Internet向全球分布式用户提供高质量的个性化360视频的机制是一个未解决的科学问题，需要以下挑战：1）超高的带宽； 2）超低延迟； 3）查看适应（对用户头运动）； 4）复杂的视频元数据和交付； 5）视频体验质量（QOE）。多年来，传统的视频Qoe进行了广泛的研究。但是，对360个视频QoE的贡献的理解要少得多，需要对新指标进行构想。 拟议的MI360World系统结合了三个主要的研究推力，以应对上述挑战：视频创建推力，可以通过生成导航图和摄影法规来实现个性化观看，同时保持高QOE并降低网络障碍。导航图的构建和摄影规则的包含代表了该项目的主要创新，并以360视频的三层元数据表示封装：传输层，语义层和交互式讲故事层。第二个推力重点是将360个视频的可扩展分布方式扩展到全球一组不同的观众，利用导航图和摄影规则，用于高效的过渡性预测性预取和缓存。第三个推力集中在Qoe上，其目的是设计新颖的QOE指标和评估方法来评估Cyber​​sickness。系统体系结构和算法将通过使用测试床进行模拟，仿真和基准测试来评估拟议研究的成功。该奖项反映了NSF的法定任务，并被认为是通过基金会的知识分子优点和更广泛的影响审查标准来评估的。

项目成果

360BroadView: Viewer Management for Viewport Prediction in 360-Degree Video Live Broadcast

• DOI: 10.1145/3551626.3564939
• 发表时间: 2022-12
• 期刊: Proceedings of the 4th ACM International Conference on Multimedia in Asia
• 作者: Qian Zhou;Zhe Yang;Hongpeng Guo;Beitong Tian;K. Nahrstedt

Navigation Graph for Tiled Media Streaming

• DOI: 10.1145/3343031.3351021
• 发表时间: 2019-10
• 期刊: Proceedings of the 27th ACM International Conference on Multimedia
• 作者: Jounsup Park;K. Nahrstedt

Ultra-Sparse 360-Degree Camera View Synthesis for Immersive Virtual Tourism

用于沉浸式虚拟旅游的超稀疏 360 度摄像机视图合成

• DOI: 10.1109/mipr54900.2022.00057
• 发表时间: 2022
• 期刊: 2022 IEEE 5th International Conference on Multimedia Information Processing and Retrieval (MIPR
• 作者: Zhou, Qian;Nahrstedt, Klara
• 通讯作者: Nahrstedt, Klara

L3BOU: Low Latency, Low Bandwidth, Optimized Super-Resolution Backhaul for 360-Degree Video Streaming

L3BOU：低延迟、低带宽、优化的超分辨率回程，适用于 360 度视频流

• DOI: 10.1109/ism52913.2021.00031
• 发表时间: 2021
• 期刊: IEEE International Symposium on Multimedia (ISM
• 作者: Sarkar, Ayush;Murray, John;Dasari, Mallesham;Zink, Michael;Nahrstedt, Klara
• 通讯作者: Nahrstedt, Klara

360ViewPET: View Based Pose EsTimation for Ultra-Sparse 360-Degree Cameras

360ViewPET：超稀疏 360 度相机的基于视图的姿势估计

• DOI: 10.1109/ism52913.2021.00008
• 发表时间: 2021
• 期刊: IEEE International Symposium on Multimedia
• 作者: Zhou, Qian;Chen, Bo;Yang, Zhe;Guo, Hongpeng;Nahrstedt, Klara
• 通讯作者: Nahrstedt, Klara