CNS Core: Medium: Collaborative Research: Scalable Dissemination and Navigation of Video 360 Content for Personalized Viewing

CNS 核心：媒介：协作研究：视频 360 内容的可扩展传播和导航以实现个性化观看

• 批准号: 1900875
• 负责人: Klara Nahrstedt
• 金额: $ 40万
• 依托单位: University of Illinois at Urbana-Champaign
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-10-01 至 2024-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1900875&HistoricalAwards=false
• 关键词: CNS; Core; Medium; Collaborative; Research

360 video is a form of virtual reality (VR) that allows the viewer to experience media content in an immersive fashion. In contrast to traditional video, 360 video is recorded with a special camera that captures the complete surroundings from almost all directions. Viewers consuming such a video can select the direction they are looking at by using a pointing device on a regular display or through head movement using a head-mounted device. This new format allows a viewer to change their viewing direction when watching the video, e.g., a viewer can watch a sporting event from multiple perspectives on the field. However, creating, storing, and disseminating 360 videos at a large scale over the Internet poses significant challenges. These challenges are the focus of this project, which will develop a new system and framework, called mi360World, to enable smooth delivery of and interaction with 360 video by any user on the Internet. This project, if successful, will significantly improve 360 video delivery, and will enable new and much richer educational, training and entertainment experiences. It will also help train a new class of multimedia systems researchers and practitioners. The mechanisms for delivering a high-quality, personalized 360 video over the Internet to a globally distributed set of users is an unsolved scientific problem that entails the following challenges: 1) Ultra-high Bandwidth; 2) Ultra-low Delay; 3) View Adaptation (to user head movement); 4) Complex video metadata and delivery; 5) Video Quality of Experience (QoE). Traditional video QoE has seen extensive research over the years; however, what contributes to 360 video QoE is much less understood and will require conceiving of and measuring new metrics. The proposed mi360World system incorporates three major research thrusts to address the above challenges: A video creation thrust that enables personalized viewing by generating navigation graphs and cinematographic rules, while maintaining a high QoE and reducing cybersickness. The construction of navigation graphs and inclusion of cinematographic rules represent the main innovations of this project, and are encapsulated in a three-layered metadata representation of the 360 video: a transport layer, a semantic layer, and an interactive story-telling layer. The second thrust focuses on scalable distribution of 360 videos to a global set of diverse viewers, utilizing navigation graphs and cinematographic rules for highly efficient transition-predictive prefetching and caching. The third thrust focuses on QoE and has the goal of devising novel QoE metrics and evaluation methods to assess cybersickness. System architectures and algorithms will be extensively evaluated through simulation, emulation, and benchmarking using testbeds to assess the success of the proposed research.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

360 度视频是虚拟现实 (VR) 的一种形式，允许观看者以身临其境的方式体验媒体内容。与传统视频相比，360 度视频是用特殊摄像机录制的，可以从几乎所有方向捕捉完整的周围环境。观看此类视频的观看者可以通过使用常规显示器上的指点设备或使用头戴式设备通过头部运动来选择他们正在观看的方向。这种新格式允许观看者在观看视频时改变观看方向，例如观看者可以从球场上的多个角度观看体育赛事。 然而，通过互联网大规模创建、存储和传播 360 度视频带来了重大挑战。 这些挑战是该项目的重点，该项目将开发一个名为 mi360World 的新系统和框架，以使互联网上的任何用户能够顺利交付 360 度视频并与之交互。 该项目如果成功，将显着改善 360 度视频交付，并将带来新的、更丰富的教育、培训和娱乐体验。 它还将有助于培训新一代多媒体系统研究人员和从业人员。通过互联网向全球分布的用户群提供高质量、个性化 360 度视频的机制是一个尚未解决的科学问题，它带来了以下挑战： 1) 超高带宽； 2）超低时延； 3）视图适应（针对用户头部运动）； 4）复杂的视频元数据和交付； 5) 视频体验质量 (QoE)。多年来，传统视频 QoE 得到了广泛的研究；然而，对于 360 度视频 QoE 的影响因素却知之甚少，需要构思和衡量新的指标。 拟议的 mi360World 系统融合了三个主要研究重点来应对上述挑战： 视频创作重点，通过生成导航图和电影规则来实现个性化观看，同时保持高 QoE 并减少晕机症。导航图的构建和电影规则的包含代表了该项目的主要创新，并封装在 360 度视频的三层元数据表示中：传输层、语义层和交互式故事讲述层。第二个重点是将 360 度视频可扩展地分发给全球不同的观看者，利用导航图和电影规则进行高效的转换预测预取和缓存。第三个重点是 QoE，目标是设计新颖的 QoE 指标和评估方法来评估晕机症。系统架构和算法将通过模拟、仿真和使用测试平台的基准测试进行广泛评估，以评估拟议研究的成功。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值和更广泛的影响审查进行评估，被认为值得支持标准。

项目成果

360BroadView: Viewer Management for Viewport Prediction in 360-Degree Video Live Broadcast

• DOI: 10.1145/3551626.3564939
• 发表时间: 2022-12
• 期刊: Proceedings of the 4th ACM International Conference on Multimedia in Asia
• 作者: Qian Zhou;Zhe Yang;Hongpeng Guo;Beitong Tian;K. Nahrstedt

Navigation Graph for Tiled Media Streaming

• DOI: 10.1145/3343031.3351021
• 发表时间: 2019-10
• 期刊: Proceedings of the 27th ACM International Conference on Multimedia
• 作者: Jounsup Park;K. Nahrstedt

Ultra-Sparse 360-Degree Camera View Synthesis for Immersive Virtual Tourism

用于沉浸式虚拟旅游的超稀疏 360 度摄像机视图合成

• DOI: 10.1109/mipr54900.2022.00057
• 发表时间: 2022
• 期刊: 2022 IEEE 5th International Conference on Multimedia Information Processing and Retrieval (MIPR
• 作者: Zhou, Qian;Nahrstedt, Klara
• 通讯作者: Nahrstedt, Klara

L3BOU: Low Latency, Low Bandwidth, Optimized Super-Resolution Backhaul for 360-Degree Video Streaming

L3BOU：低延迟、低带宽、优化的超分辨率回程，适用于 360 度视频流

• DOI: 10.1109/ism52913.2021.00031
• 发表时间: 2021
• 期刊: IEEE International Symposium on Multimedia (ISM
• 作者: Sarkar, Ayush;Murray, John;Dasari, Mallesham;Zink, Michael;Nahrstedt, Klara
• 通讯作者: Nahrstedt, Klara

360ViewPET: View Based Pose EsTimation for Ultra-Sparse 360-Degree Cameras

360ViewPET：超稀疏 360 度相机的基于视图的姿势估计

• DOI: 10.1109/ism52913.2021.00008
• 发表时间: 2021
• 期刊: IEEE International Symposium on Multimedia
• 作者: Zhou, Qian;Chen, Bo;Yang, Zhe;Guo, Hongpeng;Nahrstedt, Klara
• 通讯作者: Nahrstedt, Klara