NeTS: Small: Dynamic Predictive Streaming of 360 Degree Video

NeTS：小型：360 度视频的动态预测流

• 批准号: 1816500
• 负责人: Yong Liu
• 金额: $ 49.99万
• 依托单位: New York University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2018
• 资助国家: 美国
• 起止时间: 2018-10-01 至 2022-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1816500&HistoricalAwards=false
• 关键词: NeTS; Small; Dynamic; Predictive; Streaming

Virtual Reality (VR) and Augmented Reality (AR) applications are projected to be the next wave of "Killer Apps" in the future Internet. VR/AR applications facilitate vivid immersive virtual and augmented reality experience and create tremendous new opportunities in many domains, including education, business, healthcare, and entertainment, etc. Many VR/AR applications involve streaming of 360-degree video scenes. Compared with the traditional video streaming, 360-degree video streaming requires much higher network bandwidth and much lower packet delivery latency, and user's quality of experience is highly sensitive to the dynamics in both network environment and user viewing behaviors. Addressing these unique challenges, this project will develop novel 360-degree video coding and delivery solutions to enable high quality interactive, on-demand, and live video streaming. The project includes several research thrusts to enable novel joint coding-and-delivery solutions for high quality and robust 360-degree video streaming. For interactive streaming, novel Field-of-View (FoV) adaptive coding structure will be designed to achieve low encoding and decoding latency. Realtime joint optimization of streaming rate adaption and video coding bits allocation based on the predicted FoV will be studied to maximize the rendered video quality. For on-demand streaming, a two-tier video coding and delivery framework will be developed, and the rate allocation and video chunk scheduling between the two tiers will be investigated to strike the desired balance between the rendered video quality and streaming robustness. To facilitate predictive coding and delivery, the project will develop effective algorithms for predicting user FoVs, based on the past FoV trajectory and the audio and visual content through deep learning architectures. Personalized FoV prediction based on other users' view trajectories will also be explored under the framework of recommender systems. Fully-functional 360 video streaming prototypes will be developed and tested in controlled and real network environments to validate and improve the new designs. If successful, the research will lead to new theory and designs for 360-degree video coding and help enable the wide-spread deployment of high-quality and robust 360-video streaming systems. The research findings will be made available through publications, talks, open protocols, and open-source codes, allowing a multitude of developers, researchers, and companies to evolve 360-video streaming. The project will also create valuable research opportunities for graduate and undergraduate students, especially women and minority students. Interactions with industry will be facilitated through workshops and several research centers at the New York University.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

虚拟现实（VR）和增强现实（AR）应用预计将是未来互联网中的下一波“杀手应用程序”。 VR/AR应用程序促进了生动的沉浸式虚拟和增强现实体验，并在许多领域中创造了巨大的新机会，包括教育，商业，医疗保健和娱乐等。许多VR/AR应用程序涉及360度视频场景的流媒体。与传统的视频流相比，360度视频流需要更高的网络带宽和更低的数据包延迟，并且用户的体验质量对网络环境和用户查看行为的动态非常敏感。解决这些独特的挑战，该项目将开发新颖的360度视频编码和交付解决方案，以实现高质量的交互式，按需和实时视频流。 该项目包括几项研究推力，以启用新颖的联合编码和交付解决方案，以实现高质量和强大的360度视频流。对于交互式流，新颖的视野（FOV）自适应编码结构将旨在达到低编码和解码延迟。将研究基于预测的FOV的实时关节流量适应和视频编码位分配，以最大程度地提高渲染的视频质量。 对于按需流媒体，将开发两个两层的视频编码和交付框架，并将研究两个层之间的费率分配和视频块计划，以实现渲染的视频质量和流式稳定性之间所需的平衡。为了促进预测性编码和交付，该项目将根据过去的FOV轨迹以及通过深度学习体系结构来开发有效的算法来预测用户FOV。基于其他用户的视图轨迹的个性化FOV预测也将在推荐系统的框架下进行探索。将在受控和真实的网络环境中开发和测试完整的360个视频流原型，以验证和改进新设计。如果成功，这项研究将为360度视频编码提供新的理论和设计，并有助于实现高质量和强大的360-Video流媒体系统的广泛部署。研究结果将通过出版物，谈判，开放协议和开源代码提供，使许多开发人员，研究人员和公司可以进化360-Video流媒体。该项目还将为研究生和本科生，尤其是妇女和少数族裔学生创造宝贵的研究机会。与行业的互动将通过研讨会和纽约大学的几个研究中心进行促进。该奖项反映了NSF的法定任务，并被认为是值得通过基金会的知识分子优点和更广泛的影响评估标准通过评估来支持的。

项目成果

A Two-Tier System for On-Demand Streaming of 360 Degree Video Over Dynamic Networks

• DOI: 10.1109/jetcas.2019.2898877
• 发表时间: 2019-02
• 期刊: IEEE Journal on Emerging and Selected Topics in Circuits and Systems
• 影响因子: 4.6
• 作者: Liyang Sun;F. Duanmu;Y. Liu;Yao Wang;Y. Ye;Hang Shi;David H. Dai

Live 360 Degree Video Delivery Based on User Collaboration in a Streaming Flock

• DOI: 10.1109/tmm.2022.3149642
• 发表时间: 2023
• 期刊: IEEE Transactions on Multimedia
• 影响因子: 7.3
• 作者: Liyang Sun;Yixiang Mao;Tongyu Zong;Yong Liu;Yao Wang

Very Long Term Field of View Prediction for 360-degree Video Streaming

• DOI: 10.1109/mipr.2019.00060
• 发表时间: 2019-01-01
• 期刊: 2019 2ND IEEE CONFERENCE ON MULTIMEDIA INFORMATION PROCESSING AND RETRIEVAL (MIPR 2019)
• 作者: Li, Chenge;Zhang, Weixi;Wang, Yao
• 通讯作者: Wang, Yao

Realtime mobile bandwidth and handoff predictions in 4G/5G networks

• DOI: 10.1016/j.comnet.2021.108736
• 发表时间: 2022-01-13
• 期刊: COMPUTER NETWORKS
• 影响因子: 5.6
• 作者: Mei, Lifan;Gou, Jinrui;Liu, Yong
• 通讯作者: Liu, Yong

Tightrope walking in low-latency live streaming: optimal joint adaptation of video rate and playback speed

低时延直播走钢丝：视频码率与播放速度的优化联合适配

• DOI: 10.1145/3458305.3463382
• 发表时间: 2021
• 期刊: MMSys '21: Proceedings of the 12th ACM Multimedia Systems Conference
• 作者: Sun, Liyang;Zong, Tongyu;Wang, Siquan;Liu, Yong;Wang, Yao
• 通讯作者: Wang, Yao