CNS Core: Small: Collaborative: Content-Based Viewport Prediction Framework for Live Virtual Reality Streaming

CNS 核心：小型：协作：用于直播虚拟现实流的基于内容的视口预测框架

• 批准号: 2109982
• 负责人: Yan Yan
• 金额: $ 11万
• 依托单位: Illinois Institute of Technology
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-01-01 至 2023-07-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2109982&HistoricalAwards=false
• 关键词: CNS; Core; Small; Collaborative; Content

Virtual reality (VR) video streaming has been gaining popularity recently with the rapid adoption of mobile head mounted display (HMD) devices in the consumer video market. As the cost for the immersive experience drops, VR video streaming introduces new bandwidth and performance challenges, especially in live streaming, due to the delivery of 360-degree views. This project develops a new content-based viewport prediction framework to improve the bandwidth and performance in live VR streaming, which predicts the user's viewport through a fusion of tracking the moving objects in the video, extracting the video semantics, and modeling the user's viewport of interest.This project consists of three research thrusts. First, it develops a content-based viewport prediction framework for live VR streaming by tracking the motions and semantics of the objects. Second, it employs hardware and software techniques to facilitate real-time execution and scale the viewport prediction mechanism to a large number of users. Third, it develops evaluation frameworks to verify the functionality, performance, and scalability of the approach. The project uniquely considers the correlation between video content and user behavior, which leverages the deterministic nature of the former to conquer the randomness of the latter.With the rapidly increasing popularity of VR systems in domain-specific immersive environments, the project will benefit several VR-related fields of studies with significant bandwidth savings and performance improvements, such as VR-based live broadcast, healthcare, and scientific visualization. Moreover, the interdisciplinary nature of the project will enhance the education and recruitment of underrepresented minorities in several science, technology, engineering, and mathematics (STEM) fields.The project repository will be stored on a publicly accessible server (https://github.com/hwsel). All the project data will be maintained for at least five years following the end of the grant period.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

随着移动头戴式显示 (HMD) 设备在消费视频市场的快速采用，虚拟现实 (VR) 视频流最近越来越受欢迎。随着沉浸式体验成本的下降，VR 视频流带来了新的带宽和性能挑战，特别是在直播中，由于提供 360 度视图。该项目开发了一种新的基于内容的视口预测框架，以提高实时 VR 流媒体的带宽和性能，该框架通过融合跟踪视频中的移动对象、提取视频语义以及对用户视口进行建模来预测用户的视口。兴趣。该项目由三个研究重点组成。首先，它通过跟踪对象的运动和语义，为实时 VR 流开发了一个基于内容的视口预测框架。其次，它采用硬件和软件技术来促进实时执行并将视口预测机制扩展到大量用户。第三，它开发评估框架来验证该方法的功能、性能和可扩展性。该项目独特地考虑了视频内容与用户行为之间的相关性，利用前者的确定性来克服后者的随机性。随着VR系统在特定领域沉浸式环境中的迅速普及，该项目将使多个VR行业受益- 显着节省带宽和提高性能的相关研究领域，例如基于 VR 的直播、医疗保健和科学可视化。此外，该项目的跨学科性质将加强在多个科学、技术、工程和数学 (STEM) 领域中代表性不足的少数群体的教育和招聘。项目存储库将存储在可公开访问的服务器上 (https://github.com/stem)。 com/hwsel）。所有项目数据将在资助期结束后保存至少五年。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Lipschitz Continuity Retained Binary Neural Network

Lipschitz 连续性保留二元神经网络

• 发表时间: 2022-10
• 期刊: European Conference on Computer Vision
• 作者: Yuzhang Shang;Dan Xu;Bin Duan;Ziliang Zong;Liqiang Nie;Yan Yan
• 通讯作者: Yan Yan

Saying the Unseen: Video Descriptions via Dialog Agents

讲述未见之事：通过对话代理进行视频描述

• DOI: 10.1109/tpami.2021.3093360
• 发表时间: 2021-06-26
• 期刊: IEEE Transactions on Pattern Analysis and Machine Intelligence
• 影响因子: 23.6
• 作者: Ye Zhu;Yu Wu;Yi Yang;Yan Yan
• 通讯作者: Yan Yan

Revisiting Optical Flow Estimation in 360 Videos

重新审视 360 度视频中的光流估计

• DOI: 10.1109/icpr48806.2021.9412035
• 发表时间: 2021-01
• 期刊: International Conference on Pattern Recognition (ICPR
• 作者: Bhandari, Keshav;Zong, Ziliang;Yan, Yan
• 通讯作者: Yan, Yan

Learning Audio-Visual Correlations From Variational Cross-Modal Generation

从变分跨模态生成中学习视听相关性

• DOI: 10.1109/icassp39728.2021.9414296
• 发表时间: 2021-06
• 期刊: Speech and Signal Processing (ICASSP
• 作者: Zhu, Ye;Wu, Yu;Latapie, Hugo;Yang, Yi;Yan, Yan
• 通讯作者: Yan, Yan

Cross-View Exocentric to Egocentric Video Synthesis

跨视图从外中心到以自我为中心的视频合成

• DOI: 10.1145/3474085.3475596
• 发表时间: 2021-07-07
• 期刊: Proceedings of the 29th ACM International Conference on Multimedia
• 作者: Gaowen Liu;Hao Tang;Hugo Latapie;Jason J. Corso;Yan Yan
• 通讯作者: Yan Yan