CNS Core: Small: Collaborative: Content-Based Viewport Prediction Framework for Live Virtual Reality Streaming

CNS 核心：小型：协作：用于直播虚拟现实流的基于内容的视口预测框架

• 批准号: 1909172
• 负责人: Yanzhi Wang
• 金额: $ 17.12万
• 依托单位: Northeastern University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-08-15 至 2024-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1909172&HistoricalAwards=false
• 关键词: CNS; Core; Small; Collaborative; Content

Virtual reality (VR) video streaming has been gaining popularity recently with the rapid adoption of mobile head mounted display (HMD) devices in the consumer video market. As the cost for the immersive experience drops, VR video streaming introduces new bandwidth and performance challenges, especially in live streaming, due to the delivery of 360-degree views. This project develops a new content-based viewport prediction framework to improve the bandwidth and performance in live VR streaming, which predicts the user's viewport through a fusion of tracking the moving objects in the video, extracting the video semantics, and modeling the user's viewport of interest.This project consists of three research thrusts. First, it develops a content-based viewport prediction framework for live VR streaming by tracking the motions and semantics of the objects. Second, it employs hardware and software techniques to facilitate real-time execution and scale the viewport prediction mechanism to a large number of users. Third, it develops evaluation frameworks to verify the functionality, performance, and scalability of the approach. The project uniquely considers the correlation between video content and user behavior, which leverages the deterministic nature of the former to conquer the randomness of the latter.With the rapidly increasing popularity of VR systems in domain-specific immersive environments, the project will benefit several VR-related fields of studies with significant bandwidth savings and performance improvements, such as VR-based live broadcast, healthcare, and scientific visualization. Moreover, the interdisciplinary nature of the project will enhance the education and recruitment of underrepresented minorities in several science, technology, engineering, and mathematics (STEM) fields.The project repository will be stored on a publicly accessible server (https://github.com/hwsel). All the project data will be maintained for at least five years following the end of the grant period.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

随着消费者视频市场中移动头部安装显示器（HMD）设备的迅速采用，虚拟现实（VR）视频流最近一直在越来越受欢迎。随着身临其境体验的成本下降，VR视频流媒体引入了新的带宽和性能挑战，尤其是在实时流中，由于提供了360度的观看次数。该项目开发了一个新的基于内容的视口预测框架，以改善实时VR流中的带宽和性能，该框架通过融合来跟踪视频中的移动对象，提取视频语义，并建模用户对用户的兴趣视图来预测用户的视口。首先，它通过跟踪对象的动作和语义来开发一个基于内容的视口预测框架，以实时VR流。其次，它采用硬件和软件技术来促进实时执行，并将视口预测机制扩展到大量用户。第三，它开发了评估框架来验证方法的功能，性能和可扩展性。 The project uniquely considers the correlation between video content and user behavior, which leverages the deterministic nature of the former to conquer the randomness of the latter.With the rapidly increasing popularity of VR systems in domain-specific immersive environments, the project will benefit several VR-related fields of studies with significant bandwidth savings and performance improvements, such as VR-based live broadcast, healthcare, and scientific visualization.此外，该项目的跨学科性质将增强多种科学，技术，工程和数学（STEM）字段中代表性不足的少数群体的教育和招聘。项目存储库将存储在公共访问服务器上（https://github.com/hwsel）。在赠款期结束后，所有项目数据将至少维持五年。该奖项反映了NSF的法定任务，并且使用基金会的知识分子优点和更广泛的影响审查标准，被认为值得通过评估来获得支持。

项目成果

All-in-One: A Highly Representative DNN Pruning Framework for Edge Devices with Dynamic Power Management

• DOI: 10.1145/3508352.3549379
• 发表时间: 2022-10
• 期刊: 2022 IEEE/ACM International Conference On Computer Aided Design (ICCAD)
• 作者: Yifan Gong;Zheng Zhan;Pu Zhao;Yushu Wu;Chaoan Wu;Caiwen Ding;Weiwen Jiang;Minghai Qin;Yanzhi Wang

Teachers Do More Than Teach: Compressing Image-to-Image Models

• DOI: 10.1109/cvpr46437.2021.01339
• 发表时间: 2021-03
• 期刊: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
• 作者: Qing Jin;Jian Ren;Oliver J. Woodford;Jiazhuo Wang;Geng Yuan;Yanzhi Wang;S. Tulyakov

Mix and Match: A Novel FPGA-Centric Deep Neural Network Quantization Framework

• DOI: 10.1109/hpca51647.2021.00027
• 发表时间: 2021-01-01
• 期刊: 2021 27TH IEEE INTERNATIONAL SYMPOSIUM ON HIGH-PERFORMANCE COMPUTER ARCHITECTURE (HPCA 2021)
• 作者: Chang, Sung-En;Li, Yanyu;Lin, Xue
• 通讯作者: Lin, Xue

Non-Structured DNN Weight Pruning--Is It Beneficial in Any Platform?

• DOI: 10.1109/tnnls.2021.3063265
• 发表时间: 2021-03-18
• 期刊: IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS
• 影响因子: 10.4
• 作者: Ma, Xiaolong;Lin, Sheng;Wang, Yanzhi
• 通讯作者: Wang, Yanzhi

Layer Freezing & Data Sieving: Missing Pieces of a Generic Framework for Sparse Training

• DOI: 10.48550/arxiv.2209.11204
• 发表时间: 2022-09
• 期刊: ArXiv
• 作者: Geng Yuan;Yanyu Li;Sheng Li;Zhenglun Kong;S. Tulyakov;Xulong Tang;Yanzhi Wang;Jian Ren