CIF: Small: Mobile Immersive Communication: View Sampling and Rate-Distortion Limits

CIF：小型：移动沉浸式通信：查看采样和率失真限制

• 批准号: 2031881
• 负责人: Jacob Chakareski
• 金额: $ 3.37万
• 依托单位: New Jersey Institute of Technology
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-09-01 至 2023-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2031881&HistoricalAwards=false
• 关键词: CIF; Small; Mobile; Immersive; Communication

Free-viewpoint video is an emerging technology for visual communication that creates the sensation of 3D immersion in the remote scene by allowing the user to dynamically switch between arbitrary viewpoints. It has the potential to advance society by enabling virtual human transportation and boost the global economy and quality of life. At present, free-viewpoint video is limited to high-end computing environments and studio-type settings, due to its higher bandwidth and complexity expansion over single-view video. Furthermore, the fundamental questions of viewpoint sampling (camera location) and resource allocation across the captured views are largely unanswered, due to the nascency of the technology, and are addressed using suboptimal heuristic approaches, thus penalizing system efficiency. These two characteristics would otherwise make free-viewpoint video impractical and preclude its broader deployment, in particular on mobile devices, due to their constrained bandwidth, battery power, and CPU capabilities. However, the latter have become a primary platform for computing and communication needs, anywhere and anytime, a trend that will only accelerate in the future. Thus, it is anticipated that only by enabling ubiquitous and seamless mobile free-viewpoint video may the full potential of immersive communication be achieved. This project seeks to achieve this goal via concerted advances in signal representation, wireless video communication, and user-action modeling that will be integrated holistically. The advances delivered by this investigation will have broad impact across diverse fields that involve live video communication via multiple viewpoints, including telemedicine, telepresence and telecollaboration, remote monitoring and control, entertainment (3D and free-viewpoint TV), gaming and virtual worlds, people-centric sensing and connected-community applications.The project will pursue the following technical thrusts: (i) Characterization of the fundamental trade-offs between viewpoint space sampling, rate allocation, and signal fidelity in immersive mobile communication; (ii) Derivation of the optimal sampling policy, at the view and data-unit levels, in uplink communication; (iii) Design of novel view and rate scalable coding for multi-view broadcast and derivation of the optimal view embedding policy, as a function of the broadcast rate, in downlink communication; (iv) Characterization of the optimal scheduling policy for local ad-hoc communication between mobile clients; and (v) Integration of energy conservation and characterization of the trade-offs between battery lifetime and multi-view application performance.

自由视点视频是一种新兴的视觉通信技术，它允许用户在任意视点之间动态切换，从而在远程场景中营造 3D 沉浸感。它有潜力通过实现虚拟人类交通来推动社会进步，并促进全球经济和生活质量。目前，由于自由视点视频比单视点视频具有更高的带宽和复杂性扩展，因此仅限于高端计算环境和工作室类型的设置。此外，由于该技术还处于初级阶段，视点采样（摄像机位置）和跨捕获视图的资源分配的基本问题在很大程度上没有得到解答，并且使用次优启发式方法来解决，从而损害了系统效率。否则，这两个特性将使自由视点视频变得不切实际，并妨碍其更广泛的部署，特别是在移动设备上，因为它们的带宽、电池电量和 CPU 功能有限。然而，后者已成为随时随地满足计算和通信需求的主要平台，这一趋势在未来只会加速。因此，预计只有通过实现无处不在、无缝的移动自由视点视频，才能充分发挥沉浸式通信的潜力。该项目旨在通过信号表示、无线视频通信和用户动作建模方面的协调进步来实现这一目标，这些方面将进行整体集成。 这项调查取得的进展将对涉及多视角实时视频通信的各个领域产生广泛影响，包括远程医疗、远程呈现和远程协作、远程监控和控制、娱乐（3D 和自由视点电视）、游戏和虚拟世界、人该项目将追求以下技术主旨：（i）表征沉浸式移动通信中视点空间采样、速率分配和信号保真度之间的基本权衡； (ii) 上行链路通信中视图和数据单元级别的最佳采样策略的推导； (iii) 设计用于多视图广播的新颖视图和速率可伸缩编码，以及在下行链路通信中导出最佳视图嵌入策略（作为广播速率的函数）； (iv) 移动客户端之间本地临时通信的最佳调度策略的表征； (v) 节能的整合以及电池寿命和多视图应用性能之间权衡的表征。

项目成果

Millimeter Wave meets Edge Computing for Mobile VR with High-Fidelity 8K Scalable 360° Video

毫米波与移动 VR 边缘计算结合，提供高保真 8K 可扩展 360° 视频

• DOI: 10.1109/mmsp.2019.8901752
• 发表时间: 2019-09-01
• 期刊: 2019 IEEE 21st International Workshop on Multimedia Signal Processing (MMSP)
• 作者: Sabyasachi Gupta;Jacob Chakareski;P. Popovski
• 通讯作者: P. Popovski

FBDT: Forward and Backward Data Transmission Across RATs for High Quality Mobile 360-Degree Video VR Streaming

FBDT：跨 RAT 的前向和后向数据传输，实现高质量移动 360 度视频 VR 流媒体

• DOI: 10.1145/3587819.3590987
• 发表时间: 2023-06
• 期刊: ACM Multimedia Systems
• 作者: Srinivasan, Suresh;Shippey, Sam;Aryafar, Ehsan;Chakareski, Jacob
• 通讯作者: Chakareski, Jacob

Collaborative Content Placement Among Wireless Edge Caching Stations With Time-to-Live Cache

具有生存时间缓存的无线边缘缓存站之间的协作内容放置

• DOI: 10.1109/tmm.2019.2929004
• 发表时间: 2020-02-01
• 期刊: IEEE Transactions on Multimedia
• 影响因子: 7.3
• 作者: Lixing Chen;Linqi Song;Jacob Chakareski;Jie Xu
• 通讯作者: Jie Xu

UAV-IoT for Next Generation Virtual Reality

用于下一代虚拟现实的无人机物联网

• DOI: 10.1109/tip.2019.2921869
• 发表时间: 2019-12
• 期刊: IEEE Transactions on Image Processing
• 影响因子: 10.6
• 作者: Chakareski; Jacob
• 通讯作者: Jacob

Delay-Sensitive Energy-Harvesting Wireless Sensors: Optimal Scheduling, Structural Properties, and Approximation Analysis

延迟敏感能量收集无线传感器：最优调度、结构特性和近似分析

• DOI: 10.1109/tcomm.2019.2956510
• 发表时间: 2020-04
• 期刊: IEEE Transactions on Communications
• 影响因子: 8.3
• 作者: Sharma, Nikhilesh;Mastronarde, Nicholas;Chakareski, Jacob
• 通讯作者: Chakareski, Jacob