Collaborative Research: SHF: Small: Software Hardware Architecture Co-Design for Enabling True Virtual Reality on Mobile Devices

合作研究：SHF：小型：软件硬件架构协同设计，在移动设备上实现真正的虚拟现实

• 批准号: 2215043
• 负责人: Guohong Cao
• 金额: $ 30万
• 依托单位: Pennsylvania State Univ University Park
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-07-01 至 2025-06-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2215043&HistoricalAwards=false
• 关键词: Collaborative; Research; SHF; Small; Software

Virtual Reality (VR) provides users with immersive senses of physical presence in the virtual world. Emerging VR applications such as Metaverse aim to realize a virtual counterpart of the real human world, which is truly indistinguishable from the physical world and allows complete simulation of real human life. Providing such true VR experiences results in an unprecedented amount of computation demand, and creates tremendous challenges on resource-constrained mobile VR devices. Since the computation demand cannot be met by only redesigning either the mobile software or hardware systems, this project addresses these challenges via software-hardware architecture co-design. As VR is considered as one emerging technique that can fundamentally transform the way people interact with the environment and each other, this project has great potential to benefit society by facilitating large-scale adoption of VR systems into every aspect of people’s daily lives. This project is also contributing to society through developing new curricula, disseminating research for education and training, engaging under-represented students in research, and outreaching to high-school students.The primary goal of this project is to provide true VR experiences on mobile devices by fully understanding and utilizing the close correlation between different VR software systems and hardware components. The project consists of three research tasks: (1) developing new VR computing techniques and GPU hardware designs to minimize the amount of VR computation, by adaptively identifying the redundancy among the rendered VR frames of different users and reusing these frames whenever possible; (2) investigating VR frame tiling techniques based on different VR users’ commonality in Field of Views, to minimize the mobile device’s energy consumption on receiving and processing VR frames according to the mobile hardware characteristics; (3) exploring the potential of heterogeneous mobile hardware, especially AI accelerators, to enable AI-based image upscaling on mobile devices that compensates the possible VR quality degradation due to communication link variations.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

虚拟现实 (VR) 为用户提供虚拟世界中的身临其境的感觉 新兴的 VR 应用（例如 Metaverse）旨在实现现实人类世界的虚拟对应，与物理世界真正没有区别，并允许完全模拟真实世界。提供这种真实的 VR 体验会带来前所未有的计算需求，并对资源有限的移动 VR 设备带来巨大的挑战，因为仅通过重新设计移动软件或硬件系统无法满足计算需求。该项目通过软件-硬件架构协同设计来解决这些挑战，因为 VR 被认为是一种可以从根本上改变人们与环境以及彼此交互方式的新兴技术，因此该项目具有通过促进大规模采用来造福社会的巨大潜力。该项目还通过开发新课程、传播教育和培训研究、让代表性不足的学生参与研究以及向高中生进行推广，为社会做出贡献。这个项目是为了提供真正的VR通过充分理解和利用不同 VR 软件系统和硬件组件之间的密切相关性，在移动设备上体验该项目包括三个研究任务：（1）开发新的 VR 计算技术和 GPU 硬件设计，以最大限度地减少 VR 计算量。自适应地识别不同用户的渲染VR帧之间的冗余，并尽可能地重用这些帧（2）研究基于不同VR用户视场共性的VR帧平铺技术，以最小化移动设备在接收和处理方面的能耗；根据移动硬件的 VR 帧（3）探索异构移动硬件（尤其是人工智能加速器）的潜力，以在移动设备上实现基于人工智能的图像升级，从而补偿由于通信链路变化而可能出现的 VR 质量下降。该奖项反映了 NSF 的法定使命，并被视为值得通过使用基金会的智力优点和更广泛的影响审查标准进行评估来支持。

项目成果

Energy-Efficient 360-Degree Video Streaming on Multicore-Based Mobile Devices

基于多核的移动设备上的节能 360 度视频流

• DOI: 10.1109/infocom53939.2023.10228863
• 发表时间: 2023-07
• 期刊: IEEE International Conference on Computer Communications
• 作者: Chen, Xianda;Cao, Guohong
• 通讯作者: Cao, Guohong

Macrotile: Toward QoE-Aware and Energy-Efficient 360-Degree Video Streaming

Macrotile：迈向 QoE 感知和节能的 360 度视频流

• DOI: 10.1109/tmc.2022.3233022
• 发表时间: 2024-01
• 期刊: IEEE Transactions on Mobile Computing
• 影响因子: 7.9
• 作者: Chen, Xianda;Tan, Tianxiang;Cao, Guohong
• 通讯作者: Cao, Guohong

Deep Learning on Mobile Devices With Neural Processing Units

具有神经处理单元的移动设备上的深度学习

• DOI: 10.1109/mc.2022.3215780
• 发表时间: 2023-11
• 期刊: Computer
• 影响因子: 2.2
• 作者: Tan, Tianxiang;Cao, Guohong
• 通讯作者: Cao, Guohong