CNS Core: Small: A Split Software Architecture for Enabling High-Quality Mixed Reality on Commodity Mobile Devices

CNS 核心：小型：用于在商用移动设备上实现高质量混合现实的分离式软件架构

• 批准号: 2112778
• 负责人: Charlie Hu
• 金额: $ 42.44万
• 依托单位: Purdue University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-10-01 至 2024-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2112778&HistoricalAwards=false
• 关键词: CNS; Core; Small; Split; Software

By blending the physical and digital worlds into a programmed experience, Mixed Reality (MR) allows users to visualize and interact with digital information such as 3D overlays and real-time data and has important applications in many societal domains including education, remote working, military training, and health care such as tele-medicine. Despite the tremendous potential of the MR technology, the MR solutions available in today’s market are either enterprise-grade which are costly or consumer-grade which can only support low-quality MR content which leads to poor user experience. The high cost and/or low-quality of current enterprise-grade and consumer-grade MR solutions lead to a fundamental “content-adoption” dilemma faced by the MR industry: the lack of MR content has limited the market penetration of custom-made MR headsets, and the low market penetration of MR headsets in turn has hindered the development of MR content. This NSF CSR project proposal will develop key technologies to enable high-quality MR on commodity mobile devices like smartphones, etc.., viewed by a simple see-through head-mount devices (HMD) with a high-resolution camera for input and a projector for output such as Nreal Light glasses. Such technologies will transform millions of smartphones (equipped with the above inexpensive HMDs) into ubiquitous MR devices and in doing so help the MR industry to overcome the “content-adoption” dilemma and pave the way for wide adoption of the MR technology and its many important applications. This project aims to create the first split software architecture that enables high-quality MR applications to run on commodity mobile devices; the capability to jointly optimize offloading multiple Deep Neural Network (DNN)-based tasks constituting a complex, resource-intensive application such as MR over the bandwidth-limited and time-varying wireless network; the capability to jointly schedule multiple DNN-based tasks of resource-intensive applications such as MR to efficiently share all local resources such as the CPU, GPU, and other processors such as NPU on emerging mobile devices; and the capability to support high-quality multi-player MR on commodity mobile devices by scaling the split software architecture across multiple mobile devices to efficiently share the limited global resources such as the wireless network and the edge cloud.The proposed research will have lasting impact on knowledge discovery, the computer industry, and the society. Technically, this work anticipates having far-reaching impacts outside the area of supporting AR/VR/MR on commodity smartphones by developing general edge-assisted software architectures for enabling the class of latency-sensitive 5G/6G applications on current and future mobile computing platforms such as smart glasses. Developing the proposed technologies for MR have the potential to fundamentally overcome the “deployment-content” dilemma faced by the industry as well as fostering the proliferation and wide adoption of MR technologies and its many societal applications. The importance of this work will be further heightened by making smartphones an important enabler of accessing information and new technologies like AR/VR/MR for people in both developed and developing countries and hence being an important tool in overcoming the “digital divide".This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

通过将物理和数字世界融入程序的体验中，混合现实（MR）允许用户可视化并与数字信息（例如3D叠加层和实时数据）进行互动，并在许多社会领域中都有重要的应用，包括教育，远程工作，军事培训和诸如远程医疗等医疗保健。尽管MR技术具有巨大的潜力，但当今市场上可用的MR解决方案要么是企业级，要么是昂贵的，要么是消费者级，只能支持低质量的MR内容，从而导致用户体验差。当前企业级和消费级MR解决方案的高成本和/或低质量导致MR行业面临的基本“内容添加”困境：缺乏MR内容限制了定制的MR Headsets的市场渗透，而MR耳机的市场渗透率又妨碍了MR内容的开发。该NSF CSR项目建议将开发关键技术，以使商品移动设备（如智能手机等）能够高质量的MR，该设备由简单的透明头部设备（HMD）查看，并配备具有高分辨率摄像头的输入和投影仪的投影仪，以供输出投影仪。这样的技术将将数百万个智能手机（配备上述廉价的HMD）转变为无处不在的MR设备，并在此过程中帮助MR行业克服“内容添加”困境，并为广泛采用MR技术及其许多重要应用铺平道路。该项目旨在创建第一个拆分软件体系结构，以使高质量的MR应用程序能够在商品移动设备上运行；共同优化卸载多个深神经网络（DNN）的任务的能力，构成了复杂的资源密集型应用程序，例如在带宽限制和时变的无线网络上的MR；共同安排多个基于DNN的资源密集型应用程序的任务，例如MR有效共享所有本地资源，例如CPU，GPU和其他处理器，例如在新兴移动设备上的NPU；以及通过扩展跨多个移动设备的拆分软件体系结构来有效地共享有限的全球资源（例如无线网络和边缘云），可以在商品移动设备上支持高质量的多人MR。从技术上讲，这项工作预计，通过开发一般边缘辅助软件体系结构来支持AR/VR/MR在支持AR/VR/MR之外产生深远的影响，以在当前和未来的移动计算平台（例如智能眼镜）上启用延迟敏感的5G/6G应用程序。为MR开发拟议的技术有可能从根本上克服该行业所面临的“部署”困境，并促进MR Technologies及其许多社交应用的扩散和广泛采用。通过使智能手机成为访问信息和新技术（例如AR/VR/MR）对发达国家和发展中国家的人员（例如，是克服“数字鸿沟”）的重要工具，这项工作的重要性将进一步提高。这是NSF的法定任务，反映了NSF的法定任务，并通过使用基金会的智能评估来评估诚实的支持。

项目成果

Poster: BystandAR: Protecting Bystander Visual Data in Augmented Reality Systems

• DOI: 10.1145/3581791.3597377
• 发表时间: 2023-06
• 期刊: Proceedings of the 21st Annual International Conference on Mobile Systems, Applications and Services
• 作者: Matthew Corbett;Brendan David-John;Jiacheng Shang;Y. C. Hu;Bo Ji

BystandAR: Protecting Bystander Visual Data in Augmented Reality Systems

• DOI: 10.1145/3581791.3596830
• 发表时间: 2023-06
• 期刊: Proceedings of the 21st Annual International Conference on Mobile Systems, Applications and Services
• 作者: Brendan David-John;Jiacheng Shang;Bo Ji;Matthew Corbett;Y. C. Hu;Bo Ji. 2023. BystandAR

Do Larger (More Accurate) Deep Neural Network Models Help in Edge-assisted Augmented Reality?

更大（更准确）的深度神经网络模型有助于边缘辅助增强现实吗？

• DOI: 10.1145/3472727.3472807
• 发表时间: 2021
• 期刊: NAI'21: Proceedings of the ACM SIGCOMM 2021 Workshop on Network-Application Integration
• 作者: Meng, Jiayi;Kong, Zhaoning;Xu, Qiang;Hu, Y. Charlie
• 通讯作者: Hu, Y. Charlie

An In-Depth Study of Uplink Performance of 5G mmWave Networks

5G毫米波网络上行链路性能的深入研究

• DOI: 10.1145/3538394.3546042
• 发表时间: 2022
• 期刊: and Use Cases (5G-MeMU
• 作者: Moinak Ghoshal, Z. Jonny

Can 5G mmWave Support Multi-user AR?

5G毫米波能否支持多用户AR？

• DOI: 10.1007/978-3-030-98785-5_8
• 发表时间: 2022
• 期刊: vol 13210
• 作者: Moinak Ghoshal, Pranab Dash