SHF: Medium: Exploring an Edge Platform Design Trajectory for Next Generation XR Applications

SHF：中：探索下一代 XR 应用的边缘平台设计轨迹

• 批准号: 2211018
• 负责人: Chitaranjan Das
• 金额: $ 120万
• 依托单位: Pennsylvania State Univ University Park
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-10-01 至 2025-09-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2211018&HistoricalAwards=false
• 关键词: SHF; Medium; Exploring; Edge; Platform

Extended Reality (XR) encompassing virtual reality (VR), where a computer-generated environment can be explored and interacted with by users, at one end of the spectrum and augmented reality (AR), where parts of users' physical world are enhanced with computer-generated data, at the other end of the spectrum, is an emerging technology with transformational potential in many domains of national importance including science, healthcare, education, entertainment, and commerce. While various flavors of XR products (tethered, standalone, screenless) are commercially available today, they only offer point solutions for very specific applications varying in cost, performance and power envelope. Furthermore, as many XR applications will run in an edge-computing scenario, their design demands not only power and energy considerations, but also the adaptability and field-customization to efficiently support any future applications. As a result, there is a critical need to develop frameworks that enable designers to generate, test, evaluate and customize a large spectrum of next-generation XR devices with accompanying software support. Such frameworks can expedite time-to-discovery for AR/VR-based applications and redefine the future of computing. This project seeks to design an agile and cost-effective hardware and software platform, AVATAR, that can be progressively adapted and customized to the performance, quality-of-service (QoS), and energy-efficiency needs of a diverse set of XR applications. AVATAR consists of four inter-twined thrusts. First, a detailed profiling of four types of XR applications on three different edge platforms is conducted to understand the latency, power and utilization of different stages in an XR pipeline for identifying the bottleneck stages/kernels. Second, it undertakes a coordinated design of the compiler and runtime system for controlling tunable knobs such as code partitioning and dynamic reconfiguration. Next, hardware design alternatives embracing device heterogeneity such as general-purpose CPU cores, graphic processing units (GPUs), and field programmable gate arrays (FPGAs) for the targeted kernels/pipeline stages of an XR application along with scope for reconfigurability are explored for desired performance-power tradeoffs. In addition, the scope for exploiting Wi-Fi/5G communication to facilitate efficient computation partitioning between an edge device and cloud is investigated. Finally, a comprehensive evaluation framework to evaluate the efficacy of the proposed solutions for different XR applications is being developed. On the educational front, AVATAR includes a new special-topics course and involvement of undergraduate and graduate students in this emerging research field, where students get exposure to cross-cutting topics in computer architecture, compilers, runtime systems and networking, as well as new classes of XR applications. It also includes several Broadening Participation in Computing (BPC) activities such as a summer camp targeting girls and collaboration with the Education Department at Penn State to expose K-12 students to many areas of computer science and engineering, with the primary focus being on XR applications.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

扩展现实（XR）包括虚拟现实（VR）和增强现实（AR），其中用户可以探索计算机生成的环境并与之交互，而增强现实（AR）则通过增强现实来增强用户的部分物理世界。另一方面，计算机生成的数据是一种新兴技术，在科学、医疗保健、教育、娱乐和商业等许多国家重要领域具有变革潜力。虽然目前市面上有各种类型的 XR 产品（系留式、独立式、无屏式），但它们仅针对成本、性能和功率范围各不相同的非常特定的应用提供单点解决方案。此外，由于许多 XR 应用程序将在边缘计算场景中运行，因此它们的设计不仅需要考虑功耗和能源，还需要适应性和现场定制，以有效支持任何未来的应用程序。因此，迫切需要开发框架，使设计人员能够生成、测试、评估和定制各种下一代 XR 设备并附带软件支持。此类框架可以加快基于 AR/VR 的应用程序的发现时间，并重新定义计算的未来。该项目旨在设计一个灵活且经济高效的硬件和软件平台 AVATAR，该平台可以逐步适应和定制，以满足各种 XR 应用程序的性能、服务质量 (QoS) 和能效需求。 《阿凡达》由四个相互交织的主力组成。首先，对三个不同边缘平台上的四种类型的 XR 应用程序进行详细分析，以了解 XR 管道中不同阶段的延迟、功耗和利用率，从而识别瓶颈阶段/内核。其次，它对编译器和运行时系统进行协调设计，以控制代码分区和动态重新配置等可调旋钮。接下来，探索包含设备异构性的硬件设计替代方案，例如用于 XR 应用程序的目标内核/管道阶段的通用 CPU 内核、图形处理单元 (GPU) 和现场可编程门阵列 (FPGA)，以及可重构性范围所需的性能-功耗权衡。此外，还研究了利用 Wi-Fi/5G 通信促进边缘设备和云之间高效计算划分的范围。最后，正在开发一个综合评估框架，以评估所提出的解决方案针对不同 XR 应用的功效。 在教育方面，《阿凡达》包括一门新的专题课程，并让本科生和研究生参与这一新兴研究领域，学生可以接触到计算机体系结构、编译器、运行时系统和网络等跨领域主题，以及新的知识。 XR 应用程序类别。它还包括多项扩大计算参与 (BPC) 活动，例如针对女孩的夏令营以及与宾夕法尼亚州立大学教育部的合作，让 K-12 学生接触计算机科学和工程的许多领域，其中主要重点是 XR该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

I am an Earphone and I can Hear my Users Face: Facial Landmark Tracking using Smart Earphones

我是耳机，我可以听到用户的脸：使用智能耳机进行面部地标跟踪

• DOI: 10.1145/3614438
• 发表时间: 2023-08
• 期刊: ACM Transactions on Internet of Things
• 作者: Zhang, Shijia;Lu, Taiting;Zhou, Hao;Liu, Yilin;Liu, Runze;Gowda, Mahanth
• 通讯作者: Gowda, Mahanth