CCRI: New: An Open End-to-End Extended Reality System Infrastructure: Enabling Domain-Specific Edge Systems Research

CCRI：新：开放的端到端扩展现实系统基础设施：支持特定领域的边缘系统研究

• 批准号: 2120464
• 负责人: Sarita Adve
• 金额: $ 100万
• 依托单位: University of Illinois at Urbana-Champaign
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-10-01 至 2024-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2120464&HistoricalAwards=false
• 关键词: CCRI; New; Open; End; Extended

Extended reality (XR), which encompasses virtual, augmented, and mixed reality (AR, VR, MR) and also referred to as immersive computing, is expected to pervade most human endeavors --- it will affect the way we teach, conduct science, practice medicine, entertain ourselves, train professionals, interact socially, and more. Many have said it will be the next interface for most of computing. While current XR systems exist today, they are far from providing a tetherless experience approaching perceptual abilities of humans. There is a gap of several orders of magnitude between what is needed and achievable in performance, power, and usability, requiring deep innovations from system researchers. At the same time, with the end of Dennard scaling and Moore's law, application-driven specialization or domain-specific computing has emerged as a key architectural technique to meet the requirements of emerging applications, Computer architects have responded with an explosion of research on highly efficient accelerators, targeting machine learning and other domains. To truly achieve the promise of efficient domain-specific computing in general and for the XR domain in particular, however, requires systems researchers to broaden their portfolio beyond specialization for individual accelerators. Instead, researchers must develop the science for specializing for a domain-specific system which may consist of multiple sub-domains requiring multiple parallel heterogeneous accelerators that interact with each other to collectively meet end-user demands. A key obstacle to domain-specific systems research for XR is that (until our work) there have been no open source benchmarks or testbeds covering the entire XR workflow to drive such research. This project develops an open source end-to-end infrastructure for XR devices. It builds on an initial research prototype, ILLIXR (Illinois Extended Reality Testbed). The system is being designed to contain state-of-the-art components for a complete XR workflow, an extensible runtime that orchestrates the scheduling of these components, and extensive telemetry support to measure performance, power, and end-to-end quality of experience metrics. The system is extensible and supports a variety of operating systems (e.g., Linux, Android) and heterogeneous platforms (e.g., NVIDIA Jetson, Qualcomm Snapdragon, etc.), sensors (e.g., cameras, IMUs, etc.), and various XR applications. It enables new research opportunities in all parts of the computing stack to tackle end-to-end XR system innovations that were previously not possible. Systems researchers benefit from using the infrastructure to drive new research in post-Moore domain-specific systems, in the areas of computer architecture, programming languages, compilers, runtime systems, and security and privacy. The end-to-end infrastructure drives new techniques in co-designed systems that are optimized for end-to-end user experiences. For applications, XR encompasses multiple sub-domains such as computer vision, robotics, graphics, signal processing, and machine learning. Algorithms researchers in these areas can prototype and test new algorithms that are optimized for end-to-end system efficiencies without worrying about implementing the rest of the stack, and XR researchers in particular will be able to design systems optimized for the end-to-end user experience. This work addresses two of the most important problems in computing -- dealing with the end of Moore's law and designing systems that achieve the potential of immersive computing. Both have the potential for tremendous impact on society at large.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

扩展现实 (XR) 包括虚拟现实、增强现实和混合现实（AR、VR、MR），也称为沉浸式计算，预计将渗透到大多数人类活动中——它将影响我们教学、开展科学的方式、行医、娱乐、培训专业人士、社交互动等等。许多人表示它将成为大多数计算的下一个接口。虽然目前的 XR 系统已经存在，但它们还远远不能提供接近人类感知能力的无线体验。在性能、功耗和可用性方面，所需的与可实现的之间存在几个数量级的差距，需要系统研究人员进行深入创新。与此同时，随着登纳德缩放和摩尔定律的结束，应用程序驱动的专业化或特定领域计算已成为满足新兴应用程序要求的关键架构技术，计算机架构师对此做出了回应，对高度计算的研究呈爆炸式增长。高效的加速器，针对机器学习和其他领域。 然而，为了真正实现高效特定领域计算的承诺，特别是 XR 领域，需要系统研究人员扩大其产品组合，超越针对单个加速器的专业化。相反，研究人员必须开发专门针对特定领域系统的科学，该系统可能由多个子域组成，需要多个并行异构加速器，这些加速器彼此交互以共同满足最终用户的需求。 XR 领域特定系统研究的一个主要障碍是（直到我们的工作）还没有覆盖整个 XR 工作流程的开源基准或测试平台来推动此类研究。该项目为 XR 设备开发开源端到端基础设施。它建立在初始研究原型 ILLIXR（伊利诺伊州扩展现实测试平台）的基础上。该系统的设计包含用于完整 XR 工作流程的最先进组件、协调这些组件调度的可扩展运行时以及用于测量性能、功耗和端到端质量的广泛遥测支持。经验指标。该系统具有可扩展性，支持多种操作系统（例如Linux、Android）和异构平台（例如NVIDIA Jetson、Qualcomm Snapdragon等）、传感器（例如相机、IMU等）以及各种XR应用。它为计算堆栈的所有部分提供了新的研究机会，以解决以前不可能实现的端到端 XR 系统创新。系统研究人员受益于使用该基础设施来推动后摩尔特定领域系统的新研究，涉及计算机体系结构、编程语言、编译器、运行时系统以及安全和隐私等领域。端到端基础设施推动了协同设计系统中的新技术的发展，这些系统针对端到端用户体验进行了优化。对于应用而言，XR 涵盖计算机视觉、机器人、图形、信号处理和机器学习等多个子领域。这些领域的算法研究人员可以原型设计并测试针对端到端系统效率进行优化的新算法，而无需担心实现堆栈的其余部分，尤其是 XR 研究人员将能够设计针对端到端系统效率进行优化的系统。最终用户体验。这项工作解决了计算中两个最重要的问题——应对摩尔定律的终结和设计实现沉浸式计算潜力的系统。两者都有可能对整个社会产生巨大影响。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Power, Performance, and Image Quality Tradeoffs in Foveated Rendering

焦点渲染中的功耗、性能和图像质量权衡

• DOI: 10.1109/vr55154.2023.00036
• 发表时间: 2023-03
• 期刊: 2023 IEEE Conference Virtual Reality and 3D User Interfaces (VR
• 作者: Singh, Rahul;Huzaifa, Muhammad;Liu, Jeffrey;Patney, Anjul;Sharif, Hashim;Zhao, Yifan;Adve, Sarita
• 通讯作者: Adve, Sarita

ILLIXR: An Open Testbed to Enable Extended Reality Systems Research

ILLIXR：支持扩展现实系统研究的开放测试平台

• DOI: 10.1109/mm.2022.3161018
• 发表时间: 2022-07
• 期刊: IEEE Micro
• 影响因子: 3.6
• 作者: Huzaifa, Muhammad;Desai, Rishi;Grayson, Samuel;Jiang, Xutao;Jing, Ying;Lee, Jae;Lu, Fang;Pang, Yihan;Ravichandran, Joseph;Sinclair, Finn;et al
• 通讯作者: et al

On-Device CPU Scheduling for Robot Systems

机器人系统的设备上 CPU 调度

• DOI: 10.1109/iros47612.2022.9982085
• 发表时间: 2022-10
• 期刊: 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS
• 作者: Partap, Aditi;Grayson, Samuel;Huzaifa, Muhammad;Adve, Sarita;Godfrey, Brighten;Gupta, Saurabh;Hauser, Kris;Mittal, Radhika
• 通讯作者: Mittal, Radhika