Graphics Pipelines for Next Generation Mixed Reality Systems

下一代混合现实系统的图形管道

• 批准号: EP/T01346X/1
• 负责人: Anthony Steed
• 金额: $ 106.49万
• 依托单位: University College London
• 依托单位国家: 英国
• 项目类别: Research Grant
• 财政年份: 2020
• 资助国家: 英国
• 起止时间: 2020 至 无数据
• 项目状态: 未结题
• 来源: https://gtr.ukri.org/projects?ref=EP%2FT01346X%2F1
• 关键词: Graphics; Pipelines; Next; Generation; Mixed

In the past twenty years display technology has moved on considerablywith improvements such as higher resolution, faster frame rates andhigh-dynamic range colour. In the same period graphics processingunits (GPUs or graphics cards) have become significantly faster withbroader functionality. However, we argue that current implementationsof the traditional graphics pipeline, which is based on therasterisation and z-buffering, are unsuited to emerging displays.In particular, the types of near-eye displays used for augmentedreality and virtual reality provide new challenges. Since theircontexts of use are very different, it is not even clear whatproperties are ideal. Should the displays be multi-focal orvari-focal? How fast do display update rates need to be to supportregistered augmented reality systems? How do we exploit the propertiesof the human visual system to render more efficiently?The traditional computer graphics pipeline puts emphasis on generatingfull images at the display size and rate, where that display might be1920x1080 at 60Hz or higher. For a near-eye display, it is clear thatsuch a pipeline is nowhere near suitable: already one consumer HMD isdemanding 2480x1080 at 90Hz, roughly double the bandwidth that a commondesktop display requires. At these rates, there will still be problemswith visual acuity and latency (e.g. there is an inherent 11ms displaylag). Future HMDs are touted with resolutions of 8K pixels across, butthis will require a very significant increase in graphics compute andthus power consumption. Not only is this expensive, it could limit theform factor of the device and thus user-acceptance of the technology.There is very exciting work going on in display technologies at themoment. For example, Varjo (a project partner) are building a HMD thathas a moving display insert that tracks the eye; Facebook have built ademonstration display that uses a focal surface to generate multiplefocal depths in the same image.In comparison, the graphics pipeline is taken mostly as a given.Recently, the proposers, along with a small group of colleagues in thefield, have started to the challenge the status quo. We don't proposeto ditch the highly optimised compute units in graphics cards, butrather to study frameworks within they can be exploited more readily.We believe that by reformulating the graphics pipeline and payingattention to the very specific needs of near-eye displays, we canradically reduce the power required from GPUs, and thus make near-eyedisplay more usable.We will focus on three connected challenges that we have labelled thelatency, redundancy and bandwidth challenges. First, we will targetextremely low latency displays. We will develop systems that achieve>1000 fps visual output, with latency under 1ms and study how theseimpact visual response. Second, we will explore stronger decoupling offrame-based rendering from display. We note that in near-eye displaysmost pixels are wasted, and thus we target novel spatial and temporalalgorithms that reduce redundancy. Third, to exploit redundancy moregenerally, we need to use it to reduce bandwidth between graphics cardand display. Taking inspiration from the concept of surface lightfields, our concept of ambient fields will render to buffers that areexpected to be valid for re-rendering to the user for 10-100ms. In summary, we believe that the current graphics pipeline and itsassociated implementation in GPUs is unsuited to drive near-eyedisplays. We want near-eye displays to be low-cost, power efficientand highly acceptable to users. To achieve this, we propose newalgorithms that can use GPU capabilities more effectively. We targetreducing redundant compute to enable lower latency and lower bandwidthrequirements through the graphics pipeline.

在过去的二十年中，展示技术的改进已取得了很大的改进，例如更高的分辨率，更快的帧速率和高动力范围的颜色。在同一时期，图形处理（GPU或图形卡）的功能已变得更快。但是，我们认为，基于治疗和Z挡板的传统图形管道的当前实现不适合新兴显示。尤其是，用于增强现实性和虚拟现实的近眼显示的类型提供了新的挑战。由于它们的使用范围非常不同，因此甚至不清楚Whatproperties是理想的。显示器是否应该是多焦点的Orvari-cocal？显示更新率需要多快才能支持增强现实系统？我们如何利用人类视觉系统的属性更有效地渲染？传统的计算机图形管道将重点放在以显示大小和速率的情况下生成Fultingl图像，在60hz或更高时，该显示的显示可能为1920x1080。对于近眼显示器，很明显，这是一条管道差不多：已经在90Hz处使用了一个消费者HMD ISDEMEND 2480x1080，大约是Commondesktop显示器所需的带宽两倍。以这些速度，仍然会有QualiseSwith视力和延迟（例如，有一个固有的11ms displayLag）。未来的HMD吹捧着跨越8K像素的分辨率，但这种图形计算和功耗将非常显着增加。这种昂贵不仅可以限制设备的形式，因此可以对技术的吸收来进行，因此在展示技术中正在进行非常令人兴奋的工作。例如，Varjo（一个项目合作伙伴）正在建立一个HMD，该HMD可以插入跟踪眼睛的移动显示器。 Facebook已构建了Adxontration显示屏，该显示器使用焦点表面在同一图像中生成多个焦点深度。在比较中，图形管道主要是作为给定的。实际上，提议者以及菲尔德的一小部分同事开始应对现状挑战。我们不建议抛弃图形卡中高度优化的计算单位，但可以更容易地利用它们的框架。我们相信，通过重新重新加定图形管道并向近眼显示器的特定需求进行重新计算，我们可以更加努力地进行挑战。冗余和带宽挑战。首先，我们将针对低潜伏期显示。我们将开发可实现> 1000 fps视觉输出的系统，延迟在1ms下，并研究这些影响视觉响应的方式。其次，我们将探索与Display的更强大的基于Offrame的渲染。我们注意到，在近眼中显示大多数像素，因此我们瞄准了新的空间和时间词素，以降低冗余。第三，为了大概括地利用冗余，我们需要使用它来减少图形纸牌显示器之间的带宽。从表面灯场的概念中汲取灵感，我们的环境场概念将渲染到预测的缓冲区，这些缓冲区被指向对用户重新呈现10-100ms有效。总而言之，我们认为，当前的图形管道及其在GPU中与其相关的实施不适合驱动近眼播放。我们希望近眼显示器具有低成本，有效的功率，并且用户高度可接受。为了实现这一目标，我们提出了可以更有效地使用GPU功能的新Algorithm。我们针对冗余计算，以通过图形管道降低潜伏期和较低的带宽重复。

项目成果

Ubiq-exp: A toolkit to build and run remote and distributed mixed reality experiments

Ubiq-exp：用于构建和运行远程分布式混合现实实验的工具包

• DOI: 10.3389/frvir.2022.912078
• 发表时间: 2022
• 期刊: Frontiers in Virtual Reality
• 作者: Steed A

Exploring Server-Centric Scalability for Social VR

探索以服务器为中心的社交 VR 可扩展性

• DOI: 10.1109/ds-rt58998.2023.00016
• 发表时间: 2023
• 作者: Friston S

Metameric Inpainting for Image Warping.

用于图像变形的同色异谱修复。

• DOI: 10.1109/tvcg.2022.3216712
• 发表时间: 2023
• 期刊: IEEE transactions on visualization and computer graphics
• 影响因子: 5.2
• 作者: Dos Anjos RK

Metameric Varifocal Holograms

• DOI: 10.1109/vr51125.2022.00096
• 发表时间: 2021-10
• 期刊: 2022 IEEE Conference on Virtual Reality and 3D User Interfaces (VR)
• 作者: David R. Walton;Koray Kavaklı;R. K. D. Anjos;David Swapp;T. Weyrich;H. Urey;A. Steed;Tobias Ritschel;K. Akşit

Beyond Flicker, Beyond Blur: View-coherent Metameric Light Fields for Foveated Display

超越闪烁，超越模糊：用于注视点显示的视图相干同色异谱光场

• DOI: 10.1109/vrw55335.2022.00152
• 发表时间: 2022
• 作者: Kohli P