Collaborative Research: HCC: Medium: Differentiable Rendering for Computer Graphics

合作研究：HCC：媒介：计算机图形学的可微渲染

• 批准号: 2105806
• 负责人: Ravi Ramamoorthi
• 金额: $ 80万
• 依托单位: University of California-San Diego
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-07-01 至 2025-06-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2105806&HistoricalAwards=false
• 关键词: Collaborative; Research; HCC; Medium; Differentiable

Creating realistic images in computer graphics has historically relied on accurately computing the values at each point or pixel in the image based on physically accurate simulation of lighting in the scene, but recently it has become clear that simply computing image values is not adequate. One needs to also be able to understand how these values change with changes in the environment, for example as the sun moves across the sky, a door is opened letting light into the scene, or the material properties of an object are gradually changed from velvet to metallic. Mathematically, this involves computing the derivatives of the image to determine how it changes with respect to the input parameters. This research will create a class of differentiable renderers that compute both images and their derivatives. Project outcomes will have broad impact because the computation of derivatives is increasingly central to many areas of computer graphics, computer vision, robotics and machine learning, with potential benefit to applications as diverse as perception control in self-driving cars and robots, optimization of indoor lighting for architecture, fabrication of 3D objects with a desired appearance, statistics and epidemiology. Additional impact will derive from the fact that the PIs are educators committed to broadening participation in computing who participate in early research scholars programs and will develop new online courses in rendering.Computing the derivatives or gradients of general light transport involves tackling fundamental challenges of differential calculus, Monte Carlo integration, signal processing, automatic differentiation, and metaprogramming systems. One challenge is in handling discontinuities of various forms, which lead to Dirac delta terms that require careful and analytic treatment that cannot be provided by traditional automatic differentiation. Even for the smooth variation, computing gradients involves a large number of intermediate variables that necessitate tradeoffs across bias, variance, compute and memory. Moreover, full generality requires differentiable rendering in new representations such as implicit surfaces and procedural materials, as well as new problem domains such as transient rendering for non-line-of-sight imaging and geometrical diffraction for acoustics. One also needs to effectively apply the gradients for optimization in inverse problems. This project will develop a broad transformative agenda, seeking to enable differentiable renderers to efficiently reconstruct billions of varied primitives from millions of pixels under general and diverse light transport situations. The research plan consists of four interconnected components involving computational foundations and efficient algorithms for solving visibility gradients including: analytic and area sampling methods; a unified system for exploring computational and memory tradeoffs in differentiable rendering algorithms; generalizations to new physical phenomena such as transient rendering and geometrical diffraction; and advances in inverse problems and deep learning including new approaches to continuous optimization involving Euler-Lagrange equations.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

在计算机图形学中创建逼真的图像历来依赖于基于场景中照明的物理精确模拟来准确计算图像中每个点或像素的值，但最近很明显，简单地计算图像值是不够的。人们还需要能够理解这些值如何随着环境的变化而变化，例如，当太阳在天空中移动时，一扇门打开让光线进入场景，或者物体的材质属性逐渐从天鹅绒改变至金属。 从数学上讲，这涉及计算图像的导数以确定其相对于输入参数的变化方式。 这项研究将创建一类可微分渲染器，用于计算图像及其导数。 项目成果将产生广泛的影响，因为导数的计算在计算机图形学、计算机视觉、机器人和机器学习的许多领域中越来越重要，对自动驾驶汽车和机器人的感知控制、室内优化等多种应用都有潜在的好处建筑照明、具有所需外观的 3D 物体的制造、统计和流行病学。 额外的影响将来自这样一个事实：PI 是致力于扩大计算参与的教育工作者，他们参与早期研究学者项目，并将开发新的渲染在线课程。计算一般光传输的导数或梯度涉及解决微分学的基本挑战、蒙特卡罗积分、信号处理、自动微分和元编程系统。 一项挑战是处理各种形式的不连续性，这导致狄拉克德尔塔项需要仔细和分析性的处理，而传统的自动微分无法提供这种处理。 即使对于平滑变化，计算梯度也涉及大量中间变量，需要在偏差、方差、计算和内存之间进行权衡。 此外，完全的通用性需要新的表示形式（例如隐式表面和程序材质）以及新的问题领域（例如非视距成像的瞬态渲染和声学的几何衍射）中的可微分渲染。 人们还需要有效地应用梯度来优化反问题。 该项目将制定一个广泛的变革议程，寻求使可微分渲染器能够在一般和不同的光传输情况下从数百万像素中有效地重建数十亿种不同的图元。 该研究计划由四个相互关联的部分组成，涉及解决可见度梯度的计算基础和有效算法，包括：分析和区域采样方法；用于探索可微渲染算法中的计算和内存权衡的统一系统；对新物理现象的概括，例如瞬态渲染和几何衍射；该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Warped-Area Reparameterization of Differential Path Integrals

微分路径积分的扭曲面积重新参数化

• DOI: 10.1145/3618330
• 发表时间: 2023-12
• 期刊: ACM Transactions on Graphics
• 影响因子: 6.2
• 作者: Xu, Peiyu;Bangaru, Sai;Li, Tzu;Zhao, Shuang
• 通讯作者: Zhao, Shuang

Importance Sampling BRDF Derivatives

重要性采样 BRDF 导数

• DOI: 10.1145/3648611
• 发表时间: 2023-04-08
• 期刊: ACM Transactions on Graphics
• 影响因子: 6.2
• 作者: Yash Belhe;Bing Xu;Sai Praveen Bangaru;R. Ramamoorthi;Tzu
• 通讯作者: Tzu

Designing Perceptual Puzzles by Differentiating Probabilistic Programs

通过区分概率程序来设计感知谜题

• DOI: 10.1145/3528233.3530715
• 发表时间: 2022-08
• 期刊: ACM SIGGRAPH Conference Proceedings
• 作者: Chandra, Kartik;Li, Tzu;Tenenbaum, Joshua;Ragan
• 通讯作者: Ragan

Parameter-space ReSTIR for Differentiable and Inverse Rendering

用于可微分和逆渲染的参数空间 ReSTIR

• DOI: 10.1145/3588432.3591512
• 发表时间: 2023-07
• 期刊: ACM
• 作者: Chang, Wesley;Sivaram, Venkataram;Nowrouzezahrai, Derek;Hachisuka, Toshiya;Ramamoorthi, Ravi;Li, Tzu
• 通讯作者: Li, Tzu

Differentiable time-gated rendering

可微时间选通渲染

• DOI: 10.1145/3478513.3480489
• 发表时间: 2021-12
• 期刊: ACM Transactions on Graphics
• 影响因子: 6.2
• 作者: Wu, Lifan;Cai, Guangyan;Ramamoorthi, Ravi;Zhao, Shuang
• 通讯作者: Zhao, Shuang