CRII: CHS: Learning Procedural Modeling Programs for Computer Graphics from Examples

CRII：CHS：从示例中学习计算机图形学程序建模程序

基本信息

批准号：
1753684
负责人：
Daniel Ritchie
金额：
$ 17.5万
依托单位：
Brown University
依托单位国家：
美国
项目类别：
Standard Grant
财政年份：
2018
资助国家：
美国
起止时间：
2018-05-01 至 2021-04-30
项目状态：
已结题

来源：
https://www.nsf.gov/awardsearch/showAward?AWD_ID=1753684&HistoricalAwards=false
关键词：
CRII CHS Learning Procedural Modeling

项目摘要

Procedural modeling is used to programmatically generate visual content for instruction, simulation, animation, visual effects, architecture, graphic design, and other applications. An effective procedural model can produce a variety of detailed, visually interesting, and even pleasantly surprising results. Unfortunately, such models are difficult to author, requiring both visual creativity and programming expertise. More people could be empowered to create and use procedural models were it possible to deduce them from examples. The current project will tackle this long-standing open problem in computer graphics by building on the PI's prior work to develop a research program investigating new approaches to learning procedural models from examples by combining probabilistic programs with neural nets; programs are expressive enough to represent a variety of visual content, while neural networks provide flexible learning from data. Project outcomes will help democratize procedural modeling by allowing users to create procedural models with examples rather than by writing code, so that a wider demographic of creative professionals and enthusiasts can participate. All code and data produced will be released as open-source, to allow other researchers and developers to apply and extend the new techniques.Because graphical content is often hierarchical, (probabilistic) grammars are typically used to procedurally model it. However, such content is also characterized by continuous attributes: colors, affine transformations, and so on. While grammars can be extended to support some of these attributes, there are no general-purpose methods for learning such models from examples. Existing approaches either ignore continuous attributes or are specialized to one type of content (e.g., building facades). This research presents a new general-purpose approach for example-based learning of procedural models which generate discrete hierarchical structures with continuous attributes. The key insight is representing a procedural model as a probabilistic program whose control flow and data flow can be governed by neural networks. Like a grammar, such a program can naturally represent (possibly recursive) hierarchical structure. The neural network logic of the program can represent complex functions which generate continuous attributes such as transformations. The model is efficiently learnable with stochastic-gradient-based methods and has the potential to scale from small numbers of examples to large datasets. The initial focus will be on learning procedural models of 3D scene graphs, which are 3D objects composed of a hierarchy of parts. The research will then expand into learning procedural models from large datasets of examples, applying the techniques to domains beyond 3D scene graphs, and leveraging unstructured inputs such as images as examples. Project outcomes will include new mathematical frameworks for learning procedural models from examples, algorithms for efficiently solving the learning problem, and evaluations of the quality of content generated by learned models.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

程序建模用于以编程方式生成用于教学、模拟、动画、视觉效果、建筑、图形设计和其他应用程序的视觉内容。有效的程序模型可以产生各种详细的、视觉上有趣的、甚至令人惊喜的结果。不幸的是，这样的模型很难编写，需要视觉创造力和编程专业知识。如果可以从示例中推导出程序模型，则可以让更多的人有权创建和使用程序模型。当前的项目将在 PI 之前的工作基础上开发一个研究计划，研究通过将概率程序与神经网络相结合来从示例中学习程序模型的新方法，从而解决计算机图形学中这一长期存在的开放问题；程序具有足够的表现力来表示各种视觉内容，而神经网络则提供灵活的数据学习能力。项目成果将允许用户通过示例而不是编写代码来创建程序模型，从而有助于程序建模的民主化，以便更广泛的创意专业人士和爱好者可以参与。生成的所有代码和数据都将作为开源发布，以允许其他研究人员和开发人员应用和扩展新技术。由于图形内容通常是分层的，因此通常使用（概率）语法对其进行程序建模。然而，此类内容也具有连续属性的特征：颜色、仿射变换等。虽然可以扩展语法来支持其中一些属性，但没有通用方法可以从示例中学习此类模型。现有方法要么忽略连续属性，要么专门针对一种类型的内容（例如建筑立面）。这项研究提出了一种新的通用方法，用于基于示例的过程模型学习，该方法生成具有连续属性的离散层次结构。关键的见解是将过程模型表示为概率程序，其控制流和数据流可以由神经网络控制。与语法一样，这样的程序可以自然地表示（可能是递归的）层次结构。程序的神经网络逻辑可以表示生成连续属性（例如变换）的复杂函数。该模型可以通过基于随机梯度的方法进行有效学习，并且具有从小数量示例扩展到大型数据集的潜力。最初的重点将是学习 3D 场景图的程序模型，这些模型是由零件层次结构组成的 3D 对象。然后，该研究将扩展到从大型示例数据集中学习程序模型，将这些技术应用于 3D 场景图以外的领域，并利用图像等非结构化输入作为示例。项目成果将包括用于从示例中学习程序模型的新数学框架、有效解决学习问题的算法以及对学习模型生成的内容质量的评估。该奖项反映了 NSF 的法定使命，并被认为值得通过使用评估来支持基金会的智力价值和更广泛的影响审查标准。

项目成果

期刊论文数量（9）

专著数量（0）

科研奖励数量（0）

会议论文数量（0）

专利数量（0）

Learning Generative Models of 3D Structures

学习 3D 结构的生成模型

DOI：
10.1111/cgf.14020
发表时间：
2020-05-01
期刊：
Computer Graphics Forum
影响因子：
2.5
作者：
S. Chaudhuri;Daniel Ritchie;Kai Xu;Haotong Zhang
通讯作者：
Haotong Zhang

Inferring CAD Modeling Sequences using Zone Graphs

使用区域图推断 CAD 建模序列

DOI：
发表时间：
2021-06
期刊：
IEEE Conference on Computer Vision and Pattern Recognition
影响因子：
0
作者：
Xu, Xianghao;Peng, Wenzhe;Cheng, Chin;Willis, Karl D.D.;Ritchie, Daniel
通讯作者：
Ritchie, Daniel

Example-based Authoring of Procedural Modeling Programs with Structural and Continuous Variability

基于示例的具有结构和连续可变性的程序建模程序的编写

DOI：
10.1111/cgf.13371
发表时间：
2018-05
期刊：
Computer Graphics Forum
影响因子：
2.5
作者：
Ritchie, Daniel;Jobalia, Sarah;Thomas, Anna
通讯作者：
Thomas, Anna

ShapeMOD: macro operation discovery for 3D shape programs

ShapeMOD：3D 形状程序的宏操作发现

DOI：
10.1145/3450626.3459821
发表时间：
2021-08
期刊：
ACM Transactions on Graphics
影响因子：
6.2
作者：
Jones, R. Kenny;Charatan, David;Guerrero, Paul;Mitra, Niloy J.;Ritchie, Daniel
通讯作者：
Ritchie, Daniel

PlanIT: planning and instantiating indoor scenes with relation graph and spatial prior networks

PlanIT：使用关系图和空间先验网络规划和实例化室内场景

DOI：
10.1145/3306346.3322941
发表时间：
2019-07
期刊：
ACM Transactions on Graphics
影响因子：
6.2
作者：
Wang, Kai;Lin, Yu;Weissmann, Ben;Savva, Manolis;Chang, Angel X.;Ritchie, Daniel
通讯作者：
Ritchie, Daniel