CAREER: Learning Neurosymbolic 3D Models

职业：学习神经符号 3D 模型

基本信息

批准号：
1941808
负责人：
Daniel Ritchie
金额：
$ 55万
依托单位：
Brown University
依托单位国家：
美国
项目类别：
Continuing Grant
财政年份：
2020
资助国家：
美国
起止时间：
2020-04-01 至 2025-03-31
项目状态：
未结题

来源：
https://www.nsf.gov/awardsearch/showAward?AWD_ID=1941808&HistoricalAwards=false
关键词：
CAREER Learning Neurosymbolic 3D Models

项目摘要

High-quality 3D models are increasingly in demand, driven by numerous industries and by the need for synthetic training data to scale up autonomous vision systems. But creating such models is a laborious and time-consuming process requiring years of training, so current practice will be insufficient to satisfy future data demands. One way forward is through generative models of 3D objects, that is to have machines learn to synthesize high-quality objects, a nice vision which has yet to be realized. Existing 3D generative models fall into one of two broad categories, each with limitations. Symbolic generative models such as shape grammars can enable non-experts to generate high-quality geometry but have severely limited expressiveness, while neural generative models are flexible and can in theory learn to express any shape but they are inscrutable and produce flawed geometry. This project will explore a new class of generative shape model that combines the best of both worlds: neuro-symbolic 3D models. The main insight is to use a symbolic program to model the logical part structure of a 3D object (e.g., the legs of a chair are connected to its seat), and then to use neural networks to refine this structure into high-quality geometry. Such a representation supports synthesis of new objects, reconstruction of objects from real-world sensor input, and high-level editing of object structure and geometry. It also supports modeling of higher-order object properties, including kinematics and physics. To enable massive-scale generation of synthetic 3D training data for computer vision and robotics, a neuro-symbolic version of the widely used ShapeNet dataset will be implemented and released. To help democratize 3D content creation, the project will collaborate with Unity Technologies to integrate neuro-symbolic 3D models into their popular 3D graphics engine. Project outcomes will also include an open-source, pedagogical deep learning framework to educate a new generation of researchers with the multidisciplinary skillset needed for neuro-symbolic modeling, in concert with activities (e.g., piloting new integrated visual computing curricula via summer schools and hosting visiting student researchers from historically under-represented groups) designed to improve student mastery of neural network fundamentals.The recognition-by-components theory of vision posits that people recognize objects by first understanding their fundamental parts and then using a secondary process to handle objects that are not distinguishable by these parts alone. Neuro-symbolic 3D models operationalize this theory for object synthesis via two algorithmic phases. The first phase is a new procedural representation called a hierarchical part graph program that is a human-readable computer program which, when executed, constructs a graph of connected object parts at multiple levels of detail wherein the bottom level of detail consists of parametric primitives such as cuboids and cylinders. While suggestive of shape, these graphs do not capture the full variety of geometry found in real-world objects. Thus, the second phase of the model is a new neural adaptive subdivision procedure which converts the low-fidelity parts into high-fidelity surface geometry. This decomposition is a natural fit for the common case of human-made objects, but it can also be extended to organic objects. The hypothesis is that this approach to 3D object generation will be able to efficiently synthesize and reconstruct a variety of high-quality objects in a unified, easily-editable representation.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

在众多行业以及扩展自主视觉系统所需的合成训练数据的推动下，高质量 3D 模型的需求日益增长。但创建此类模型是一个费力且耗时的过程，需要多年的训练，因此当前的实践不足以满足未来的数据需求。前进的一种方法是通过 3D 对象的生成模型，即让机器学习合成高质量的对象，这是一个尚未实现的美好愿景。现有的 3D 生成模型属于两大类之一，每一类都有局限性。形状语法等符号生成模型可以使非专家生成高质量的几何图形，但表达能力严重有限，而神经生成模型很灵活，理论上可以学习表达任何形状，但它们难以理解并产生有缺陷的几何图形。该项目将探索一类新的生成形状模型，它结合了两个领域的优点：神经符号 3D 模型。主要见解是使用符号程序对 3D 对象的逻辑部分结构进行建模（例如，椅子的腿与其座椅相连），然后使用神经网络将该结构细化为高质量的几何形状。这种表示支持新对象的合成、根据现实世界传感器输入重建对象以及对象结构和几何形状的高级编辑。它还支持高阶对象属性的建模，包括运动学和物理学。为了能够大规模生成计算机视觉和机器人技术的合成 3D 训练数据，将实现并发布广泛使用的 ShapeNet 数据集的神经符号版本。为了帮助实现 3D 内容创建的民主化，该项目将与 Unity Technologies 合作，将神经符号 3D 模型集成到其流行的 3D 图形引擎中。项目成果还将包括一个开源的教学深度学习框架，以教育新一代研究人员具备神经符号建模所需的多学科技能，并与活动相结合（例如，通过暑期学校试行新的综合视觉计算课程并主办来自历史上代表性不足群体的访问学生研究人员）旨在提高学生对神经网络基础知识的掌握。视觉成分识别理论假设人们通过首先理解其基本部分然后使用辅助过程来识别物体处理仅靠这些部分无法区分的对象。神经符号 3D 模型通过两个算法阶段将这一理论应用于对象合成。第一阶段是一种称为分层零件图程序的新程序表示，它是一种人类可读的计算机程序，在执行时会在多个细节级别构建连接的对象零件的图，其中底层细节由参数基元组成，例如如长方体和圆柱体。虽然暗示了形状，但这些图表并没有捕捉到现实世界物体中发现的全部几何形状。因此，模型的第二阶段是新的神经自适应细分过程，它将低保真度零件转换为高保真度表面几何形状。这种分解非常适合人造物体的常见情况，但它也可以扩展到有机物体。假设这种 3D 对象生成方法将能够以统一的、易于编辑的表示形式有效地合成和重建各种高质量对象。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准。

项目成果

期刊论文数量（10）

专著数量（0）

科研奖励数量（0）

会议论文数量（0）

专利数量（0）

Unsupervised Kinematic Motion Detection for Part-segmented 3D Shape Collections

用于部分分段 3D 形状集合的无监督运动检测

DOI：
10.1145/3528233.3530742
发表时间：
2022-06-17
期刊：
ACM SIGGRAPH 2022 Conference Proceedings
影响因子：
0
作者：
Xianghao Xu;Yifan Ruan;Srinath Sridhar;Daniel Ritchie
通讯作者：
Daniel Ritchie

PLAD: Learning to Infer Shape Programs with Pseudo-Labels and Approximate Distributions

PLAD：学习用伪标签和近似分布推断形状程序

DOI：
10.1109/cvpr52688.2022.00964
发表时间：
2022-06
期刊：
IEE CVPR 2022
影响因子：
0
作者：
Jones, R. Kenny;Walke, Homer;Ritchie, Daniel
通讯作者：
Ritchie, Daniel

ShapeMOD: macro operation discovery for 3D shape programs

ShapeMOD：3D 形状程序的宏操作发现

DOI：
10.1145/3450626.3459821
发表时间：
2021-08
期刊：
ACM Transactions on Graphics
影响因子：
6.2
作者：
Jones, R. Kenny;Charatan, David;Guerrero, Paul;Mitra, Niloy J.;Ritchie, Daniel
通讯作者：
Ritchie, Daniel

ShapeAssembly: learning to generate programs for 3D shape structure synthesis

ShapeAssembly：学习生成 3D 形状结构合成程序

DOI：
10.1145/3414685.3417812
发表时间：
2020-11
期刊：
ACM Transactions on Graphics
影响因子：
6.2
作者：
Jones, R. Kenny;Barton, Theresa;Xu, Xianghao;Wang, Kai;Jiang, Ellen;Guerrero, Paul;Mitra, Niloy J.;Ritchie, Daniel
通讯作者：
Ritchie, Daniel

ShapeCoder: Discovering Abstractions for Visual Programs from Unstructured Primitives

ShapeCoder：从非结构化基元中发现视觉程序的抽象