Collaborative Research: RI: Medium: Learning Compositional Implicit Representations for 3D Scene Understanding

合作研究：RI：媒介：学习 3D 场景理解的组合隐式表示

• 批准号: 2211258
• 负责人: Jiajun Wu
• 金额: $ 40万
• 依托单位: Stanford University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-10-01 至 2026-09-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2211258&HistoricalAwards=false
• 关键词: Collaborative; Research; RI; Medium; Learning

Scene understanding systems take visual inputs, like images or videos, and reconstruct and interpret the underlying scene in terms of 3D structure, objects like cars and people, and other scene properties. Such systems are crucial in applications in computer vision, computer graphics, and robotics, including in self-driving cars. To represent the 3D world as observed from the input imagery, such systems use mathematical models, and in recent years neural networks have been very popular as the models used in such systems, due to their expressiveness and ability to capture fine details. However, current neural network-based scene representations are only good at modeling the specific conditions under which a scene was observed, and cannot generalize to new scenarios, limiting their use in many applications. For example, if a self-driving car is trained to model scenes using only images from sunny days, the car’s perception system might break down on rainy or snowy days. This project aims to introduce new scene modeling techniques that will enable machines to perceive and reconstruct 3D scenes in a more generalizable way. The investigators will integrate findings from this research into course development and student advising, and partner with educational and non-profit organizations to teach AI, vision, and graphics to underrepresented students. In this project, investigators will explore new methods that will make representations capable of encoding more structure (e.g., light field) and root them in physics. Designing such representations requires knowledge from AI, computer vision, and computer graphics. The key innovations include a new class of scene representations that aims to bridge the ability of implicit neural representations to capture scene details with that of physical representations to model scene structure; new methods that infer the representation from raw images and videos with new parametrizations to enable data-efficient, self-supervised learning; and new methods that leverage the representation for downstream computer vision and graphics tasks, such as interactive design and scene synthesis.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

场景理解系统采用图像或视频等视觉输入，并根据 3D 结构、汽车和人等对象以及其他场景属性重建和解释底层场景，此类系统在计算机视觉、计算机图形学和其他领域的应用中至关重要。为了表示从输入图像中观察到的 3D 世界，此类系统使用数学模型，近年来神经网络由于其表现力和性能而作为此类系统中使用的模型非常流行。捕捉精细细节的能力。然而，当前基于神经网络的场景表示仅擅长对观察场景的特定条件进行建模，而无法推广到新场景，这限制了它们在许多应用中的使用，例如，如果自动驾驶汽车被训练为：仅使用晴天的图像来建模场景，汽车的感知系统可能会在雨天或下雪天崩溃。该项目旨在引入新的场景建模技术，使机器能够以更通用的方式感知和重建 3D 场景。将这项研究的结果整合到课程中开发和学生咨询，并与教育和非营利组织合作，向代表性不足的学生教授人工智能、视觉和图形。在这个项目中，研究人员将探索新的方法，使表示能够编码更多的结构（例如，光场）。设计这种表示需要人工智能、计算机视觉和计算机图形学的知识，其中包括一类新的场景表示，旨在将隐式神经表示捕获场景细节的能力与物理表示的能力联系起来。到模型场景结构；通过新的参数化从原始图像和视频中推断表示的新方法，以实现数据高效的自我监督学习；以及利用下游计算机视觉和图形任务的表示的新方法，例如交互式设计和场景合成。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

IKEA-Manual: Seeing Shape Assembly Step by Step

宜家手册：逐步了解形状组装

• 发表时间: 2022-01
• 期刊: Advances in Neural Information Processing Systems Datasets and Benchmarks Track
• 作者: Wang, Ruocheng;Zhang, Yunzhi;Mao, Jiayuan;Zhang, Ran;Cheng, Chin;Wu, Jiajun
• 通讯作者: Wu, Jiajun

NS3D: Neuro-Symbolic Grounding of 3D Objects and Relations

NS3D：3D 对象和关系的神经符号基础

• DOI: 10.1109/cvpr52729.2023.00257
• 发表时间: 2023-03-23
• 期刊: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
• 作者: Joy Hsu;Jiayuan Mao;Jiajun Wu
• 通讯作者: Jiajun Wu

See, Hear, and Feel: Smart Sensory Fusion for Robotic Manipulation

看、听、感觉：机器人操作的智能感官融合

• DOI: 10.48550/arxiv.2212.03858
• 发表时间: 2022-12-07
• 作者: Hao Li;Yizhi Zhang;Junzhe Zhu;Shaoxiong Wang;Michelle A. Lee;Huazhe Xu;E. Adelson;Li Fei;Ruohan Gao;Jiajun Wu
• 通讯作者: Jiajun Wu

An Extensible Multi-modal Multi-task Object Dataset with Materials

具有材料的可扩展多模式多任务对象数据集

• 发表时间: 2023-01
• 期刊: International Conference on Learning Representations
• 作者: Standley, Trevor;Gao, Ruohan;Chen, Dawn;Wu, Jiajun;Savarese, Silvio
• 通讯作者: Savarese, Silvio

Multi-Object Manipulation via Object-Centric Neural Scattering Functions

通过以对象为中心的神经散射函数进行多对象操作

• 发表时间: 2023-06
• 期刊: IEEE/CVF Conference on Computer Vision and Pattern Recognition
• 作者: Tian, Stephen;Cai, Yancheng;Yu, Hong;Zakharov, Sergey;Liu, Katherine;Gaidon, Adrien;Li, Yunzhu;Wu, Jiajun
• 通讯作者: Wu, Jiajun