Collaborative Research: RI: Medium: Learning Compositional Implicit Representations for 3D Scene Understanding

合作研究：RI：媒介：学习 3D 场景理解的组合隐式表示

基本信息

批准号：
2211258
负责人：
Jiajun Wu
金额：
$ 40万
依托单位：
Stanford University
依托单位国家：
美国
项目类别：
Standard Grant
财政年份：
2022
资助国家：
美国
起止时间：
2022-10-01 至 2026-09-30
项目状态：
未结题

来源：
https://www.nsf.gov/awardsearch/showAward?AWD_ID=2211258&HistoricalAwards=false
关键词：
Collaborative Research RI Medium Learning

项目摘要

Scene understanding systems take visual inputs, like images or videos, and reconstruct and interpret the underlying scene in terms of 3D structure, objects like cars and people, and other scene properties. Such systems are crucial in applications in computer vision, computer graphics, and robotics, including in self-driving cars. To represent the 3D world as observed from the input imagery, such systems use mathematical models, and in recent years neural networks have been very popular as the models used in such systems, due to their expressiveness and ability to capture fine details. However, current neural network-based scene representations are only good at modeling the specific conditions under which a scene was observed, and cannot generalize to new scenarios, limiting their use in many applications. For example, if a self-driving car is trained to model scenes using only images from sunny days, the car’s perception system might break down on rainy or snowy days. This project aims to introduce new scene modeling techniques that will enable machines to perceive and reconstruct 3D scenes in a more generalizable way. The investigators will integrate findings from this research into course development and student advising, and partner with educational and non-profit organizations to teach AI, vision, and graphics to underrepresented students. In this project, investigators will explore new methods that will make representations capable of encoding more structure (e.g., light field) and root them in physics. Designing such representations requires knowledge from AI, computer vision, and computer graphics. The key innovations include a new class of scene representations that aims to bridge the ability of implicit neural representations to capture scene details with that of physical representations to model scene structure; new methods that infer the representation from raw images and videos with new parametrizations to enable data-efficient, self-supervised learning; and new methods that leverage the representation for downstream computer vision and graphics tasks, such as interactive design and scene synthesis.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

Scene System进行视觉输入，例如图像或视频，并根据3D结构来重建和解释基础场景，包括自动驾驶的赛车，以代表3D世界，因为从输入图像中观察到的年度神经网络在此类系统中使用的Tels均受欢迎。基于场景的代表仅擅长于观察到一个场景的特定条件，并且无法推广到新的场景，例如在许多应用程序中使用它们。汽车的感知系统可能会在下雨天或下雪的日子里崩溃。组织和非营利组织教授AI，并为登记的学生提供图形，这将使能够编码更多的结构（例如，灯场）并扎根于物理学。图形。关键创新包括一个新的场景表示，旨在用thysical catentations桥接神经捕获的能力，以模拟结构通过使用基金会的知识分子优点和更广泛的影响标准评估，学习了下游计算机视觉和图形任务的代表。

项目成果

期刊论文数量（19）

专著数量（0）

科研奖励数量（0）

会议论文数量（0）

专利数量（0）

Dynamic-Resolution Model Learning for Object Pile Manipulation

DOI：
10.48550/arxiv.2306.16700
发表时间：
2023-06
期刊：
ArXiv
影响因子：
0
作者：
Yixuan Wang;Yunzhu Li;K. Driggs-Campbell;Li Fei-Fei-Li-Fei-Fei-48004138;Jiajun Wu
通讯作者：
Yixuan Wang;Yunzhu Li;K. Driggs-Campbell;Li Fei-Fei-Li-Fei-Fei-48004138;Jiajun Wu

DisCo: Improving Compositional Generalization in Visual Reasoning through Distribution Coverage

DisCo：通过分布覆盖提高视觉推理中的构图概括

DOI：
发表时间：
2023
期刊：
Transactions on machine learning research
影响因子：
0
作者：
Hsu, Joy;Mao, Jiayuan;Wu, Jiajun
通讯作者：
Wu, Jiajun

Learning Vortex Dynamics for Fluid Inference and Prediction

DOI：
10.48550/arxiv.2301.11494
发表时间：
2023-01
期刊：
ArXiv
影响因子：
0
作者：
Yitong Deng;Hong-Xing Yu;Jiajun Wu;Bo Zhu
通讯作者：
Yitong Deng;Hong-Xing Yu;Jiajun Wu;Bo Zhu

Multi-Object Manipulation via Object-Centric Neural Scattering Functions

通过以对象为中心的神经散射函数进行多对象操作

DOI：
发表时间：
2023
期刊：
IEEE/CVF Conference on Computer Vision and Pattern Recognition
影响因子：
0
作者：
Tian, Stephen;Cai, Yancheng;Yu, Hong-Xing;Zakharov, Sergey;Liu, Katherine;Gaidon, Adrien;Li, Yunzhu;Wu, Jiajun
通讯作者：
Wu, Jiajun

3D Neural Field Generation Using Triplane Diffusion