Collaborative Research: RI: Medium: Learning Compositional Implicit Representations for 3D Scene Understanding

合作研究：RI：媒介：学习 3D 场景理解的组合隐式表示

• 批准号: 2211259
• 负责人: Noah Snavely
• 金额: $ 40万
• 依托单位: Cornell University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-10-01 至 2026-09-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2211259&HistoricalAwards=false
• 关键词: Collaborative; Research; RI; Medium; Learning

Scene understanding systems take visual inputs, like images or videos, and reconstruct and interpret the underlying scene in terms of 3D structure, objects like cars and people, and other scene properties. Such systems are crucial in applications in computer vision, computer graphics, and robotics, including in self-driving cars. To represent the 3D world as observed from the input imagery, such systems use mathematical models, and in recent years neural networks have been very popular as the models used in such systems, due to their expressiveness and ability to capture fine details. However, current neural network-based scene representations are only good at modeling the specific conditions under which a scene was observed, and cannot generalize to new scenarios, limiting their use in many applications. For example, if a self-driving car is trained to model scenes using only images from sunny days, the car’s perception system might break down on rainy or snowy days. This project aims to introduce new scene modeling techniques that will enable machines to perceive and reconstruct 3D scenes in a more generalizable way. The investigators will integrate findings from this research into course development and student advising, and partner with educational and non-profit organizations to teach AI, vision, and graphics to underrepresented students. In this project, investigators will explore new methods that will make representations capable of encoding more structure (e.g., light field) and root them in physics. Designing such representations requires knowledge from AI, computer vision, and computer graphics. The key innovations include a new class of scene representations that aims to bridge the ability of implicit neural representations to capture scene details with that of physical representations to model scene structure; new methods that infer the representation from raw images and videos with new parametrizations to enable data-efficient, self-supervised learning; and new methods that leverage the representation for downstream computer vision and graphics tasks, such as interactive design and scene synthesis.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

Scene System进行视觉输入，例如图像或视频，并根据3D结构来重建和解释基础场景，包括自动驾驶的赛车，以代表3D世界，因为从输入图像中观察到的年度神经网络在此类系统中使用的Tels均受欢迎。基于场景的代表仅擅长于观察到一个场景的特定条件，并且无法推广到新的场景，例如在许多应用程序中使用它们。汽车的感知系统可能会在下雨天或下雪的日子里崩溃。组织和非营利组织教授AI，并为登记的学生提供图形，这将使能够编码更多的结构（例如，灯场）并扎根于物理学。图形。关键创新包括一个新的场景表示，旨在用thysical catentations桥接神经捕获的能力，以模拟结构通过使用基金会的知识分子优点和更广泛的影响标准评估，学习了下游计算机视觉和图形任务的代表。