RI: Small: Semantic 3D Neural Rendering Field Models that are Accurate, Complete, Flexible, and Scalable

RI：小型：准确、完整、灵活且可扩展的语义 3D 神经渲染场模型

• 批准号: 2312102
• 负责人: Derek Hoiem
• 金额: $ 60万
• 依托单位: University of Illinois at Urbana-Champaign
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2023
• 资助国家: 美国
• 起止时间: 2023-09-15 至 2026-08-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2312102&HistoricalAwards=false
• 关键词: RI; Small; Semantic; 3D; Neural

This project will investigate methods to create, from multiple images, a scene model that enables visualization, synthesis, counting, measurement, and other analysis. The goals of the project are driven by the need for unified geometric (where, what shape, how big) and semantic (what is it, what is it like) scene models, based on the investigators' direct experience in building products for construction management and vehicle safety. So far, computer vision has arguably had its largest impact in internet domains. This project is needed for broader applications involving the physical world, and the potential impact is hard to overstate. Resulting capabilities will lay foundations for real-time modeling, augmented reality, simulation, and robotics applications. The project lays the groundwork for a queryable, editable, and actionable semantic and geometric scene model, a foundational problem in computer vision. Neural rendering fields, vision language models, and diffusion have been impressively demonstrated for separate image synthesis and analysis applications. The project brings these advances together to enable new representations and capabilities for 3D semantic scene modeling. The result is a scalable and robust approach to create, update, query, and edit models of the world inferred from multiple observations. In particular, the project involves three plans of action. The first is to create measurable and meshable 3D scene models that can be efficiently estimated from sparse views and scale to thousands of images. This includes several developments: new efficiently optimizable, compact representations; incorporation of monocular geometry estimates; joint refinement of pose, gain, and other parameters; methods to scale seamlessly to massive scenes and photo sets; and ways to extract high resolution meshes, floor maps, and other common deliverables. The second plan of action is to incorporate semantic information and decoders for counting, measuring, and change detection. This includes encoding semantics in continuous embeddings and creating decoders for visualizing, counting, measuring, and other scene-wide geometric-semantic queries, to enable real-time, flexible mapping and facility assessment. The third plan of action is to extrapolate beyond direct observations and infer and update models as new observations arrive by integrating generative and predictive processes.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该项目将研究从多个图像创建场景模型的方法，该模型能够实现可视化、合成、计数、测量和其他分析。该项目的目标是由统一几何（位置、形状、大小）的需求驱动的。 ）和语义（它是什么，它是什么样的）场景模型，基于研究人员在构建用于施工管理和车辆安全的产品方面的直接经验，迄今为止，计算机视觉在互联网领域产生了最大的影响。需要涉及物理世界的更广泛的应用，并且潜在的影响很难预测由此产生的功能将为实时建模、增强现实、模拟和机器人应用奠定基础，该项目为可查询、可编辑和可操作的语义和几何场景模型（计算机视觉的基本问题）奠定了基础。该项目将这些进步结合在一起，为 3D 语义场景建模提供了新的表示和功能，从而形成了一种可扩展且强大的方法来创建、更新、查询和编辑从多个观察中推断出的世界模型，该项目涉及三个行动计划：第一个是创建可测量和可网格化的 3D 场景模型，可以从稀疏视图和规模到数千个进行有效估计。这包括多项进展：新的可有效优化的紧凑表示；结合单目几何估计；无缝缩放到大量场景和照片集的方法；第二个行动计划是结合语义信息和解码器来进行计数、测量和变化检测，这包括在连续嵌入中编码语义以及创建用于可视化、计数、测量和其他的解码器。场景范围的几何语义查询，以实现实时、灵活的绘图和设施评估。第三个行动计划是在直接观察之外进行推断，并通过集成生成和预测过程来推断和更新模型。通过使用基金会的智力价值和更广泛的影响审查标准进行评估，NSF 的法定使命被认为值得支持。