RI: NSF-BSF: Small: Reconstructing Shape, Lighting and Reflectance Properties of Indoor Scenes from Video

RI：NSF-BSF：小型：从视频重建室内场景的形状、照明和反射率属性

• 批准号: 1910132
• 负责人: David Jacobs
• 金额: $ 49.33万
• 依托单位: University of Maryland, College Park
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-08-15 至 2023-07-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1910132&HistoricalAwards=false
• 关键词: RI; NSF; BSF; Small; Reconstructing

The goal of this project is to allow someone to use a video of an indoor scene, such as a room in a house, to reconstruct the scene. The reconstruction will contain all scene properties needed to enable images of the scene to be rendered from new viewpoints, with new lighting. To do this it is necessary to recover the depth and shape of every point in the scene. It is also necessary to describe how each point in the scene reflects light, and to recover the lighting in the scene. This technology will enable a wide range of applications in virtual and augmented reality. For example, it will assist in generating images that show what a room would look like if the furniture were changed, or to allow augmented reality teleconferencing, in which physically separated speakers appear to chat in the same room. The project will educate several graduate students. The investigators will also develop related undergraduate research projects, and projects for an AI summer camp for high school students.One reason such inverse rendering is challenging is the lack of labeled real data. The investigators will address this deficit by using multi-view input for training and solving jointly for all components of the scene. This will allow them to use a self-supervised multi-view photometric reconstruction loss, enforcing consistency across views. They will combine this real data with a large new dataset of highly realistic synthetic data. They will also develop a unified architecture that recovers pose between views and combines geometric and photometric information. This will allow the system to predict appearance changes across viewpoints and to determine whether measured appearance is consistent with recovered lighting and reflectance properties. They will also develop novel representations of lighting in scenes, and of the way that objects reflect this light that are compact and effective.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该项目的目的是允许某人使用室内场景的视频，例如房屋中的房间，以重建场景。 重建将包含所有场景属性，以通过新的照明从新的角度呈现场景的图像。 为此，有必要恢复场景中每个点的深度和形状。 还必须描述场景中的每个点如何反映光线，并恢复场景中的照明。 这项技术将在虚拟和增强现实中实现广泛的应用。 例如，它将有助于生成图像，这些图像显示了如果更改家具或允许增强现实的电信，则可以在同一房间里聊天。 该项目将教育几位研究生。 调查人员还将开发相关的本科研究项目，并为高中生的AI夏令营提供项目。这种反相反的原因是挑战性的是缺乏标记的真实数据。调查人员将通过使用多视图输入来解决这一赤字，以培训并共同解决现场的所有组成部分。这将使他们能够使用自我监督的多视图光度重建损失，从而在视图上执行一致性。他们将将这些真实数据与大型高度现实合成数据的新数据集相结合。 他们还将开发一个统一的体系结构，该体系结构在视图之间恢复姿势并结合了几何和光度信息。这将使系统能够预测跨观点的外观变化，并确定测量的外观是否与恢复的照明和反射率属性一致。 他们还将在场景中开发新颖的照明表现，以及对象反映这种光线紧凑且有效的光的方式。该奖项反映了NSF的法定任务，并使用基金会的知识分子和更广泛的影响评估标准，被认为值得通过评估来获得支持。

项目成果

LD-ZNet: A Latent Diffusion Approach for Text-Based Image Segmentation

• DOI: 10.1109/iccv51070.2023.00384
• 发表时间: 2023-03
• 期刊: 2023 IEEE/CVF International Conference on Computer Vision (ICCV)
• 作者: K. Pnvr;Bharat Singh;P. Ghosh;Behjat Siddiquie;David Jacobs

Object-Aware Cropping for Self-Supervised Learning

• 发表时间: 2021-12
• 期刊: ArXiv
• 作者: Shlok Kumar Mishra;Anshul B. Shah;Ankan Bansal;Abhyuday N. Jagannatha;Abhishek Sharma;David Jacobs;Dilip Krishnan

SharinGAN: Combining Synthetic and Real Data for Unsupervised Geometry Estimation

• DOI: 10.1109/cvpr42600.2020.01399
• 发表时间: 2020-06
• 期刊: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
• 作者: K. Pnvr;Hao Zhou;D. Jacobs

On the Spectral Bias of Convolutional Neural Tangent and Gaussian Process Kernels

• DOI: 10.48550/arxiv.2203.09255
• 发表时间: 2022-03
• 期刊: ArXiv
• 作者: Amnon Geifman;M. Galun;David Jacobs;R. Basri

HaLP: Hallucinating Latent Positives for Skeleton-based Self-Supervised Learning of Actions

• DOI: 10.1109/cvpr52729.2023.01807
• 发表时间: 2023-04
• 期刊: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
• 作者: Anshul B. Shah;A. Roy;Ketul Shah;Shlok Kumar Mishra;David W. Jacobs;A. Cherian;Ramalingam Chellappa