CHS: Small: Collaborative Research: Detailed Shape and Reflectance Capture with Light Field Cameras

CHS：小型：协作研究：使用光场相机捕获详细形状和反射率

• 批准号: 1617234
• 负责人: Ravi Ramamoorthi
• 金额: $ 25万
• 依托单位: University of California-San Diego
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2016
• 资助国家: 美国
• 起止时间: 2016-09-01 至 2020-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1617234&HistoricalAwards=false
• 关键词: CHS; Small; Collaborative; Research; Detailed

A major transformation is occurring in the way we sense the visual world. Traditional 2D photography is increasingly being replaced with light-field sensors that capture the full spatial and angular variation of the incoming light field, rather than simply pixels that integrate over incoming directions. This development opens up the possibility of ubiquitous 3D imaging of our visual world. Light-field sensors are particularly attractive as a depth acquisition device, since they are completely passive without needing to project light into the scene, and they do not experience a reduction in performance outdoors. Moreover, the rich ray-space of light fields provides significant cues for recovering fine-scale depth. However, current RGBD and light-field systems produce only coarse depth; while useful for tasks like refocusing images, the depth channel offers little benefit for photography beyond conventional 2D RGB images. This research seeks to address these challenges, by developing practical algorithms for detailed 3D shape and reflectance capture with light-field cameras, coupled with a theoretical and experimental analysis of the achievable accuracy. Project outcomes will have broad impact on diverse fields including computer graphics and virtual/augmented reality, enabling acquisition of high-quality detailed 3D shape and the subsequent use of the 3D geometry with computer-generated synthetic objects. Methods to acquire 3D images, including on mobile sensors, will transform the photographic process from 2D to 3D, with immense industrial and societal impact.The PIs will address four important problems in light-field shape acquisition. First, they will exploit the rich nature of light-field data, combining multiple cues (defocus, correspondence, shading, specularity) in a unified way to obtain the overall global 3D shape. Moreover, they will seek to go beyond the common Lambertian reflectance assumption, developing a novel BRDF-invariant framework for surface reconstruction with general glossy materials like metals, plastics, or ceramics, while supporting textures and spatially-varying reflectance. Another key objective will be to ground the practical results in a theoretical framework that can establish the limits of light-field camera shape resolution, and the signal-to-noise accuracy, and how this relates to novel designs for light-field cameras to obtain the best achievable resolution in 3D shape capture. Finally, the PIs will move from overall shape to fine-scale surface detail, proposing new methods for shape/reflectance capture for fine-scale geometry like hair. The ultimate goal is to enable a full 3D processing pipeline for photography, computer graphics and applications like virtual and augmented reality.

我们感知视觉世界的方式正在发生重大转变。 传统的 2D 摄影越来越多地被光场传感器所取代，光场传感器捕获入射光场的完整空间和角度变化，而不是简单地集成入射方向上的像素。 这一发展为我们视觉世界中无处不在的 3D 成像开辟了可能性。 光场传感器作为深度采集设备特别有吸引力，因为它们是完全无源的，不需要将光投射到场景中，并且它们在户外的性能不会下降。 此外，光场丰富的光线空间为恢复精细尺度深度提供了重要线索。 然而，当前的 RGBD 和光场系统只能产生粗略的深度；虽然深度通道对于重新聚焦图像等任务很有用，但除了传统的 2D RGB 图像之外，深度通道对摄影几乎没有什么好处。 本研究旨在通过开发使用光场相机捕获详细 3D 形状和反射率的实用算法，并对可实现的精度进行理论和实验分析，来应对这些挑战。 项目成果将对计算机图形学和虚拟/增强现实等不同领域产生广泛影响，从而能够获取高质量的详细 3D 形状，并随后将 3D 几何图形与计算机生成的合成对象一起使用。 获取 3D 图像的方法（包括移动传感器）将把摄影过程从 2D 转变为 3D，从而产生巨大的工业和社会影响。PI 将解决光场形状采集中的四个重要问题。 首先，他们将利用光场数据的丰富性质，以统一的方式组合多种线索（散焦、对应、阴影、镜面反射）以获得整体的全局 3D 形状。 此外，他们将寻求超越常见的朗伯反射率假设，开发一种新颖的 BRDF 不变框架，用于使用金属、塑料或陶瓷等一般光泽材料进行表面重建，同时支持纹理和空间变化的反射率。 另一个关键目标是将实际结果建立在理论框架中，该框架可以确定光场相机形状分辨率和信噪比精度的极限，以及这与光场相机的新颖设计有何关系，以获得3D 形状捕捉中可实现的最佳分辨率。 最后，PI 将从整体形状转向精细表面细节，提出针对头发等精细几何形状/反射率捕获的新方法。 最终目标是为摄影、计算机图形以及虚拟和增强现实等应用程序提供完整的 3D 处理流程。