CAREER: Perceptual Cameras: Forming Images Through Scene Interpretation

职业：感知相机：通过场景解释形成图像

• 批准号: 2047359
• 负责人: Felix Heide
• 金额: $ 50万
• 依托单位: Princeton University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-02-01 至 2026-01-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2047359&HistoricalAwards=false
• 关键词: CAREER; Perceptual; Cameras; Forming; Images

Today’s cameras provide a digital window into the real world with broad applications across societal and scientific areas. Despite their remarkably diverse applications, existing cameras are engineered as general-purpose sensing and signal-processing pipelines. This project breaks with this conventional approach and proposes methods to “computationally evolve” the cameras of tomorrow. As such, the results will drastically expand our understanding of how to develop and optimize entire cameras and signal processing chain for a specific application domain, including medical imaging, robotics, scientific imaging, virtual/artificial reality, and self-driving vehicles. We will develop a completely new breed of cameras for these diverse application domains, for example, ones that may consider the scene as part of the camera. The research efforts are tightly integrated with an outreach program that introduces underrepresented and at-risk students in the New Jersey and New York area to science and technology through domain-specific cameras for self-driving vehicles.The research of this project will develop a novel comprehensive learning framework that allows the researchers to reason over a distribution of cameras in a continuous and differentiable fashion. This framework will hinge on a theory that models the illumination, acquisition, processing, scene light transport, and illumination stages of a broad space of cameras. As such, the research team will be able to optimize over the architecture and parameters of full sensing and processing stacks, resulting in fundamentally novel cameras tailored to specific imaging and perception tasks. These new cameras learn to shift complexity between optics, compute, illumination, sensing, and the scene light transport, exploiting the scene and scene semantics as part of the imaging process. This enables unprecedented capabilities for domain-specific imaging in scattering media, ultra-miniaturized learned cameras, neural optical compute, and imaging at ultra-large scales in adverse conditions and ultra-small scales, all of which will be explored in this project.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

当今的摄像机为现实世界提供了一个数字窗口，并在社会和科学领域进行了广泛的应用。尽管它们的应用非常多样化，但现有的相机还是以通用敏感性和信号处理管道进行设计的。该项目以这种常规的方法和建议方法“计算进化”了明天的相机。因此，结果将大大扩展我们对如何开发和优化特定应用领域的整个相机和信号处理链的理解，包括医学成像，机器人技术，科学成像，虚拟/人造现实和自动驾驶工具。例如，我们将为这些不同的应用域开发全新的相机，例如，可以将场景视为相机的一部分。研究工作与外展计划紧密整合，该计划通过针对自动驾驶车辆的领域特异性摄像机介绍了新泽西州和纽约地区的代表性不足和处于危险的学生，用于科学和技术。该项目的研究将开发出一个新颖的综合学习框架，该框架使研究人员能够在持续和不同的时尚和不同的时尚中分配摄影师。该框架将取决于一个理论，该理论模拟了相机广泛空间的照明，获取，处理，场景光传输和照明阶段。因此，研究团队将能够对完全敏感性和处理堆栈的体系结构和参数进行优化，从而产生针对特定成像和感知任务量身定制的新型相机。这些新的相机学会在光学，计算，照明，灵敏度和场景光传输之间转移复杂性，并利用场景和场景语义作为成像过程的一部分。这使得在散射媒体中特定于领域的特定成像，超弥补的学习相机，神经光学计算以及在不利条件下的超大尺度上的成像具有前所未有的功能 标准。

项目成果

ZeroScatter: Domain Transfer for Long Distance Imaging and Vision through Scattering Media

• DOI: 10.1109/cvpr46437.2021.00348
• 发表时间: 2021-02
• 期刊: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
• 作者: Z. Shi;Ethan Tseng;Mario Bijelic;W. Ritter;Felix Heide

Mask-ToF: Learning Microlens Masks for Flying Pixel Correction in Time-of-Flight Imaging

• DOI: 10.1109/cvpr46437.2021.00900
• 发表时间: 2021-03
• 期刊: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
• 作者: Ilya Chugunov;Seung-Hwan Baek;Q. Fu;W. Heidrich;Felix Heide

Learned Hardware-in-the-loop Phase Retrieval for Holographic Near-Eye Displays

• DOI: 10.1145/3414685.3417846
• 发表时间: 2020-12-01
• 期刊: ACM TRANSACTIONS ON GRAPHICS
• 影响因子: 6.2
• 作者: Chakravarthula, Praneeth;Tseng, Ethan;Heide, Felix
• 通讯作者: Heide, Felix

Differentiable Compound Optics and Processing Pipeline Optimization for End-to-end Camera Design

• DOI: 10.1145/3446791
• 发表时间: 2021-06-01
• 期刊: ACM TRANSACTIONS ON GRAPHICS
• 影响因子: 6.2
• 作者: Tseng, Ethan;Mosleh, Ali;Heide, Felix
• 通讯作者: Heide, Felix

Polka Lines: Learning Structured Illumination and Reconstruction for Active Stereo

• DOI: 10.1109/cvpr46437.2021.00570
• 发表时间: 2021-01-01
• 期刊: 2021 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION, CVPR 2021
• 作者: Baek, Seung-Hwan;Heide, Felix
• 通讯作者: Heide, Felix