Real-Time Light Field Processing for Autonomous Robotics

Typical autonomous robotics systems incorporate multiple cameras, LIDAR sensors and sophisticated computing resources. In this paper we present a software framework for utilizing any array of multiple cameras with sufficient field-of-view (FOV) overlap as a light field imaging system. We show that the typical linear arrays that exist on autonomous cars are sufficient to capture stable time resolved light fields even when moving at highway speeds. We elaborate on the potential pitfalls associated with such a technique namely loss of calibration between cameras due to high frequency vibrations and sudden shocks associated with driving over potholes and highlight a method that can compensate for such effects. We demonstrate that the light fields collected by simple linear arrays can be processed in real time for a wide variety of useful applications including occlusion removal, for signal enhancement in featureless images captured in very low light, for reflection removal and for improved visibility in extreme conditions associated with snow and heavy rain.

典型的自主机器人系统包含多个摄像头、激光雷达传感器和复杂的计算资源。在本文中，我们提出了一个软件框架，用于将任何具有足够视场（FOV）重叠的多摄像头阵列用作光场成像系统。我们表明，自动驾驶汽车上常见的线性阵列即使在高速公路行驶速度下也足以捕捉稳定的时间分辨光场。我们详细阐述了与这种技术相关的潜在陷阱，即由于高频振动以及驶过坑洼时的突然冲击导致的摄像头之间校准丢失，并强调了一种可以补偿此类影响的方法。我们证明，由简单线性阵列收集的光场可以实时处理，用于多种有用的应用，包括去除遮挡、在极弱光下拍摄的无特征图像中增强信号、去除反射以及在与雪和大雨相关的极端条件下提高能见度。