FlowNet: Learning Optical Flow with Convolutional Networks

Convolutional neural networks (CNNs) have recently been very successful in a variety of computer vision tasks, especially on those linked to recognition. Optical flow estimation has not been among the tasks CNNs succeeded at. In this paper we construct CNNs which are capable of solving the optical flow estimation problem as a supervised learning task. We propose and compare two architectures: a generic architecture and another one including a layer that correlates feature vectors at different image locations. Since existing ground truth data sets are not sufficiently large to train a CNN, we generate a large synthetic Flying Chairs dataset. We show that networks trained on this unrealistic data still generalize very well to existing datasets such as Sintel and KITTI, achieving competitive accuracy at frame rates of 5 to 10 fps.

卷积神经网络（CNN）最近在各种计算机视觉任务中非常成功，尤其是在与识别相关的方面。 CNN成功的任务尚未获得光流估计。在本文中，我们构建了能够将光流估计问题作为监督的学习任务解决的CNN。我们提出并比较了两个架构：一个通用体系结构，另一个包括一个将特征向量在不同图像位置关联的图层。由于现有的地面真相数据集不足以训练CNN，因此我们生成了一个大型的合成飞行椅数据集。我们表明，经过此不切实际数据培训的网络仍然很好地推广到Sintel和Kitti等现有数据集，以5至10 fps的帧速率实现了竞争精度。