Multi-Modal Geometric Learning for Grasping and Manipulation

This work provides an architecture that incorporates depth and tactile information to create rich and accurate 3D models useful for robotic manipulation tasks. This is accomplished through the use of a 3D convolutional neural network (CNN). Offline, the network is provided with both depth and tactile information and trained to predict the object's geometry, thus filling in regions of occlusion. At runtime, the network is provided a partial view of an object. Tactile information is acquired to augment the captured depth information. The network can then reason about the object's geometry by utilizing both the collected tactile and depth information. We demonstrate that even small amounts of additional tactile information can be incredibly helpful in reasoning about object geometry. This is particularly true when information from depth alone fails to produce an accurate geometric prediction. Our method is benchmarked against and outperforms other visual-tactile approaches to general geometric reasoning. We also provide experimental results comparing grasping success with our method.

这项工作提供了一种架构，该架构融合了深度和触觉信息，以创建对机器人操作任务有用的丰富且准确的3D模型。这是通过使用3D卷积神经网络（CNN）实现的。在离线状态下，该网络被提供深度和触觉信息，并接受训练以预测物体的几何形状，从而填补遮挡区域。在运行时，该网络被提供一个物体的局部视图。获取触觉信息以增强所捕获的深度信息。然后，该网络可以通过利用所收集的触觉和深度信息来推断物体的几何形状。我们证明，即使是少量的额外触觉信息在推断物体几何形状方面也非常有帮助。当仅从深度获取的信息无法产生准确的几何预测时，尤其如此。我们的方法与其他视觉 - 触觉通用几何推理方法进行了对比测试，并且表现更优。我们还提供了比较我们的方法抓取成功率的实验结果。