RI: Medium: Learning to Map and Navigate with Vision and Language

RI：媒介：学习用视觉和语言绘制地图和导航

• 批准号: 2212433
• 负责人: Kostas Daniilidis
• 金额: $ 120万
• 依托单位: University of Pennsylvania
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-09-01 至 2026-08-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2212433&HistoricalAwards=false
• 关键词: RI; Medium; Learning; Map; Navigate

This project aims to advance the state of the art in robotic mapping and navigation by enabling spatial understanding using semantic maps and spatial reasoning for following language instructions given only visual inputs. Current performance in those tasks is low because of the inability to ground semantic entities and instructions spatially. Instead of grounding semantics to images, spatial understanding and navigation can be achieved if a system uses maps as an intermediate representation, as also indicated by behavioral and neural findings in spatial cognition. Building a map of an unseen space without exhaustive exploration can be learned, and this process can be facilitated by cross-modal language-vision attentional mechanisms. The project will integrate research with education and outreach underrepresented groups in Philadelphia neighborhoods as a target broadening the participation.This research is centered around understanding how vision and language interact to create better spatial representations like maps and facilitate navigation. The project will approach the vision-language from three angles. (i) How robots can learn to predict a map when entering an unseen environment using active learning. (ii) How navigation instructions can be encoded into spatial configuration schemata and navigational concepts that can be better aligned to maps and paths than raw language embeddings, and (iii) how navigational language representations can facilitate the creation of maps in unseen environments, and how one can follow instructions by using maps and language to create paths to follow.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该项目旨在通过使用语义地图和空间推理来实现空间理解，以遵循仅给出视觉输入的语言指令，从而推进机器人测绘和导航的最新技术。由于无法在空间上定位语义实体和指令，这些任务的当前性能较低。如果系统使用地图作为中间表示，则可以实现空间理解和导航，而不是将语义扎根于图像，空间认知中的行为和神经发现也表明了这一点。无需进行详尽的探索即可构建看不见的空间地图，并且可以通过跨模态语言视觉注意力机制来促进这一过程。该项目将把研究与教育和推广费城社区中代表性不足的群体结合起来，作为扩大参与的目标。这项研究的重点是了解视觉和语言如何相互作用，以创建更好的空间表示（如地图）并促进导航。该项目将从三个角度探讨视觉语言。 (i) 机器人如何使用主动学习在进入看不见的环境时学习预测地图。 （ii）如何将导航指令编码为空间配置模式和导航概念，从而比原始语言嵌入更好地与地图和路径对齐，以及（iii）导航语言表示如何促进在看不见的环境中创建地图，以及如何人们可以通过使用地图和语言来创建遵循的路径来遵循指示。该奖项反映了 NSF 的法定使命，并且通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Cross-modal Map Learning for Vision and Language Navigation

用于视觉和语言导航的跨模态地图学习

• DOI: 10.1109/cvpr52688.2022.01502
• 发表时间: 2022-03-10
• 期刊: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
• 作者: G. Georgakis;Karl Schmeckpeper;Karan Wanchoo;Soham Dan;E. Miltsakaki;D. Roth;Kostas Daniilidis
• 通讯作者: Kostas Daniilidis