NRI: Collaborative Research: Jointly Learning Language and Affordances

NRI：协作研究：共同学习语言和功能可供性

• 批准号: 1426452
• 负责人: Stefanie Tellex
• 金额: $ 33.78万
• 依托单位: Brown University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2014
• 资助国家: 美国
• 起止时间: 2014-08-01 至 2017-07-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1426452&HistoricalAwards=false
• 关键词: NRI; Collaborative; Research; Jointly; Learning

The investigators of this project envision a world where robots surround us, in our homes, in our hospitals, and in our factories, helping people by delivering medicine, preparing food, and assembling objects. Achieving this vision requires robots to communicate with people about their needs, and then plan their activities to help meet those needs. Previous research has addressed these two problems separately, leading to technical solutions that do not work reliably in real-world situations, and to difficulties in human-robot communication. To solve these problems, we are developing the Physically-Grounded Language with Affordances (PGLA) framework and concentrate our research into two thrusts: 1) enable a robot to observe a patient, then answer a nurse's questions about the patient's activity, and 2) enable a robot to respond to natural language requests in a collaborative cooking task and in a manufacturing setting. We will release our open-source data sets and code, which will have impact in other technical areas beyond robotics, such as computer vision and machine learning. The results of our proposed research will find direct applications in industries such as manufacturing and assistive robotics.This project takes a probabilistic approach to jointly learn to recognize affordances in the environment and predict associated natural language requests and descriptions. Since the affordance map is grounded to perceptual data, our robots will learn to robustly manipulate objects in the physical world, respond to natural language commands, and describe their experiences using words. Our learning approach enables the robot to infer cross-model knowledge from large data sets of people carrying out activities paired with natural language descriptions of the activities, leveraging the strength of each modality to inform the others. Our novel learning algorithms will integrate and learn from multi-domain databases such as the semantic web, visual scenes, and a novel activity database paired with natural language descriptions.

该项目的研究人员设想了一个机器人围绕在我们周围的世界，在我们的家里、医院和工厂里，通过运送药品、准备食物和组装物品来帮助人们。实现这一愿景需要机器人与人们沟通他们的需求，然后规划他们的活动来帮助满足这些需求。 先前的研究分别解决了这两个问题，导致技术解决方案在现实情况下无法可靠地工作，并且导致人机通信困难。 为了解决这些问题，我们正在开发具有功能可供性的物理基础语言（PGLA）框架，并将我们的研究集中在两个重点上：1）使机器人能够观察患者，然后回答护士关于患者活动的问题，2）使机器人能够在协作烹饪任务和制造环境中响应自然语言请求。我们将发布我们的开源数据集和代码，这将对机器人以外的其他技术领域产生影响，例如计算机视觉和机器学习。 我们提出的研究结果将在制造和辅助机器人等行业中找到直接应用。该项目采用概率方法来共同学习识别环境中的可供性并预测相关的自然语言请求和描述。 由于可供性地图以感知数据为基础，我们的机器人将学习如何稳健地操纵物理世界中的物体，响应自然语言命令，并使用文字描述他们的经历。 我们的学习方法使机器人能够从人们进行活动的大数据集中推断出跨模型知识，并结合活动的自然语言描述，利用每种模式的优势来通知其他模式。我们新颖的学习算法将集成并学习多领域数据库，例如语义网络、视觉场景以及与自然语言描述配对的新颖活动数据库。