EAGER: Learning Language in Simulation for Real Robot Interaction

EAGER：在模拟中学习语言以实现真实的机器人交互

• 批准号: 1940931
• 负责人: Cynthia Matuszek
• 金额: $ 21.95万
• 依托单位: University of Maryland Baltimore County
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-12-01 至 2021-11-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1940931&HistoricalAwards=false
• 关键词: EAGER; Learning; Language; Simulation; Real

While robots are rapidly becoming more capable and ubiquitous, theirutility is still severely limited by the inability of regular users tocustomize their behaviors. This EArly Grant for Exploratory Research (EAGER) will explore how examples of language, gaze, and other communications can be collected from avirtual interaction with a robot in order to learn how robots caninteract better with end users. Current robots' difficulty of use andinflexibility are major factors preventing them from being morebroadly available to populations that might benefit, such asaging-in-place seniors. One promising solution is to let users controland teach robots with natural language, an intuitive and comfortablemechanism. This has led to active research in the area of groundedlanguage acquisition: learning language that refers to and is informedby the physical world. Given the complexity of robotic systems, thereis growing interest in approaches that take advantage of the latest invirtual reality technology, which can lower the barrier of entry tothis research.This EAGER project develops infrastructure that will lay the necessarygroundwork for applying simulation-to-reality approaches to naturallanguage interactions with robots. This project aims to bootstraprobots' learning to understand language, using a combination of datacollected in a high-fidelity virtual reality environment withsimulated robots and real-world testing on physical robots. A personwill interact with simulated robots in virtual reality, and his or heractions and language will be recorded. By integrating with existingrobotics technology, this project will model the connection betweenthe language people use and the robot's perceptions and actions.Natural language descriptions of what is happening in simulation willbe obtained and used to train a joint model of language and simulatedpercepts as a way to learn grounded language. The effectiveness of theframework and algorithms will be measured on automaticprediction/generation tasks and transferability of learned models to areal, physical robot. This work will serve as a proof of concept forthe value of combining robotics simulation with human interaction, aswell as providing interested researchers with resources to bootstraptheir own work.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

尽管机器人迅速变得越来越有能力和无处不在，但它们的行为无法定期将其行为限制在很大程度上受到严重限制。这项探索性研究的早期赠款（急切）将探讨如何从与机器人的天年互动中收集语言，凝视和其他通信的示例，以了解机器人如何与最终用户更好地与最终用户进行更好的影响。当前的机器人的使用难度和互补性是阻止他们在可能受益的人口（例如实地老年人）的人口中更加宽敞地使用的主要因素。一种有希望的解决方案是让用户控制和用自然语言教授机器人，这是一种直观而舒适的机制。这导致了在扎根语言获取领域的积极研究：指的是涉及物理世界的学习语言。鉴于机器人系统的复杂性，人们对利用最新的Inviral Reality技术的方法越来越兴趣，从而降低了入境研究的障碍。此急切的项目开发基础架构，这将为将模拟对现实性方法应用于与机器人的自然语言相互作用。该项目的目的是使用在高保真虚拟现实环境中使用的数据进行结合，并在物理机器人上模拟了现实的测试，以实现Bootstraprobot的学习来理解语言。一个人将与虚拟现实中的模拟机器人互动，将记录其或遗传和语言。通过与现有生物体技术集成，该项目将在人们使用的语言和机器人的看法和动作之间建模连接。对模拟中正在发生的事情的自然语言描述将获得并用于培训语言的联合模型和模拟模型，并将其作为一种学习接地语言的方式。框架和算法的有效性将以自动预测/生成任务以及学习模型转移到Areal，物理机器人的能力进行衡量。这项工作将作为将机器人模拟与人类互动相结合的价值的概念证明，同时为有兴趣的研究人员提供了Bootstraptheir自己的作品的资源。该奖项反映了NSF的法定任务，并被认为是值得通过基金会的知识分子优点和更广泛的审查标准来通过评估来支持的。

项目成果

Planning with Abstract Learned Models While Learning Transferable Subtasks

在学习可转移子任务的同时使用抽象学习模型进行规划

• DOI: 10.1609/aaai.v34i06.6555
• 发表时间: 2020
• 期刊: Proceedings of the AAAI Conference on Artificial Intelligence
• 作者: Winder, John;Milani, Stephanie;Landen, Matthew;Oh, Erebus;Parr, Shane;Squire, Shawn;desJardins, Marie;Matuszek, Cynthia
• 通讯作者: Matuszek, Cynthia

Jointly Identifying and Fixing Inconsistent Readings from Information Extraction Systems

联合识别和修复信息提取系统的不一致读数

• 发表时间: 2022
• 期刊: Third Deep Learning Inside Out (DeeLIO
• 作者: Padia, Ankur;Ferraro, Francis;Finin, Tim
• 通讯作者: Finin, Tim

Bridging the Gap: Using Deep Acoustic Representations to Learn Grounded Language from Percepts and Raw Speech

• DOI: 10.1609/aaai.v36i10.21335
• 发表时间: 2021-12
• 作者: Gaoussou Youssouf Kebe;Luke E. Richards;Edward Raff;Francis Ferraro;Cynthia Matuszek

Neural Variational Learning for Grounded Language Acquisition

• DOI: 10.1109/ro-man50785.2021.9515374
• 发表时间: 2021-07
• 期刊: 2021 30th IEEE International Conference on Robot & Human Interactive Communication (RO-MAN)
• 作者: Nisha Pillai;Cynthia Matuszek;Francis Ferraro

Head Pose for Object Deixis in VR-Based Human-Robot Interaction

基于 VR 的人机交互中物体指示的头部姿势

• 发表时间: 2022
• 期刊: International Conference on Robot & Human Interactive Communication (Ro-Man
• 作者: Higgins, Padraig;Barron, Ryan;Matuszek, Cynthia
• 通讯作者: Matuszek, Cynthia