NRI: FND: Semi-Supervised Deep Learning for Domain Adaptation in Robotic Language Acquisition

NRI：FND：用于机器人语言习得领域适应的半监督深度学习

• 批准号: 2024878
• 负责人: Cynthia Matuszek
• 金额: $ 74.87万
• 依托单位: University of Maryland Baltimore County
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-10-01 至 2024-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2024878&HistoricalAwards=false
• 关键词: NRI; FND; Semi; Supervised; Deep

This project will enable robots to learn to perform tasks with human teammates from language and other human modalities, and then transfer the learned knowledge across heterogeneous platforms and tasks. This will ultimately allow human-robot teaming in domains where people use varied language and instructions to complete complex tasks. As robots become more capable and ubiquitous, they are increasingly moving into complex, human-centric environments such as workplaces and homes. Being able to deploy useful robots in settings where human specialists are stretched thin, such as assistive technology, elder care, and education, has the potential to have far-reaching impacts on human quality of life. Achieving this will require the development of robots that learn, from natural interaction, about an end user's goals and environment. This work is intended to make robots more accessible and usable for non-specialists. In order to verify success and involve the broader community, tasks will be drawn from and tested in conjunction with community Makerspaces, which are strongly linked with both education and community involvement. The award includes an education and outreach plan designed to increase participation by and retention of women and underrepresented minorities (URM) in robotics and computing, engaging with UMBC's large URM population and world-class programs in this space.This award addresses how collaborative learning and successful performance during human-robot interactions can be accomplished by learning from and acting on grounded language. To accomplish this, this project will revolve around learning structured representations of abstract knowledge with goal-directed task completion, grounded in a physical context. There are three high-level research thrusts. In the first, new perceptual models to learn an alignment among a robot's multiple, heterogeneous sensor and data streams will be developed. In the second, synchronous grounded language models will be developed to better capture both general linguistic and implicit contextual expectations that are needed for completing tasks. In the third, a deep reinforcement learning framework will be developed that can leverage the advances achieved by the first two thrusts, allowing the development of techniques for learning conceptual knowledge. Taken together, these advances will allow an agent to achieve domain adaptation, improve its behaviors in new environments, and transfer conceptual knowledge among robotic agents.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该项目将使机器人能够从语言和其他人类模式中学习与人类队友执行任务，然后通过异质平台和任务转移学识关系。这最终将允许在人们使用各种语言和说明完成复杂任务的领域中进行人手组合。随着机器人变得越来越强大和无处不在，它们越来越多地进入复杂的以人为中心的环境（例如工作场所和房屋）。能够在人类专家稀薄的环境中部署有用的机器人，例如辅助技术，老年人护理和教育，有可能对人类生活质量产生深远的影响。实现这一目标将需要开发从自然互动中学习最终用户的目标和环境的机器人。这项工作旨在使机器人更容易访问，并且可用于非专家。为了验证成功并涉及更广泛的社区，将与社区创作空间一起绘制和测试，这些任务与教育和社区的参与密切相关。该奖项包括一个教育和外展计划，旨在通过在机器人技术和计算机领域的妇女和代表性不足的少数群体（URM）来增加参与，并与UMBC的大型URM人口和该领域的世界一流计划互动。该奖项解决了如何通过学习在人类互动期间的协作学习和成功表现，可以通过学习和在基础语言上的学习来实现。为了实现这一目标，该项目将围绕学习以目标指导的任务完成的学习结构化表示，并以物理背景为基础。有三个高级研究推力。首先，将开发出新的感知模型，以学习机器人多个，异构传感器和数据流之间的对齐方式。在第二个中，将开发同步的基础语言模型，以更好地捕获完成任务所需的一般语言和隐性上下文期望。在第三个中，将开发一个深厚的增强学习框架，可以利用前两个推力所取得的进步，从而开发学习概念知识的技术。综上所述，这些进步将使代理商能够实现领域的适应性，改善其在新环境中的行为，并在机器人代理之间转移概念知识。该奖项反映了NSF的法定任务，并被认为是值得通过基金会的知识分子来评估的支持，并具有更广泛的影响。

项目成果

Measuring Equality in Machine Learning Security Defenses: A Case Study in Speech Recognition

• DOI: 10.1145/3605764.3623911
• 发表时间: 2023-02
• 期刊: Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security
• 作者: Luke E. Richards;Edward Raff;Cynthia Matuszek

POQue: Asking Participant-specific Outcome Questions for a Deeper Understanding of Complex Events

• DOI: 10.48550/arxiv.2212.02629
• 发表时间: 2022-12
• 作者: Sai Vallurupalli;Sayontan Ghosh;K. Erk;Niranjan Balasubramanian;Francis Ferraro

Jointly Identifying and Fixing Inconsistent Readings from Information Extraction Systems

联合识别和修复信息提取系统的不一致读数

• 发表时间: 2022
• 期刊: Third Deep Learning Inside Out (DeeLIO
• 作者: Padia, Ankur;Ferraro, Francis;Finin, Tim
• 通讯作者: Finin, Tim

Lessons From A Small-Scale Robot Joining Experiment in VR

小型机器人参与 VR 实验的经验教训

• 发表时间: 2023
• 期刊: and Mixed-Reality for Human-Robot Interactions (VAM-HRI
• 作者: Higgins, Padraig;Barron, Ryan;Engel, Don;Matuszek, Cynthia
• 通讯作者: Matuszek, Cynthia

Augmenting Simulation Data with Sensor Effects for Improved Domain Transfer

利用传感器效应增强仿真数据以改进域传输

• 发表时间: 2022
• 期刊: Tenth International Workshop on Assistive Computer Vision and Robotics (ACVR
• 作者: Berlier, Adam J;Bhatt, Anjali;Matuszek, Cynthia
• 通讯作者: Matuszek, Cynthia