Collaborative Research: CISE: Large: Executing Natural Instructions in Realistic Uncertain Worlds

合作研究：CISE：大型：在现实的不确定世界中执行自然指令

• 批准号: 2321851
• 负责人: Alan Fern
• 金额: $ 281.25万
• 依托单位: Oregon State University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2023
• 资助国家: 美国
• 起止时间: 2023-10-01 至 2028-09-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2321851&HistoricalAwards=false
• 关键词: Collaborative; Research; CISE; Large; Executing

For robots to fluidly operate as human assistants they must be able to take natural language instructions from humans and act to achieve those instructions in complex, uncertain environments. While there are commodity robots that are physically capable of carrying out a wide variety of useful instructions, current artificial intelligence (AI) frameworks are not able to fully understand and autonomously execute most of those instructions. Rather, current AI techniques for robot instruction following have generally been limited to highly constrained, unrealistic environments and instruction formats that are unnatural, brittle, and rigid. The overarching goal of the project is to study the fundamental AI principles that enable robots to reliably execute natural instructions in realistic, uncertain worlds. The project has the potential to dramatically increase the physical labor available to society, without increasing the amount of human labor, by enabling typical human workers to direct semi-automated robots for a multitude of mundane tasks. This will only be possible if the machines are easily instructable in natural environments by humans who only require minimal specialized training. This envisioned labor multiplier is extremely relevant given the need for increased capacity to build physical infrastructure in the US. This includes, for example, new and upgraded public infrastructure such as bridges and energy systems, as well as efficient construction of affordable housing. These same advances will also result in broader impacts to other parts of the economy, such as logistics, healthcare, household assistants. The project will also contribute to education and outreach through K-12 initiatives, undergraduate research experiences, and recruiting of underrepresented graduate student talent.The project will design and develop a novel integrated framework for embodied AI agents that is comprised of synergistic advances in computer vision, language understanding, world modeling, planning, and control. The framework will evolve over a staged plan of increasing capabilities, starting with step-by-step instruction execution and progressing to executing general types of goal-oriented instructions. The research will test and demonstrate the framework in both physically-realistic simulation environments and real-world environments using commodity robots. In addition, user studies will be conducted at each capability stage to focus the work toward end-user utility. Central to the framework is a new knowledge structure for spatio-temporal scenes, the multi-modal entity map (MEM), which is updated based on vision and language and used for both planning and skill execution. The research will study new ideas in 3D vision and language understanding for continually maintaining the MEM based on realistic inputs in a way that captures uncertainty in the environment. The project will also study a new approach to low-level full-body control for robot skills, inspired by recent successes in language modeling, that facilitates both modular skill learning and knowledge sharing. Finally, the project will advance automated planning capabilities by studying new ideas for learning dynamics models over the MEMs, which are used by a novel approach to high-level skill planning based on dynamics-conditioned language models. Importantly all of these innovations will be developed in a synchronized way to allow for rigorous testing and demonstration of the integrated framework.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

为了使机器人能够以人为助手的身份流畅地操作，他们必须能够从人类那里采取自然语言指示并采取行动以在复杂，不确定的环境中实现这些指示。尽管有一些商品机器人在身体上能够执行各种有用的说明，但当前的人工智能（AI）框架无法完全理解和自主执行这些说明。相反，以下机器人指导的当前AI技术通常仅限于高度约束，不现实的环境和指令格式，这些格式是不自然，脆弱和僵化的。该项目的总体目标是研究基本的人工智能原则，使机器人能够可靠地在现实，不确定的世界中执行自然指示。该项目有可能通过使典型的人类工人能够指导半自动化的机器人来完成多种平凡的任务，从而显着增加社会可用的体力劳动，而无需增加人工劳动的数量。只有当机器在自然环境中轻松地指导的人类只需要最少的专业培训的人类就可以指导这些机器。鉴于在美国建立物理基础设施的能力需要提高，这种设想的劳动乘数非常重要。例如，其中包括新的和升级的公共基础设施，例如桥梁和能源系统，以及有效的负担得起住房建设。这些相同的进步也将导致对经济其他地区的广泛影响，例如物流，医疗保健，家庭助手。该项目还将通过K-12举措，本科研究经验以及招募代表性不足的研究生人才的招募来为教育和宣传做出贡献。该项目将为体现的AI代理设计和开发一个新颖的集成框架，该框架由计算机视觉，语言理解，世界模型，计划，计划，计划，计划和控制。该框架将在阶段性的增加能力的计划上发展，从分步说明执行开始，然后发展到执行面向目标的指令的一般类型。这项研究将在使用商品机器人的物理现实模拟环境和现实环境中测试和证明框架。此外，将在每个能力阶段进行用户研究，以将工作集中在最终用户实用程序上。框架的核心是时空场景的新知识结构，即多模式实体图（MEM），它根据视觉和语言进行更新，并用于计划和技能执行。该研究将研究3D视觉和语言理解中的新思想，以根据现实的投入不断地维护MEM，以捕捉环境中的不确定性。该项目还将研究一种新的方法，以实现语言建模的最新成功启发，以促进模块化技能和知识共享。最后，该项目将通过研究MEMS学习动态模型的新想法来提高自动计划功能，该想法是由一种基于动态条件的语言模型的新型方法用于高级技能计划的新方法。重要的是，所有这些创新都将以同步的方式开发，以允许进行严格的测试并展示集成框架。该奖项反映了NSF的法定任务，并被认为是值得通过基金会的知识分子和更广泛影响的评估评估标准的评估值得支持的。