Collaborative Research: CISE: Large: Executing Natural Instructions in Realistic Uncertain Worlds

合作研究：CISE：大型：在现实的不确定世界中执行自然指令

• 批准号: 2321851
• 负责人: Alan Fern
• 金额: $ 281.25万
• 依托单位: Oregon State University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2023
• 资助国家: 美国
• 起止时间: 2023-10-01 至 2028-09-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2321851&HistoricalAwards=false
• 关键词: Collaborative; Research; CISE; Large; Executing

For robots to fluidly operate as human assistants they must be able to take natural language instructions from humans and act to achieve those instructions in complex, uncertain environments. While there are commodity robots that are physically capable of carrying out a wide variety of useful instructions, current artificial intelligence (AI) frameworks are not able to fully understand and autonomously execute most of those instructions. Rather, current AI techniques for robot instruction following have generally been limited to highly constrained, unrealistic environments and instruction formats that are unnatural, brittle, and rigid. The overarching goal of the project is to study the fundamental AI principles that enable robots to reliably execute natural instructions in realistic, uncertain worlds. The project has the potential to dramatically increase the physical labor available to society, without increasing the amount of human labor, by enabling typical human workers to direct semi-automated robots for a multitude of mundane tasks. This will only be possible if the machines are easily instructable in natural environments by humans who only require minimal specialized training. This envisioned labor multiplier is extremely relevant given the need for increased capacity to build physical infrastructure in the US. This includes, for example, new and upgraded public infrastructure such as bridges and energy systems, as well as efficient construction of affordable housing. These same advances will also result in broader impacts to other parts of the economy, such as logistics, healthcare, household assistants. The project will also contribute to education and outreach through K-12 initiatives, undergraduate research experiences, and recruiting of underrepresented graduate student talent.The project will design and develop a novel integrated framework for embodied AI agents that is comprised of synergistic advances in computer vision, language understanding, world modeling, planning, and control. The framework will evolve over a staged plan of increasing capabilities, starting with step-by-step instruction execution and progressing to executing general types of goal-oriented instructions. The research will test and demonstrate the framework in both physically-realistic simulation environments and real-world environments using commodity robots. In addition, user studies will be conducted at each capability stage to focus the work toward end-user utility. Central to the framework is a new knowledge structure for spatio-temporal scenes, the multi-modal entity map (MEM), which is updated based on vision and language and used for both planning and skill execution. The research will study new ideas in 3D vision and language understanding for continually maintaining the MEM based on realistic inputs in a way that captures uncertainty in the environment. The project will also study a new approach to low-level full-body control for robot skills, inspired by recent successes in language modeling, that facilitates both modular skill learning and knowledge sharing. Finally, the project will advance automated planning capabilities by studying new ideas for learning dynamics models over the MEMs, which are used by a novel approach to high-level skill planning based on dynamics-conditioned language models. Importantly all of these innovations will be developed in a synchronized way to allow for rigorous testing and demonstration of the integrated framework.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

为了使机器人能够像人类助手一样流畅地操作，它们必须能够接受人类的自然语言指令，并在复杂、不确定的环境中采取行动以实现这些指令。虽然商用机器人在物理上能够执行各种有用的指令，但当前的人工智能 (AI) 框架无法完全理解并自主执行大部分指令。相反，当前用于机器人指令跟踪的人工智能技术通常仅限于高度受限、不切实际的环境和不自然、脆弱且僵化的指令格式。该项目的总体目标是研究人工智能的基本原理，使机器人能够在现实、不确定的世界中可靠地执行自然指令。该项目有可能在不增加人类劳动力数量的情况下大幅增加社会可用的体力劳动，让典型的人类工人能够指挥半自动化机器人完成大量平凡的任务。只有当机器能够在自然环境中由人类轻松指导且只需要最少的专业培训时，这才有可能实现。考虑到美国需要增加建设有形基础设施的能力，这一设想的劳动力乘数极为重要。例如，这包括桥梁和能源系统等新建和升级的公共基础设施，以及经济适用房的高效建设。这些进步还将对经济的其他领域产生更广泛的影响，例如物流、医疗保健、家庭助理。该项目还将通过 K-12 计划、本科生研究经验以及招募代表性不足的研究生人才，为教育和推广做出贡献。该项目将为具体人工智能代理设计和开发一个新颖的集成框架，该框架由计算机视觉领域的协同进步组成、语言理解、世界建模、规划和控制。该框架将按照增强功能的分阶段计划不断发展，从逐步指令执行开始，逐步发展到执行一般类型的面向目标的指令。该研究将使用商用机器人在物理真实的模拟环境和现实环境中测试和演示该框架。此外，将在每个能力阶段进行用户研究，以将工作重点放在最终用户效用上。该框架的核心是时空场景的新知识结构，即多模态实体图（MEM），它基于视觉和语言进行更新，用于规划和技能执行。该研究将研究 3D 视觉和语言理解方面的新想法，以便以捕获环境不确定性的方式持续维护基于现实输入的 MEM。该项目还将研究一种新的机器人技能低级全身控制方法，其灵感来自于最近在语言建模方面取得的成功，该方法有助于模块化技能学习和知识共享。最后，该项目将通过研究 MEM 上学习动态模型的新思路来提高自动化规划能力，基于动态条件语言模型的高级技能规划新方法将使用这些新思路。重要的是，所有这些创新都将以同步方式开发，以便对集成框架进行严格的测试和演示。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力优点和更广泛的影响审查标准进行评估，被认为值得支持。