Collaborative Research: Inverse Task Planning from Few-Shot Vision Language Demonstrations

协作研究：基于少镜头视觉语言演示的逆向任务规划

• 批准号: 2327974
• 负责人: Jeannette Bohg
• 金额: $ 45万
• 依托单位: Stanford University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2024
• 资助国家: 美国
• 起止时间: 2024-01-01 至 2026-12-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2327974&HistoricalAwards=false
• 关键词: Collaborative; Research; Inverse; Task; Planning

This project advances national prosperity and welfare by taking a step towards making collaborative robots more accessible across various settings, such as homes, factories, and logistics operations. Today, robots can only be programmed by engineers, limiting the tasks they can do to a narrow set of design choices made by these engineers. For robots to be widely adopted in households, they need to be versatile enough to tackle a much broader range of tasks. Many of these tasks, such as preparing meals or organizing the home, require personalization to the individual user's needs and preferences. This project aims to enable a robot to learn such personalized tasks through natural interactions with the user who may provide visual demonstrations combined with language narration to describe the task. Neither vision nor language alone is perfect, but together, they capture the task that the user wants to convey. This research leverages the broad impact Large Language Model (LLM) interfaces, such as ChatGPT, has had on engaging with everyday users and brings that into the physical realm with robots. It addresses fundamental challenges like summarizing vision-language demonstrations, verifying automatically generated plans and efficiently solving tasks that require many steps to achieve a goal. If successful, the project could transform many consumer robotics applications, enabling robots to be more usable, personalized and aligned with user values.The primary objective of this project is to develop a framework for learning complex, long-horizon tasks from few-shot vision-language demonstrations. While existing approaches in Inverse Reinforcement Learning (IRL) enable learning simple, short-horizon skills from demonstrations, scaling these approaches to longer horizons with fewer demonstrations poses fundamental statistical and computational challenges. To address these challenges, a novel framework called Inverse Task Planning (ITP) that combines the generalization power of Large Language Models (LLMs) with performance guarantees of IRL to both efficiently and verifiably learn tasks will be used. This approach is uniquely different from existing work in LLM and task planning as it creates a closed-loop system to align LLM outputs with human demonstrations. Concretely, the plan is to: (1) parse vision-language demonstrations as robot state-action trajectories using visual question answering (2) learn language-based reward summaries from long-horizon state-action trajectories, and (3) optimize rewards by generating high-level task-code in a verifiable, closed-loop fashion. This research has broad implications for creating new interfaces that allow everyday users to program robots, developing courses on generative models and robotics, and providing immersive and engaging programming activities for K-12 students.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该项目通过迈出一步，使协作机器人在各种环境（例如房屋，工厂和物流运营）方面更容易访问国家繁荣和福利。如今，工程师只能对机器人进行编程，从而将他们可以执行的任务限制为这些工程师做出的一组狭窄的设计选择。为了使机器人在家庭中被广泛采用，它们需要足够多的用途，以应对更广泛的任务。这些任务中的许多，例如准备餐点或组织房屋，都需要个性化用户的需求和偏好。该项目旨在通过与用户的自然互动来学习这种个性化任务，这些用户可以提供视觉演示与语言叙述相结合以描述任务。视觉和语言都不是完美的，但是他们一起捕捉了用户想要传达的任务。这项研究利用了诸如Chatgpt之类的广泛影响大语言模型（LLM）界面与日常用户互动并将其带入机器人的物理领域。它解决了基本挑战，例如总结视觉语言示范，验证自动生成的计划并有效地解决需要实现目标的许多步骤的任务。如果成功的话，该项目可能会改变许多消费者机器人的应用程序，使机器人能够更可用，个性化和与用户价值观保持一致。该项目的主要目的是从几个示意的视觉示范中开发一个框架，以学习学习复杂，长期的长途任务。虽然现有的逆增强学习方法（IRL）使学习能够从演示中学习简单，短马技能，但将这些方法扩展到更较少的示威的较长视野，带来了基本的统计和计算挑战。为了应对这些挑战，一个称为逆任务计划（ITP）的新颖框架将大型语言模型（LLMS）的概括功率与IRL的性能保证相结合，以有效地和可验证地学习任务。这种方法与LLM和任务计划中的现有工作完全不同，因为它创建了一个闭环系统，以使LLM输出与人类的示范相结合。具体而言，该计划是：（1）使用视觉问题答案解析视觉示范作为机器人状态行动轨迹（2）从长途状态状态轨迹中学习基于语言的奖励摘要，（3）（3）通过以可验证的，封闭的方式来生成高级任务报道来优化奖励。这项研究对创建新的接口具有广泛的影响，使日常用户能够为机器人编程，开发有关生成模型和机器人技术的课程，并为K-12学生提供沉浸式和引人入胜的编程活动。该奖项反映了NSF的法定任务，并认为通过基金会的知识分子和更广泛的影响，可以通过评估来进行评估。