CAREER: Learning from demonstrations and beyond -- consolidating imitation and reinforcement learning

职业：从演示中学习以及超越——巩固模仿和强化学习

• 批准号: 2238979
• 负责人: Guni Sharon
• 金额: $ 58.45万
• 依托单位: Texas A&M Engineering Experiment Station
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2023
• 资助国家: 美国
• 起止时间: 2023-06-01 至 2028-05-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2238979&HistoricalAwards=false
• 关键词: CAREER; Learning; demonstrations; beyond; consolidating

Recent advancements in deep reinforcement learning (RL) hold unprecedented potential for automating and optimizing control of real-world tasks such as autonomous driving, traffic management, medical procedures, robotic manufacturing, and energy management. Unfortunately, it is common for RL algorithms to exhibit unstable and/or inefficient learning, which limits their applicability. Seeking to address this critical concern, this CAREER project leverages imitation learning (IL), or behavior copying, which is better understood and typically more stable. The project targets the unification of IL and RL into a holistic paradigm that can safely and effectively learn from, and outperform, existing solutions. This project will address outstanding knowledge gaps in both types of learning through a novel curriculum decomposition of the tasks, where simplified demonstrations are used to bootstrap the learner’s behavior. The project will also foster education and outreach activities. Specifically, it will enhance undergraduate STEM training by providing students with exposure to scientific research and knowledge discovery processes relating to safety-critical AI applications through an original multidisciplinary undergraduate engineering program. Moreover, it will facilitate a unique K12 outreach activity within a large minority (Hispanic/latino) community (Bryan, TX). The project will support and advance an existing research collaboration with an industrial partner in the context of defense technology. This collaboration, in turn, is expected to advance the US national defense.This project will form the basis for a new research thrust in ML---one that combines IL and RL toward a holistic, robust, and safe learning framework. It will define and prove a no-regret bound on the training process within the Markov-Decision Process formalization. The approach is to reduce an IL problem to an RL one that includes a domain-independent curriculum-learning trajectory. The resulting algorithms and solutions are expected to achieve state-of-the-art performance in complex control domains as well as to deepen theoretical understanding of the potential and limitations of the resulting solutions. Specifically, the research seeks to prove conditions guaranteeing policy convergence and monotonic improvement during training. Moreover, the project will develop domain-specific adaptation to and analysis of real-world applications (autonomous driving and robotics testbeds) while providing stable and efficient RL from demonstrations.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

深度强化学习 (RL) 的最新进展在自动化和优化现实世界任务（例如自动驾驶、交通管理、医疗程序、机器人制造和能源管理）方面具有前所未有的潜力，但不幸的是，RL 算法表现出的现象很常见。不稳定和/或低效的学习，限制了它们的适用性，为了解决这个关键问题，这个职业项目利用了更好理解并且通常更稳定的模仿学习（IL）。强化学习成为一种可以安全有效地学习并超越现有解决方案的整体范式，该项目将通过任务的新颖课程分解来解决这两种学习类型中的突出知识差距，其中使用简化的演示来引导学习者的行为。具体来说，该项目还将通过原创的多学科本科工程项目，让学生接触与安全关键型人工智能应用相关的科学研究和知识发现过程，从而加强本科生 STEM 培训。促进独特的K12该项目将支持和推进与工业合作伙伴在国防技术领域的现有研究合作。美国国防。该项目将构成 ML 的新研究方向的基础——将 IL 和 RL 结合起来，形成一个全面、稳健和安全的学习框架。它将定义并证明训练的无悔界限。马尔可夫决策过程中的过程该方法旨在将 IL 问题简化为 RL 问题，其中包含独立于领域的课程学习轨迹，预计所得算法和解决方案将在复杂的控制领域实现最先进的性能。具体来说，该研究旨在证明训练过程中保证政策收敛和单调改进的条件。此外，该项目还将开发针对实际应用（自主）的特定领域的适应和分析。驾驶和机器人技术测试床），同时通过演示提供稳定和高效的强化学习。该奖项反映了高效的 NSF 法定使命，并且通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Comparison between popular Genetic Algorithm (GA)-based tool and Covariance Matrix Adaptation - Evolutionary Strategy (CMA-ES) for optimizing indoor daylight

用于优化室内日光的流行的基于遗传算法 (GA) 的工具与协方差矩阵适应 - 进化策略 (CMA-ES) 的比较

• DOI: 10.26868/25222708.2023.1218
• 发表时间: 2023
• 期刊: Proceedings of Building Simulation 2023: 18th Conference of IBPSA
• 作者: Anis, Manal;Pendurkar, Sumedh;Yi, Yun Kyu;Sharon, Guni
• 通讯作者: Sharon, Guni

The (Un)Scalability of Informed Heuristic Function Estimation in NP-Hard Search Problems

NP 难搜索问题中知情启发式函数估计的（非）可扩展性

• 发表时间: 2023
• 期刊: Transactions on Machine Learning Research
• 作者: Sumedh Pendurkar;Taoan Huang;Brendan Juba;Jiapeng Zhang;Sven Koenig;Guni Sharon
• 通讯作者: Guni Sharon

Bilevel Entropy based Mechanism Design for Balancing Meta in Video Games

• DOI: 10.5555/3545946.3598887
• 发表时间: 2023
• 作者: Sumedh Pendurkar;Chris Chow;Luo Jie;Guni Sharon

Task Phasing: Automated Curriculum Learning from Demonstrations

任务阶段化：从演示中自动进行课程学习

• DOI: 10.1609/icaps.v33i1.27235
• 发表时间: 2023
• 期刊: Proceedings of the International Conference on Automated Planning and Scheduling
• 作者: Bajaj, Vaibhav;Sharon, Guni;Stone, Peter
• 通讯作者: Stone, Peter