Robot Learning and Discovering Through Memorizing Visual and Auditory Interactions

机器人通过记忆视觉和听觉交互来学习和发现

• 批准号: RGPIN-2022-04036
• 负责人: Michaud, François
• 金额: $ 3.5万
• 依托单位: Université de Sherbrooke
• 依托单位国家: 加拿大
• 项目类别: Discovery Grants Program - Individual
• 财政年份: 2022
• 资助国家: 加拿大
• 起止时间: 2022-01-01 至 2023-12-31
• 项目状态: 已结题
• 来源: https://www.nserc-crsng.gc.ca/ase-oro/Details-Detailles_eng.asp?id=754136
• 关键词: Robot; Learning; Discovering; Through; Memorizing

Since its foundation, Artificial Intelligence (AI) aims at becoming a system science targeting working in the wild, messy world, addressing key challenges such as architecture, learning and evaluation. To accomplish this, human-robot interaction (HRI) brings rich opportunities by addressing the integration challenges of embodied AI. The long term objective of my Discovery Grant (DG) research program is the study of how to integrate all the required decision-making processes, ranging from navigation to task accomplishment and interaction with others (robots or humans), so that robots can operate in real life settings over long periods of time. My strategy consists of designing robots with an increasing set of advanced perceptual, reasoning and action capabilities under real-time execution, robustness, adaptability and scalability, capable of being used in real life settings. In the short term, my research program focuses on learning from the use of spatial-temporal visual and auditory patterns to make the robots derive knowledge from its operation environments. Memory plays a central role in recognizing patterns and predicting upcoming percepts and action consequences, which is a key feature and critical component of intelligence. Building on my work on vision-based navigation and episodic memory models, along with cross-referenced information from visual and audio data (using spatial auditory data linked to visual data), I will study how to increase the ability of an autonomous robot to derive concepts, predict future events and elaborate behavioral strategies, by operating in dynamic conditions and interacting with people. Deep Neural Networks (DNNs) reveal to be powerful tools to learn the complexities of visual and audio data, which can be beneficial in providing cues and indications of what could be interesting elements in what the robot is experiencing. Using DNNs designed to detect objects, people, faces, pose, sounds and voice, I will use different representations/models to memorize the interaction history from which to derive knowledge and understanding. Experimentation involves conducting trials in long-lasting HRI scenarios using commercially available and custom-designed robots. Involving 2 postdoc, 2 PhDs, 2 Master's and 2 undergraduates, my research program integrates a large set of components, from vision and audio processing to robot control architecture, memory models, decision-making processes and HRI, with contributions to the fields of visuo-auditory cognition and semantic interpretation of embodied multimodal interaction. Its impacts range from applications on factory assembly lines to healthcare, rehabilitation and aging, and surveillance. All my research contributions funded by the DG program are open source, allowing sharing and contributing to the joint effort of bringing robots closer to people, improving quality of life and our understanding and ability of designing truly intelligent robots.

自成立以来，人工智能（AI）的目标是成为一门系统科学，针对在狂野、混乱的世界中工作，解决架构、学习和评估等关键挑战。为了实现这一目标，人机交互 (HRI) 通过解决嵌入式人工智能的集成挑战带来了丰富的机遇。 我的发现补助金（DG）研究计划的长期目标是研究如何整合所有必需的决策过程，从导航到任务完成以及与其他人（机器人或人类）的交互，以便机器人可以在长时间的现实生活设置。我的策略包括设计机器人，使其在实时执行、鲁棒性、适应性和可扩展性下具有越来越多的先进感知、推理和行动能力，能够在现实生活中使用。 在短期内，我的研究计划侧重于利用时空视觉和听觉模式进行学习，使机器人从其操作环境中获取知识。记忆在识别模式和预测即将到来的感知和行动后果方面发挥着核心作用，这是智力的关键特征和关键组成部分。基于我在基于视觉的导航和情景记忆模型方面的工作，以及来自视觉和音频数据的交叉引用信息（使用与视觉数据链接的空间听觉数据），我将研究如何提高自主机器人推导的能力通过在动态条件下操作并与人互动，概念、预测未来事件并制定行为策略。 深度神经网络（DNN）被证明是学习视觉和音频数据复杂性的强大工具，这有助于提供机器人正在经历的有趣元素的线索和指示。使用旨在检测物体、人、面部、姿势、声音和语音的 DNN，我将使用不同的表示/模型来记住交互历史，从中获取知识和理解。实验包括使用商用和定制设计的机器人在持久的 HRI 场景中进行试验。 我的研究项目涉及2名博士后、2名博士、2名硕士和2名本科生，集成了从视觉和音频处理到机器人控制架构、记忆模型、决策过程和HRI的大量组件，在视觉和视觉领域做出了贡献-具体多模​​态交互的听觉认知和语义解释。其影响范围从工厂装配线的应用到医疗保健、康复和老龄化以及监控。我由 DG 计划资助的所有研究贡献都是开源的，允许分享并为让机器人更接近人类、提高生活质量以及我们设计真正智能机器人的理解和能力的共同努力做出贡献。