RI: Small: A Cognitive Framework for Technical, Hard and Explainable Question Answering (THE-QA) with respect to Combined Textual and Visual Inputs

RI：小：结合文本和视觉输入的技术性、硬性和可解释性问答 (THE-QA) 的认知框架

基本信息

批准号：
1816039
负责人：
Chitta Baral
金额：
$ 50万
依托单位：
Arizona State University
依托单位国家：
美国
项目类别：
Standard Grant
财政年份：
2018
资助国家：
美国
起止时间：
2018-08-01 至 2024-07-31
项目状态：
已结题

来源：
https://www.nsf.gov/awardsearch/showAward?AWD_ID=1816039&HistoricalAwards=false
关键词：
RI Small Cognitive Framework Technical

项目摘要

Understanding of visual and textual inputs are important aspects of Artificial Intelligence systems. Often such inputs are presented together to instruct and explain. As examples, an intelligent robot might learn about its tasks and environment by observing both language and gesture; and an intelligent system addressing scientific questions must interpret figures and diagrams along with text. While there has been a lot of research concerning visual understanding and textual understanding in isolation, there has been very little research that addresses them jointly. This project is developing a framework for answering hard questions about combined visual and textual inputs, and providing supporting explanations. By developing a system that integrates visual and linguistic information for this task, the project could provide the basis for automated tutoring systems in K-12 education, and interpretable interfaces for the workers operating intelligent machines. The project will employ an integrated approach of deep model-based visual recognition and natural language processing, and knowledge representation and reasoning to develop a question answering engine and its components. It will create a challenge corpus that has visual and textual inputs and questions about those inputs given in natural language. It will provide a baseline for semantic image and text parsing and reasoning-based question answering systems. It will develop semantic parsing of non-continuous text items, such as figures, diagrams, and graphs. It will enhance semantic parsing to various formats of natural language text and questions. It will develop methods to acquire knowledge and reasoning with them for answering questions and providing explanations to the answers. Together these contributions of the project will advance Artificial General Intelligence and allow future service robots and personal mobile applications to understand combined visual and textual inputs. The findings from this project will advance the development of knowledge-driven, reasoning-based question answering by filling the current gap on how to efficiently conduct explainable probabilistic reasoning over deep models. This helps to overcome the fragility of the trained visual and textual understanding models. It will also uncover the intrinsic connections between deep model-based vision and language understanding algorithms and probabilistic knowledge representation and reasoning by exploring a joint solution for answering the hard questions. In general, this project may result in advances in multiple sub-fields of Artificial Intelligence; namely, computer vision, natural language processing, and question answering; and may impact others such as robotics.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

对视觉和文本输入的理解是人工智能系统的重要方面。通常，这些输入会一起呈现以进行指导和解释。例如，智能机器人可以通过观察语言和手势来了解其任务和环境；解决科学问题的智能系统必须能够解释图形和图表以及文本。虽然已经有很多关于视觉理解和文本理解的独立研究，但很少有研究将它们结合起来。该项目正在开发一个框架，用于回答有关视觉和文本输入相结合的难题，并提供支持性解释。通过为此任务开发一个集成视觉和语言信息的系统，该项目可以为 K-12 教育中的自动辅导系统以及操作智能机器的工人的可解释界面提供基础。该项目将采用基于深度模型的视觉识别和自然语言处理以及知识表示和推理的集成方法来开发问答引擎及其组件。它将创建一个挑战语料库，其中包含视觉和文本输入以及有关以自然语言给出的这些输入的问题。它将为语义图像和文本解析以及基于推理的问答系统提供基线。它将开发非连续文本项的语义解析，例如图形、图表和图表。它将增强对各种格式的自然语言文本和问题的语义解析。它将开发获取知识和推理的方法，以回答问题并对答案提供解释。该项目的这些贡献将共同推进通用人工智能，并允许未来的服务机器人和个人移动应用程序理解组合的视觉和文本输入。该项目的研究结果将填补当前如何在深度模型上有效进行可解释的概率推理的空白，从而推动知识驱动、基于推理的问答的发展。这有助于克服经过训练的视觉和文本理解模型的脆弱性。它还将通过探索回答难题的联合解决方案，揭示基于深度模型的视觉和语言理解算法与概率知识表示和推理之间的内在联系。总体而言，该项目可能会带来人工智能多个子领域的进步；即计算机视觉、自然语言处理和问答；该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。