RI: Small: Modeling Multiple Modalities for Knowledge-Base Construction

RI：小型：知识库构建的多种模式建模

• 批准号: 1817183
• 负责人: Sameer Singh
• 金额: $ 44.8万
• 依托单位: University of California-Irvine
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2018
• 资助国家: 美国
• 起止时间: 2018-08-01 至 2022-07-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1817183&HistoricalAwards=false
• 关键词: RI; Small; Modeling; Multiple; Modalities

Information-rich documents are prevalent in many domains such as news articles, social media posts, online retail pages, healthcare records, financial reports, and scientific papers. Automatically extracting knowledge from such documents is useful in many applications, such as for answering questions, searching the web, automated dialogs, and analyzing trends. Existing machine learning methods focus only on the text in the documents, and ignore other information sources such as images, tables, and numbers. Thus, much of the information is not extracted, leading to incomplete knowledge and incorrect conclusions. This research advances research in machine learning and natural language processing to address these problems. With support for accurate extraction and reasoning, this project will pave the way for novel applications to domains with unstructured, multimodal documents.The specific aim of the project is to investigate a novel construction pipeline for knowledge bases, taking the first steps in combining textual and relational evidence with numerical, image, and tabular data. To address the many interconnected challenges therein, the project focuses on two sub-tasks. First, the team will extract new facts about an entity from a document, such as its attributes, by combining the different parts (text, images, and tables). Second, the team will develop models to identify missing relations in graphs that contain multimodal facts. For each task, the project includes plans to introduce new datasets, propose benchmark evaluations, and develop appropriate baselines. Further, the team will build upon recent advances in deep neural encoders to investigate machine learning approaches that learn unified, semantic embeddings to model multimodal data. With these contributions, the project will initiate a body of research in machine learning and natural language processing that uses unstructured multimodal data in all its forms for accurate knowledge extraction.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

信息丰富的文档在许多领域中都普遍存在，例如新闻文章，社交媒体帖子，在线零售页面，医疗保健记录，财务报告和科学论文。从此类文档中自动提取知识在许多应用程序中很有用，例如回答问题，搜索网络，自动对话框和分析趋势。现有的机器学习方法仅关注文档中的文本，而忽略其他信息源，例如图像，表格和数字。因此，许多信息未提取，导致知识不完整和结论不正确。这项研究推进了机器学习和自然语言处理的研究，以解决这些问题。为了支持准确的提取和推理，该项目将为具有非结构化的多模式文档的域名铺平道路。该项目的具体目的是研究知识基础的新型构建管道，采取了将文本和关系证据与数字，图像和图像和表格数据相结合的第一步。为了应对其中的许多相互联系的挑战，该项目着重于两个子任务。首先，团队将通过组合不同的部分（文本，图像和表格）来从文档（例如其属性）中提取有关实体的新事实。其次，团队将开发模型，以识别包含多模式事实的图表中的缺失关系。对于每个任务，该项目包括介绍新数据集，提出基准评估并开发适当基准的计划。此外，该团队将基于深度神经编码者的最新进展，以研究学习统一的语义嵌入以建模多模式数据的机器学习方法。有了这些贡献，该项目将启动机器学习和自然语言处理方面的研究，该研究使用其所有形式的非结构化多模式数据进行准确的知识提取。该奖项反映了NSF的法定任务，并被认为是通过基金会的知识分子优点和更广泛的影响审查标准来通过评估来获得支持的。

项目成果

Tweeki: Linking Named Entities on Twitter to a Knowledge Graph

• DOI: 10.18653/v1/2020.wnut-1.29
• 发表时间: 2020-11
• 期刊: ArXiv
• 作者: Bahareh Harandizadeh;Sameer Singh

Knowledge Enhanced Contextual Word Representations

• DOI: 10.18653/v1/d19-1005
• 发表时间: 2019-09
• 期刊: ArXiv
• 作者: Matthew E. Peters;Mark Neumann;IV RobertL.Logan;Roy Schwartz;Vidur Joshi;Sameer Singh;Noah A. Smith-

Enforcing Consistency in Weakly Supervised Semantic Parsing

增强弱监督语义解析的一致性

• DOI: 10.18653/v1/2021.acl-short.22
• 发表时间: 2021
• 期刊: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers
• 作者: Gupta, Nitish;Singh, Sameer;Gardner, Matt
• 通讯作者: Gardner, Matt

Embedding Multimodal Relational Data for Knowledge Base Completion

• DOI: 10.18653/v1/d18-1359
• 发表时间: 2018-09
• 期刊: ArXiv
• 作者: Pouya Pezeshkpour;Liyan Chen;Sameer Singh

Barack’s Wife Hillary: Using Knowledge Graphs for Fact-Aware Language Modeling

• DOI: 10.18653/v1/p19-1598
• 发表时间: 2019-06
• 期刊: ArXiv
• 作者: IV RobertL.Logan;Nelson F. Liu;Matthew E. Peters;Matt Gardner;Sameer Singh