III: Small: Towards Speech-Driven Multimodal Querying

III：小型：迈向语音驱动的多模式查询

• 批准号: 1816701
• 负责人: Arun Kumar
• 金额: $ 50万
• 依托单位: University of California-San Diego
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2018
• 资助国家: 美国
• 起止时间: 2018-10-01 至 2022-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1816701&HistoricalAwards=false
• 关键词: III; Small; Towards; Speech; Driven

Modern automatic speech recognition (ASR) tools offer near-human accuracy in many scenarios. This has increased the popularity of speech-driven input in many applications on modern device environments such as tablets and smartphones, while also enabling personal conversational assistants. In this context, this project will study a seemingly simple but important fundamental question: how should one design a speech-driven system to query structured data? Structured data querying is ubiquitous in the enterprise, healthcare, and other domains. Typing queries in the Structured Query Language (SQL) is the gold standard for such querying. But typing SQL is painful or impossible in the above environments, which restricts when and how users can consume their data. SQL also has a learning curve. Existing alternatives such as typed natural language interfaces help improve usability but sacrifice query sophistication substantially. For instance, conversational assistants today support queries mainly over curated vendor-specific datasets, not arbitrary database schemas, and they often fail to understand query intent. This has widened the gap with SQL's high query sophistication and unambiguity. This project will bridge this gap by enabling users to interact with structured data using spoken queries over arbitrary database schemas. It will lead to prototype systems on popular tablet, smartphone, and conversational assistant environments. This could help many data professionals such as data analysts, business reporters, and database administrators, as well as non-technical data enthusiasts. For instance, nurse informaticists can retrieve patient details more easily and unambiguously to assist doctors, while analysts can slice and dice their data even on the move. The research will be disseminated as publications in database and natural language processing conferences. The research and artifacts produced will be integrated into graduate and undergraduate courses on database systems. The PIs will continue supporting students from under-represented groups as part of this project.This project will create three new systems for spoken querying at three levels of "naturalness." The first level targets a tractable and meaningful subset of SQL. This research will exploit three powerful properties of SQL that regular English speech lacks--unambiguous context-free grammar, knowledge of the database schema queried, and knowledge of tokens from the database instance queried--to support arbitrary database schemas and tokens not present in the ASR vocabulary. The PIs will synthesize and innovate upon ideas from information retrieval, natural language processing, and database indexing and combine them with human-in-the-loop query correction to improve accuracy and efficiency. The second version will make SQL querying even more natural and stateful by changing its grammar. This will lead to the first speech-oriented dialect of SQL. The third version will apply the lessons from the previous versions to two state-of-the-art typed natural language interfaces for databases. This will lead to a redesign of such interfaces that exploits both the properties of speech and the database instance queried.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

在许多情况下，现代自动语音识别（ASR）工具提供了几乎人类的准确性。这增加了在现代设备环境（例如平板电脑和智能手机）上的许多应用程序中语音驱动输入的普及，同时还可以使个人对话助手。在这种情况下，该项目将研究一个看似简单但重要的基本问题：一个人应该如何设计语音驱动的系统来查询结构化数据？结构化数据查询在企业，医疗保健和其他域中无处不在。在结构化查询语言（SQL）中键入查询是此类查询的金标准。但是在上述环境中，打字SQL是痛苦或不可能的，这限制了用户何时以及如何消耗数据。 SQL还具有学习曲线。现有的替代方法（例如键入的自然语言界面）有助于提高可用性，但牺牲查询的复杂性大大提高。例如，今天的对话助理支持查询主要是策划的供应商特定数据集，而不是任意数据库模式，并且他们通常无法理解查询意图。这已经通过SQL的高查询精致和不兼具扩大了差距。该项目将通过使用户能够使用任意数据库模式的口头查询与结构化数据进行交互来弥合这一差距。它将在流行的平板电脑，智能手机和对话助理环境上产生原型系统。这可以帮助许多数据专业人员，例如数据分析师，业务记者和数据库管理员以及非技术数据爱好者。例如，护士信息家可以更轻松，明确地检索患者的详细信息以帮助医生，而分析师也可以将数据切成薄片和切成薄片。该研究将作为数据库和自然语言处理会议中的出版物传播。所产生的研究和工件将集成到数据库系统上的研究生和本科课程中。作为该项目的一部分，PI将继续为来自代表性不足的群体的学生提供支持。该项目将创建三个新的系统，用于以“自然性”的三个级别进行口头查询。第一个级别针对SQL的可处理且有​​意义的子集。这项研究将利用常规英语语音所缺乏的SQL的三个强大属性 - 不明确的无上下文语法，对数据库架构查询的知识以及数据库实例中的代币知识查询 - 支持任意数据库模式和代币在ASR词汇中不存在。 PI将根据信息检索，自然语言处理和数据库索引的想法进行综合和创新，并将其与人类的查询校正结合在一起，以提高准确性和效率。第二版将通过更改语法使SQL查询更加自然和状态。这将导致SQL的第一个面向语音的方言。第三个版本将将以前版本的课程应用于数据库的两个最先进的自然语言接口。这将导致对此类接口的重新设计，这些界面既利用语音的属性又征询了数据库实例。该奖项反映了NSF的法定任务，并认为使用基金会的知识分子和更广泛的影响审查标准，认为值得通过评估来获得支持。

项目成果

Demonstration of SpeakQL: Speech-driven Multimodal Querying of Structured Data

SpeakQL 演示：语音驱动的结构化数据多模态查询

• DOI: 10.1145/3299869.3320224
• 发表时间: 2019
• 期刊: Proceedings of the 2019 International Conference on Management of Data
• 作者: Shah, Vraj;Li, Side;Yang, Kevin;Kumar, Arun;Saul, Lawrence
• 通讯作者: Saul, Lawrence

SpeakQL: Towards Speech-driven Multimodal Querying of Structured Data

• DOI: 10.1145/3318464.3389777
• 发表时间: 2020-05
• 期刊: Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data
• 作者: Vraj Shah;Side Li;Arun Kumar;L. Saul

Structured Data Representation in Natural Language Interfaces

• 发表时间: 2022
• 期刊: IEEE Data Eng. Bull.
• 作者: Yutong Shao;Arun Kumar;Ndapandula Nakashole