CHS: Medium: Collaborative Research: Scalable Integration of Data-Driven and Model-Based Methods for Large Vocabulary Sign Recognition and Search

CHS：中：协作研究：用于大词汇量符号识别和搜索的数据驱动和基于模型的方法的可扩展集成

• 批准号: 1763523
• 负责人: Dimitris Metaxas
• 金额: $ 69万
• 依托单位: Rutgers University New Brunswick
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2018
• 资助国家: 美国
• 起止时间: 2018-08-01 至 2022-07-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1763523&HistoricalAwards=false
• 关键词: CHS; Medium; Collaborative; Research; Scalable

It is surprisingly difficult to look up an unfamiliar sign in American Sign Language (ASL). Most ASL dictionaries list signs in alphabetical order based on approximate English translations, so a user who does not understand a sign or know its English translation would not know how to find it. ASL lacks a written form or intuitive "alphabetical sorting" based on such a writing system. Although some dictionaries make available alternative ways to search for a sign, based on explicit specification of various properties, a user must often still look through hundreds of pictures of signs to find a match to the unfamiliar sign (if it is present at all in that dictionary). This research will create a framework that will enable the development of a user-friendly, video-based sign-lookup interface, for use with online ASL video dictionaries and resources, and for facilitation of ASL annotation. Input will consist of either a webcam recording of a sign by the user, or user identification of the start and end frames of a sign from a digital video. To test the efficacy of the new tools in real-world applications, the team will partner with the leading producer of pedagogical materials for ASL instruction in high schools and colleges, which is developing the first multimedia ASL dictionary with video-based ASL definitions for signs. The lookup interface will be used experimentally to search the ASL dictionary in ASL classes at Boston University and RIT. Project outcomes will revolutionize how deaf children, students learning ASL, or families with deaf children search ASL dictionaries. They will accelerate research on ASL linguistics and technology, by increasing efficiency, accuracy, and consistency of annotations of ASL videos through video-based sign lookup. And they will lay the groundwork for future technologies to benefit deaf users, such as search by video example through ASL video collections, or ASL-to-English translation, for which sign-recognition is a precursor. The new linguistically annotated video data and software tools will be shared publicly, for use by others in linguistic and computer science research, as well as in education. Sign recognition from video is still an open and difficult problem because of the nonlinearities involved in recognizing 3D structures from 2D video, and the complex linguistic organization of sign languages. The linguistic parameters relevant to sign production and discrimination include hand configuration and orientation, location relative to the body or in signing space, movement trajectory, and in some cases, facial expressions/head movements. An additional complication is that signs belonging to different classes have distinct internal structures, and are thus subject to different linguistic constraints and require distinct recognition strategies; yet prior research has generally failed to address these distinctions. The challenges are compounded by inter- and intra- signer variations, and, in continuous signing, by co-articulation effects (i.e., influence from adjacent signs) with respect to several of the above parameters. Purely data-driven approaches are ill-suited to sign recognition given the limited quantities of available, consistently annotated data and the complexity of the linguistic structures involved, which are hard to infer. Prior research has, for this reason, generally focused on selected aspects of the problem, often restricting the work to a limited vocabulary, and therefore resulting in methods that are not scalable. More importantly, few if any methods involve 4D (spatio-temporal) modeling and attention to the linguistic properties of specific types of signs. A new approach to computer-based recognition of ASL from video is needed. In this research, the approach will be to build a new hybrid, scalable, computational framework for sign identification from a large vocabulary, which has never before been achieved. This research will strategically combine state-of-the-art computer vision, machine-learning methods, and linguistic modeling. It will leverage the team's existing publicly shared ASL corpora and Sign Bank - linguistically annotated and categorized video recordings produced by native signers - which will be augmented to meet the requirements of this project.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

出人意料的是，很难在美国手语（ASL）上查找一个陌生的标志。大多数ASL词典根据大概的英语翻译按字母顺序排列列出标志，因此不了解标志或知道其英语翻译的用户不知道如何找到它。 ASL缺乏基于此类写作系统的书面形式或直观的“字母排序”。尽管某些词典可以根据各种属性的明确规范提供搜索标志的替代方法，但用户通常仍然必须查看数百个标志的图片，以找到与不熟悉的符号的匹配（如果在该字典中完全存在）。这项研究将创建一个框架，该框架将使用户友好的，基于视频的签名界面的开发，用于在线ASL视频词典和资源，以及促进ASL注释。 输入将包括用户对符号的网络摄像头记录，或者用户对数字视频符号的开始和端帧的识别。为了测试新工具在现实世界应用中的功效，该团队将与高中和大学的ASL教学材料的领先生产商合作，该材料正在开发首个具有基于视频的ASL ASL定义的Multimedia ASL词典。查找界面将在波士顿大学和RIT的ASL课程中进行实验中用于搜索ASL词典。项目成果将彻底改变聋哑儿童，学生学习ASL或有聋哑儿童搜索ASL词典的家庭。他们将通过基于视频的签名查找来提高ASL视频注释的效率，准确性和一致性，从而加速有关ASL语言学和技术的研究。他们将为未来的技术奠定基础，以使聋人用户受益，例如通过ASL视频收集或ASL-To-English翻译进行搜索，而符号识别为先驱。新的语言注释的视频数据和软件工具将公开共享，供其他人在语言和计算机科学研究中以及教育中使用。视频的标志识别仍然是一个开放而困难的问题，因为识别2D视频的3D结构以及符号语言的复杂语言组织所涉及的非线性。与签名生产和歧视相关的语言参数包括手动配置和方向，相对于身体，签名空间，运动轨迹以及在某些情况下，面部表情/头部运动中的位置。另一个并发症是属于不同类别的符号具有不同的内部结构，因此受到不同的语言约束，需要独特的识别策略。然而，先前的研究通常未能解决这些区别。这些挑战是由签名内和签名内变化加重的，并且在连续签名中，与上述几个参数相对于上述几个参数而通过共同引起效应（即来自相邻符号的影响）。鉴于可用数量有限的可用数据和所涉及的语言结构的复杂性，纯粹的数据驱动方法是不适合识别识别的，因此很难推断。因此，先前的研究通常集中在问题的某些方面上，通常将工作限制在有限的词汇上，因此导致无法扩展的方法。更重要的是，很少有任何方法涉及4D（时空）建模和对特定类型符号的语言特性的关注。需要从视频中对基于计算机的ASL识别的新方法。在这项研究中，该方法将是建立一个新的混合，可扩展的计算框架，以从大型词汇中识别标志，从未实现。这项研究将在战略上结合最先进的计算机视觉，机器学习方法和语言建模。它将利用该团队现有的公开共享的ASL Corpora和Sign Bank - 语言注释和分类由本地签名者制作的录像带 - 将进行增强以满足该项目的要求。该奖项反映了NSF的法定任务，并认为使用基金会的智力功能和广泛的影响来评估CRETERIA CREITERIA。

项目成果

Isolated Sign Recognition using ASL Datasets with Consistent Text-based Gloss Labeling and Curriculum Learning

• 发表时间: 2022
• 作者: Konstantinos M. Dafnis;Evgenia Chroni;C. Neidle;Dimitris N. Metaxas

American Sign Language Video Anonymization to Support Online Participation of Deaf and Hard of Hearing Users

• DOI: 10.1145/3441852.3471200
• 发表时间: 2021-10
• 期刊: Proceedings of the 23rd International ACM SIGACCESS Conference on Computers and Accessibility
• 作者: Sooyeon Lee;Abraham Glasser;Becca Dingman;Zhaoyang Xia;Dimitris N. Metaxas;C. Neidle;Matt Huenerfauth