Collaborative Research: Quantifying sign reduction in sign language using human pose estimation

合作研究：使用人体姿势估计量化手语中的符号减少

• 批准号: 2234787
• 负责人: Naomi Caselli
• 金额: $ 20.81万
• 依托单位: Trustees of Boston University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2023
• 资助国家: 美国
• 起止时间: 2023-09-01 至 2026-08-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2234787&HistoricalAwards=false
• 关键词: Collaborative; Research; Quantifying; sign; reduction

The project investigates the phenomenon of lexically conditioned phonetic variation in sign language. This refers to how the pronunciation of a sign can vary based on the properties of signs, such as frequency of occurrence or the existence of similar-sounding signs (referred to as phonological neighbors) in the mental lexicon. In spoken languages, words that are used frequently may be phonetically reduced, that is, articulated with shorter, contracted vowel space as compared to low-frequency words. On the other hand, words with many phonological neighbors may be phonetically enhanced, e.g., produced with expanded vowel space compared to more isolated words. The articulators used to produce speech (the vocal tract) are fundamentally different from those used to produce signs (hands, face, body). This modality difference may lead to distinct predictions about phonetic variation. Limited research has been done on how these principles apply in signed languages, primarily due to a lack of large-scale machine-readable sign language datasets and adequate techniques to extract phonetic measurements from signed videos.The researchers leverage computer vision and the lexical database for a sign language that they developed under prior National Science Foundation (NSF) support, to analyze patterns of phonetic reduction in over 100,000 videos of signs produced by deaf signers. Specifically, they use human pose estimation techniques to extract signers’ anatomical landmarks and joint positions from the videos and transform the estimates into meaningful phonetic measurements, such as dispersion of the signers’ hands in the signing space. By examining the interactions between the underlying and surface properties of signs, this research provides insights into how the linguistic structure of sign languages affects sign articulation. By understanding the phonetic properties of signs, this research facilitates the development of sign recognition technologies, benefiting the signing community.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该项目研究手语中词汇条件下的语音变化现象，这是指符号的发音如何根据符号的属性（例如出现频率或相似发音符号的存在）而变化（称为语音符号）。在口语词典中，经常使用的单词可能会在语音上减少，即与低频单词相比，使用更短、收缩的元音空间来发音。许多语音邻居可能会在语音上得到增强，例如，与产生更孤立的单词相比，用于产生更多孤立单词的发音器与用于产生符号（手，脸，身体）的发音器根本不同。模态差异可能会导致对语音变化的不同预测，但对于这些原则如何应用于手语语言的研究有限，这主要是由于缺乏大规模机器可读的手语数据集和从手语中提取语音测量的适当技术。研究人员利用计算机视觉和他们在美国国家科学基金会 (NSF) 的支持下开发的手语词汇数据库来分析聋人手语者制作的超过 100,000 个手语视频中的语音缩减模式。具体来说，他们使用了人类手语。姿势估计技术从视频中提取手语者的解剖标志和关节位置，并将估计值转换为有意义的语音测量，例如通过检查手语的底层和表面属性之间的相互作用，手语者的手在手语空间中的分散情况。这项研究深入了解了手语的语言结构如何影响手语发音。通过了解手语的语音特性，这项研究促进了手语识别技术的发展，使手语社区受益。该奖项反映了 NSF 的法定使命和使命。通过使用基金会的智力价值和更广泛的影响审查标准进行评估，该项目被认为值得支持。