Harnessing multimodal data to enhance machine learning of children’s vocalizations

利用多模态数据增强儿童发声的机器学习

基本信息

批准号：
10411575
负责人：
DANIEL S MESSINGER
金额：
$ 20万
依托单位：
UNIVERSITY OF MIAMI CORAL GABLES
依托单位国家：
美国
项目类别：
财政年份：
2021
资助国家：
美国
起止时间：
2021-02-01 至 2026-01-31
项目状态：
未结题

来源：
https://reporter.nih.gov/project-details/10411575
关键词：
Administrative Supplement Adult Affect African American African Caribbean Age Algorithms Archives Asians Benchmarking Cells Characteristics Child Child Development Child Language Child Support Cochlear implant procedure Code Complex Computer Models Computer Vision Systems Computer software Consent Data Data Collection Data Set Databases Detection Development Dimensions Environment Equilibrium Exhibits Exposure to Face Feedback Female Fostering Frequencies Funding Gardenal Gaussian model Goals Hearing Hispanics Hour Human Individual Language Language Development Learning Left Legal patent Life Linguistics Link Location Machine Learning Manuscripts Measurement Measures Metadata Modeling Modernization Movement Nursery Schools Output Parents Participant Pattern Persons Play Positioning Attribute Postdoctoral Fellow Predictive Factor Privacy Probability Procedures Process Production Publications Pythons Radial Radio Randomized Reporting Research Research Personnel Research Project Grants Resources Sampling Series Sex Distribution Shoulder Signal Transduction Social Development Source Speech Speech Sound Statistical Models Stream System TensorFlow Testing Time TimeLine Training United States National Institutes of Health Walkers Weight automated analysis base computerized data processing contextual factors convolutional neural network cost data analysis pipeline data de-identification data integration data pipeline deafness deep learning deep neural network demographics denoising design dyadic interaction educational atmosphere experience feeding functional outcomes hearing impairment improved interactive feedback interest light weight machine learning algorithm male markov model meter multi-ethnic multimodal data multimodality multiple datasets open data open source parent grant peer prediction algorithm prototype radio frequency recurrent neural network recursive neural network repository response sensor signal processing social socioeconomics sound speech processing supervised learning supplemental instruction teacher tool undergraduate student vocalization

项目摘要

Project Summary This Administrative Supplement proposes implementation of a multimodal data pipeline to support machine learning of child language production in complex naturalistic environments. The Supplement builds on the parent R01 (DC018542) that gathers objective, longitudinal data to capture the vocal interactions of children with hearing loss (HL). Even with cochlear implantation, HL is a life-altering condition with high social costs. Inclusion of children with HL and typically hearing (TH) peers in preschool classrooms is a national standard, but it is not clear how early vocal interaction contributes to the language development of children with HL and their TH peers. The parent R01 employs computational models of child location and orientation to indicate when children are in social contact with their peers and teachers. An additional strategy for pursuing the broad goals of the R01— identifying interactive contexts in which children produce phonemically complex vocalizations and interactive speech—is machine learning. Machine learning algorithms can determine the contextual, individual, and interactive factors that predict children’s vocalizations and vocal interactions. However, the parent R01 does not propose machine learning, nor are data disseminated in a format designed to facilitate machine learning. To facilitate machine learning in the classroom, a rigorous diarization process is required to determine speaker identity, which is operationalized as the likelihood that each vocalization was spoken by a given child or teacher. We will integrate audio processing of each target child and teacher’s first-person audio recording with processing of their interactive partners’ recordings. The influence of partner recordings will be determined by their physical distance and orientation relative to the target. This will yield a weighted speaker identification score for each vocalization. For 25% of the sample, the algorithmic score will be compared to speaker identification provided by trained coders to quantify intersystem reliability. Processed datasets will include 7,160 hours of multimodal recordings of child and teacher movement in classrooms synchronized with continuously recorded, child- and teacher-specific (first-person) audio recordings. De-identified output data will characterize vocalizations with respect to algorithmically computed speaker identification probabilities, coder-identified speaker identity (25% of sample), phonemic complexity and audio characteristics (e.g., fundamental frequency), as well as the position and relative orientation of all individuals in the classroom, and child demographics (including characterizations of HL). Over the course of the supplement, output data, Python processing code, and metadata descriptions of the processing pipeline will be disseminated in dedicated distribution portals including Github, Kaggle, and the UCI repository. Recordings will be released to certified investigators via NIH-funded repositories such as Databrary and Homebank.

项目摘要这种行政补充提案实施多模式数据管道以支持在复杂的自然主义环境中儿童语言生产的机器学习。补充建立在父r01（DC018542）的基础上，该纵向数据收集了纵向数据听力损失（HL）的儿童相互作用。即使有耳蜗植入，HL还是改变了生活的社会成本较高的条件。包括HL的儿童，通常会听到（TH）同龄人学龄前教室是国家标准 HL及其TH同龄人的儿童的语言发展。父母R01员工儿童位置和方向的计算模型，以指示儿童何时处于社交接触与他们的同龄人和老师。追求R01的广泛目标的另一个策略识别儿童产生语音复杂的发声和互动语音 - 是机器学习。机器学习算法可以确定上下文，个人和互动因素可以预测儿童的发声和声音互动。然而，父r01不建议机器学习，也没有以旨在的格式传播数据促进机器学习。为了促进教室中的机器学习，严格的腹部化确定说话者身份需要过程，该身份被操作为每个的可能性给定的孩子或老师说话。我们将整合每个目标的音频处理通过处理互动伙伴的录音，儿童和老师的第一人称录音。伴侣记录的影响将取决于其身体距离和方向相对的影响到目标。这将为每种发声提供加权扬声器的标识评分。 25％样本，将算法得分与训练有素的编码人员提供的说话者识别与量化间隔系统的可靠性。处理后的数据集将包括7,160小时的多模式记录教室中的儿童和教师运动与连续记录，儿童和特定于教师的（第一人称）录音。去识别的输出数据将表征发声关于算法计算的扬声器识别可能性，编码器识别的扬声器身份（样本的25％），音素复杂性和音频特征（例如，基本频率），以及教室中所有个人的位置和相对取向，以及孩子人口统计（包括HL的字符）。在补充剂的过程中，输出数据， Python处理代码和处理管道的元数据描述将在专用的分销门户包括GitHub，Kaggle和UCI存储库。录音将是通过NIH资助的存储库（例如Databrary和Homebank）向认证的调查员发行。