Collaborative Research: CSL-MultiAD: Assessing Collaborative STEM Learning through Rich Information Flow based on Multi-Sensor Audio Diarization

协作研究：CSL-MultiAD：通过基于多传感器音频二值化的丰富信息流评估协作 STEM 学习

• 批准号: 1918012
• 负责人: Dwight Irvin
• 金额: $ 30万
• 依托单位: University of Kansas Center for Research Inc
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-08-01 至 2023-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1918012&HistoricalAwards=false
• 关键词: Collaborative; Research; CSL; MultiAD; Assessing

The ability to learn concepts, especially for science and math (STEM) based disciplines, is impacted by educators who inspire, motivate, and create supportive environments and teaching methodologies which lower the entry barrier for students learning STEM subjects. Teaching resources nationwide have historically been constrained as STEM based science content for education expands with increasing student diversity based on prior science exposure in the classroom. A key aspect of student learning is to assess the quality of human communications between student-and-student as well as teacher-and-student. In STEM learning, students who are able to ask the right questions, know what they understand as well as what they need help with, allows educators to structure their teaching methods to help students overcome learning challenges. However, to date, it has been virtually impossible to collect and measure student-to-student or student-to-teacher voice communications in the classroom. Also, current speech technology is not sufficiently effective to overcome multi-speaker and naturalistic communications in classrooms. This project will develop classroom audio collection and measurement tools for students working together to solve problems, as well as teacher involvement with individual/groups of students. The audio collection solution includes both individual recorders on a sub-set of classroom students, as well as central smart speaker microphone collection units within each student group. Computer programs will be developed to analyze who is speaking and when, as well as spot keywords of interest for STEM topics and learning assessment. Privacy is maintained, since audio analysis is focused on high level measures such as individual student word counts, anonymous tagging of each speaker, and connecting conversational turns between students and teachers. A teacher driven keyword set will be used to help measure which students are having problems understanding concepts. These individual communication measured terms will be integrated into a dashboard display, to empower teachers with easy to use feedback on student engagement for STEM learning. The project has the potential to improve the ability to assess learning through classroom communications, and potentially help teachers better direct their time/expertise more efficiently to improve STEM learning for students. This project will develop ways to assess learning in classrooms by measuring the quality of human communication engagement between students-and-peers as well as teachers-and-students. Research has shown that learning is improved if there is dynamic interaction between student-to-student and student-to-teacher in voice communications. The project introduces personal recorders in the classroom to capture voice interactions during the entire day. Next, these multi-microphone recording streams are pooled together, where speech and language processing algorithms will be formulated to perform "audio diarization" - the process of determining "who spoke, what, and when", with potential keywords of interest based on classroom topics identified. The diarization output will drive the formulation of metrics to assess communication engagement. Communication based features derived from individual audio streams (word count, talk time, turn-taking, keyword profile) will be extracted on a per student basis through audio diarization. Next, this information flow will be used to develop class based group dynamics. This solution represents an approach for teachers to monitor student engagement over time in science activity areas, helping teachers identify students who are not verbally engaged in science discourse and quickly assess the impact of changes in classroom practices to improve learning. A number of technology challenges will be addressed for automatic audio stream based voice processing of naturalistic audio data using speech activity detection, speaker diarization based on machine learning models, and keyword spotting for science topic identification and tracking. These research aims will be assessed in classroom settings with teacher feedback on the effectiveness of the resulting solutions. The resulting speech technology advancements would offer new opportunities for future smart classrooms for voice assessment for teachers to better assess student involvement in science vs. infrequent traditional standardized testing. Ultimately, this effort will equip teachers with tools to identify and frequently monitor early indicators of disengagement in science learning, and potentially increase science interest by under-represented student populations and further diversify the STEM workforce.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

学习概念的能力，尤其是基于科学和数学（STEM）的学科，受到教育者的影响，这些教育者会激发，激励和创造支持性环境和教学方法论，这些方法降低了学生学习STEM学科的入学障碍。历史上，全国教学资源受到基于STEM的科学内容的限制，教育的基于STEM的科学内容会随着课堂上的先前科学曝光而增加的学生多样性。学生学习的一个关键方面是评估学生和学生之间的人类交流质量以及教师和学生之间的质量。在STEM学习中，能够提出正确的问题，了解他们理解的知识以及他们需要帮助的学生，使教育工作者能够构建他们的教学方法，以帮助学生克服学习挑战。但是，迄今为止，几乎不可能在课堂上收集和衡量学生对学生或老师的语音交流。同样，当前的语音技术不足以克服教室中的多扬声器和自然主义交流。 该项目将为学生共同解决问题的学生开发课堂音频收集和测量工具，并与个人/组学生一起参与。音频收集解决方案包括课堂学生子集中的单个录音机，以及每个学生组中的中央智能扬声器麦克风收集单元。将开发计算机程序来分析谁在讲话，何时以及STEM主题和学习评估感兴趣的点关键字。维护隐私，因为音频分析集中在高级措施上，例如单个学生的单词计数，每个说话者的匿名标签以及在学生和老师之间连接对话转弯。教师驱动的关键字集将用于帮助衡量哪些学生在理解概念方面遇到问题。这些单独的沟通术语将集成到仪表板显示中，以赋予教师的权力，并易于使用有关学生参与的STEM学习的反馈。该项目有可能提高通过课堂沟通评估学习的能力，并有可能帮助教师更好地指导他们的时间/专业知识，从而改善学生的STEM学习。该项目将通过衡量学生和同学之间的人类交流质量以及教师和学生之间的沟通质量来开发评估课堂学习的方法。研究表明，如果在语音交流中，学生对学生和学生对教师之间存在动态互动，则学习将得到改善。该项目在课堂上介绍了个人录音机，以捕获整天的语音互动。接下来，将这些多微粒录制流汇总在一起，在这些记录流中，语音和语言处理算法将被制定为执行“音频诊断” - 确定“谁讲话，什么和何时”的过程，以及基于确定的课堂主题的潜在关键字。诊断输出将推动指标的制定以评估沟通参与度。将通过音频诊断来提取从单个音频流（单词计数，谈话时间，转弯，关键字配置文件）得出的基于沟通的功能。接下来，此信息流将用于开发基于类的组动力学。该解决方案代表了一种方法，教师可以在科学活动领域监测学生的参与度，从而帮助教师识别不从事科学话语的学生，并迅速评估课堂实践变化以改善学习的影响。使用语音活动检测，基于机器学习模型的扬声器诊断以及用于科学主题识别和跟踪的关键字发现，将针对自动音频流的自动语音处理来解决许多技术挑战。这些研究的目标将在课堂环境中进行评估，并通过教师反馈所得解决方案的有效性。由此产生的语音技术进步将为未来的智能课堂提供新的机会，以评估教师的语音评估，以更好地评估学生参与科学与罕见的传统标准化测试。 最终，这项工作将使教师提供工具，以识别和经常监控科学学习中脱离接触的早期指标，并通过代表性不足的学生人群可能增加科学的兴趣，并进一步使STEM劳动力多样化。该奖项反映了NSF的法定任务，并通过评估该基金会的智力功能和广泛的影响来评估NSF的法定任务，并被认为是值得的。

项目成果

Speech and language processing for assessing child–adult interaction based on diarization and location

基于分类和位置评估儿童与成人互动的语音和语言处理

• 发表时间: 2019
• 期刊: International journal of speech technology
• 作者: John H. L. Hansen, Maryam Najafian