HCC: Small: Collaborative Research: Real-Time Captioning by Groups of Non-Experts for Deaf and Hard of Hearing Students

HCC：小型：协作研究：由非专家小组为聋哑和听力障碍学生提供实时字幕

• 批准号: 1446129
• 负责人: Jeffrey Bigham
• 金额: $ 31.72万
• 依托单位: Carnegie-Mellon University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2014
• 资助国家: 美国
• 起止时间: 2014-01-31 至 2016-07-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1446129&HistoricalAwards=false
• 关键词: HCC; Small; Collaborative; Research; Real; HCC; Small; Collaborative; Research; Real

Many deaf and hard of hearing students use real-time captioning to participate in education. Generally, real-time captions are provided by skilled professional captionists (stenographers) who use specialized keyboards or software to keep up with natural speaking rates of up to 225 words per minute. But professional captionists are expensive and must be arranged in advance in blocks of at least an hour. Automatic speech recognition (ASR) is improving, but still experiences high error rates in real classrooms. In this collaborative effort involving the University of Rochester and Rochester Institute of Technology, the PIs will address these issues by blending human- and machine-powered captioning to produce captions on demand, in real time, for low cost. The PIs' approach is for multiple non-experts and ASR to collectively caption speech in under 5 seconds, with the help of interfaces which encourage quick, incomplete captioning of live audio. Because non-experts cannot keep up with natural speaking rates, new algorithms will merge incomplete captions in real time. (While the sequence alignment problem can be solved exactly with dynamic programming, existing approaches are too slow, are not robust to input error, and do not incorporate natural language semantics.) Systematically varying audio saliency will encourage complete coverage of speech. Non-expert captions will train ASR engines in real time, so that ASR may improve during a lecture. (Traditional approaches for ASR training assume that training occurs offline.) The quikCaption mobile application will embody these ideas and will be iteratively designed with deaf and hard of hearing students at the National Technical Institute of the Deaf (NTID) via design sessions, lab studies and in-class deployments. Non-expert captionists can be drawn from broad sources: volunteers willing to donate their time, classmates with relevant domain knowledge, or always-available paid workers. They may be local (in the classroom) or remote. Captionists may have experience from prior quikCaption sessions, or novice crowd workers recruited on demand from existing marketplaces (e.g., Mechanical Turk). A flexible worker pool will allow real-time captions to be available on demand at low cost and for only as long as needed.Broader Impacts: This research will dramatically improve education for deaf and hard of hearing students by enabling access to serendipitous opportunities, such as conversations after class or last-minute guest lectures for which no interpreter or captionist was arranged. Real-time captioning will also be useful in other settings such as school programs, artistic performances, and political events. Older hard of hearing adults usually prefer captioning, and represent a sizable and growing population; hearing people may benefit because captioning is a first step in automatic translation of aural speech. The algorithms developed as part of this project for real-time merging of incomplete natural language will likely be adaptable for other applications such as collaborative translation or communication over noisy mediums.

许多聋哑和听力的障碍学生使用实时字幕参加教育。 通常，实时字幕由熟练的专业字幕主义者（速记员）提供，他们使用专门的键盘或软件来跟上每分钟最多225个单词的自然语言。 但是专业的字幕主义者很昂贵，必须在至少一个小时内提前安排。 自动语音识别（ASR）正在改善，但在真实教室中仍然经历了很高的错误率。 在涉及罗切斯特大学和罗切斯特理工学院的这项合作努力中，PIS将通过将人为和机器的字幕融合以低成本来解决这些问题。 PIS的方法是针对多个非专家和ASR，可以在不到5秒钟内集体字幕演讲，并借助界面，鼓励快速，不完整的实时音频字幕。 由于非专家无法跟上自然的说话率，因此新算法将实时合并不完整的字幕。 （虽然可以通过动态编程来准确解决序列对齐问题，但现有方法太慢，无法强大而无法输入错误，并且不纳入自然语言语义。）系统地改变音频显着性将鼓励对语音进行完全覆盖。 非专家字幕将实时训练ASR发动机，以便在演讲中可以改善ASR。 （ASR培训的传统方法假设培训是离线发生的。）QuikCaption移动应用程序将体现这些想法，并通过设计课程，实验室研究和课堂部署在国家聋人国家技术研究所（NTID）上进行聋哑和听力的学生进行迭代设计。 可以从广泛的来源中汲取非专家字幕主义者：愿意捐赠时间的志愿者，具有相关领域知识的同学或始终可用的付费工人。 他们可能是本地（在教室中）或遥控器。 字幕主义者可能会从以前的Quikcaption会议中获得经验，或者是从现有市场（例如机械Turk）招募的新手人群工人。 灵活的工人池将允许以低成本的方式按需按需提供实时字幕。Boader的影响：这项研究将通过使能够获得偶然的机会（例如上课后的对话或最后一分钟的客人演讲，没有讲解者或安排的待遇者的讲座），从而极大地改善了对聋人和听力障碍学生的教育。 实时字幕也将在其他环境中有用，例如学校课程，艺术表演和政治活动。 年龄较大的成年人通常更喜欢字幕，代表大量和增长的人口；听力人可能会受益，因为字幕是听觉演讲自动翻译的第一步。 作为该项目开发的算法，用于实时合并不完整的自然语言的一部分，可能会适用于其他应用程序，例如在嘈杂媒介上进行协作翻译或通信。