CAREER: Sign-to-Speech: An Edge-IoT Platform and Software Library for Real Time Sign Language Recognition

职业：手语转语音：用于实时手语识别的边缘物联网平台和软件库

• 批准号: 2046972
• 负责人: Mahanth Gowda
• 金额: $ 50万
• 依托单位: Pennsylvania State Univ University Park
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-03-01 至 2026-02-28
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2046972&HistoricalAwards=false
• 关键词: CAREER; Sign; Speech; Edge; IoT

This project will advance the state-of-the-art in cross-disciplinary areas including motion signal processing, machine learning (ML), sign language modeling, and real-time ML with dynamic device/edge partitioning, to develop new technology for automatic Sign Language Recognition (SLR) and translation to spoken language that enables more seamless communication between deaf and hearing people. The technology will incorporate wearable devices (such as a smartwatch, smart ring, and earphones) that are gaining in popularity, and will have broad impact through its introduction in deaf communities along with a sign language equivalent of voice assistants such as Amazon Alexa. The project will establish a pipeline of collaboration with deaf students, as well as courses based on SLR technology that will be disseminated through MOOC platforms such as Coursera. Additional impact will derive from workshops on wearable computing that will be conducted at the K-12 level, and a "sign-to-speech" library that will be publicly released for extensibility of the new technology to multiple sign languages. To achieve its goals this research will include three thrusts: Development of ML models with efficient training that can perform accurate SLR by fusing multimodal input data from wearable devices that capture body motion and facial expressions; Implementation of efficient ML models by means of optimal partitioning between end-device and edge resources to achieve the best tradeoff in real time performance and SLR accuracy; Design of systematic user studies with fluent sign language users both for generating training data for ML models as well as for validation of accuracy, usability, and acceptability of the technology within the deaf community.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该项目将推进跨学科领域的最先进的领域，包括运动信号处理，机器学习（ML），手语建模和具有动态设备/边缘分区的实时ML，以开发新的技术以自动手语言识别（SLR）和转换为口语，以实现聋哑人和听力人之间的无缝通信。该技术将融合越来越受欢迎的可穿戴设备（例如智能手表，智能戒指和耳机），并通过其在聋人社区的介绍以及同等的语言助手（例如亚马逊Alexa）的手语中产生广泛的影响。该项目将与聋哑学生建立合作的管道，以及基于SLR技术的课程，该课程将通过MOOC平台（例如Coursera）进行分发。将在K-12级别进行的可穿戴计算的研讨会以及“标志性语音”库中造成的其他影响，该库将公开发布，以扩展新技术对多种符号语言。为了实现其目标，这项研究将包括三个推力：开发具有高效训练的ML模型，可以通过融合来自可穿戴设备的多模式输入数据来执行精确的SLR，这些数据捕获了身体运动和面部表情；通过在最终设备和边缘资源之间进行最佳分区来实现有效的ML模型，以实现实时性能和SLR准确性的最佳权衡；使用流利的手语用户进行系统的用户研究设计，既可以为ML模型生成培训数据，又要验证聋人社区中技术的准确性，可用性和可接受性。该奖项反映了NSF的法定任务，并被认为是值得通过基金会的知识分子优点和更广泛影响的审查标准来通过评估来通过评估来支持的。

项目成果

One Ring to Rule Them All: An Open Source Smartring Platform for Finger Motion Analytics and Healthcare Applications

• DOI: 10.1145/3576842.3582382
• 发表时间: 2023-05
• 期刊: Proceedings of the 8th ACM/IEEE Conference on Internet of Things Design and Implementation
• 作者: Hao Zhou;Taiting Lu;Yilin Liu;Shijia Zhang;Runze Liu;Mahanth K. Gowda

SignQuery: A Natural User Interface and Search Engine for Sign Languages with Wearable Sensors

• DOI: 10.1145/3570361.3613286
• 发表时间: 2023-10
• 期刊: Proceedings of the 29th Annual International Conference on Mobile Computing and Networking
• 作者: Hao Zhou;Taiting Lu;Kristina Mckinnie;Joseph Palagano;Kenneth DeHaan;Mahanth K. Gowda

Let's Grab a Drink: Teacher-Student Learning for Fluid Intake Monitoring using Smart Earphones

• DOI: 10.1109/iotdi54339.2022.00014
• 发表时间: 2022-05
• 期刊: 2022 IEEE/ACM Seventh International Conference on Internet-of-Things Design and Implementation (IoTDI)
• 作者: Shijia Zhang;Yilin Liu;Mahanth K. Gowda

Leveraging the Properties of mmWave Signals for 3D Finger Motion Tracking for Interactive IoT Applications

• DOI: 10.1145/3570613
• 发表时间: 2022-12
• 期刊: Proceedings of the ACM on Measurement and Analysis of Computing Systems
• 作者: Yilin Liu;Shijia Zhang;Mahanth K. Gowda;Srihari Nelakuditi