Enhancing Programming and Machine Learning Education for Students with Visual Impairments through the Use of Compilers, AI and Cloud Technologies

通过使用编译器、人工智能和云技术加强对视力障碍学生的编程和机器学习教育

• 批准号: 2202632
• 负责人: Wei Wang
• 金额: $ 77.09万
• 依托单位: University of Texas at San Antonio
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-06-15 至 2025-05-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2202632&HistoricalAwards=false
• 关键词: Enhancing; Programming; Machine; Learning; Education

Attractive high-paying and highly flexible Computer Science careers should be more readily accessible for people with blindness or visual impairments (BVI). Unfortunately, teaching the required computer programming and data science skills to students with BVI is extremely challenging due to two major difficulties. The first difficulty comes from the limited capability of current screen readers to properly read computer codes that are a mix of English letters, digits, and punctuation marks. The specialized set of keystrokes used in programming is also not conveniently read by screen readers (e.g., spaces and tabs). The second difficulty comes from time-consuming and frustrating code navigation, whereby students with BVI must repeatedly use screen readers to read every line to locate the desired line for editing. Partnering with San Antonio Lighthouse for the Blind and Vision Impaired, the project will develop new accessibility tools, including a program syntax- and semantics-aware screen reader and a voice-command-based code navigation framework to address the above two difficulties. These accessibility tools will be offered through cloud-based web interfaces to provide nationwide access to students and educators. The success of this project will improve the effectiveness of teaching computer programming and data science to students with BVI, which in turn will increase accessibility for more individuals with BVI to participate in Computing Science with high-paying career opportunities and could lead to a more-diverse Computer Science workforce. These accessibility tools will use compilers, Artificial Intelligence (AI), and cloud technologies to read computer code statements based on their meanings, rather than only reading one character at a time. The screen reader will articulate the necessary information that beginning coders need and help them more easily understand the lexicon and semantics used in computer programming and data science. The voice-command-based code navigation will employ speech recognition and natural language processing so that students will be able to use their voice to easily locate a specific statement (e.g., a variable declaration) within their code. These accessibility tools will be integrated into Jupyter notebook and offered through the cloud which will give nationwide access to students and educators. This cloud-based solution will also allow sophisticated AI models to be employed without requiring the students to have powerful and expensive computers to run these accessibility tools. The project will conduct a systematic evaluation of these accessibility tools using single-case research design to deepen the understanding of how technologies, including compilers, AI, and cloud computing, can be applied to teaching Computer Science skills to students with BVI. The evaluation will also provide feedback on the effectiveness of different speech styles and provide additional feedback for future improvements of these accessibility tools.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

对于失明或视力障碍（BVI）的人来说，有吸引力的高薪和高度灵活的计算机科学职业应该更容易获得。不幸的是，由于两个主要困难，向英属维尔京群岛的学生教授所需的计算机编程和数据科学技能极具挑战性。第一个困难来自当前屏幕阅读器正确读取由英文字母、数字和标点符号混合而成的计算机代码的能力有限。屏幕阅读器也不方便读取编程中使用的专门击键集（例如空格和制表符）。第二个困难来自耗时且令人沮丧的代码导航，BVI 学生必须反复使用屏幕阅读器阅读每一行，以找到所需的行进行编辑。该项目将与圣安东尼奥灯塔盲人和视力障碍者合作开发新的辅助工具，包括程序语法和语义感知屏幕阅读器以及基于语音命令的代码导航框架，以解决上述两个困难。这些辅助工具将通过基于云的网络界面提供，以便为全国范围内的学生和教育工作者提供访问权限。该项目的成功将提高向英属维尔京群岛学生教授计算机编程和数据科学的有效性，这反过来又将增加更多英属维尔京群岛个人参与计算科学的机会，获得高薪职业机会，并可能带来更多-多元化的计算机科学劳动力。 这些辅助工具将使用编译器、人工智能 (AI) 和云技术根据含义读取计算机代码语句，而不是一次只读取一个字符。屏幕阅读器将阐明初学者所需的必要信息，并帮助他们更轻松地理解计算机编程和数据科学中使用的词汇和语义。基于语音命令的代码导航将采用语音识别和自然语言处理，以便学生能够使用语音轻松找到代码中的特定语句（例如变量声明）。这些辅助工具将集成到 Jupyter 笔记本中，并通过云提供，这将为全国范围内的学生和教育工作者提供访问权限。这种基于云的解决方案还允许采用复杂的人工智能模型，而无需学生拥有功能强大且昂贵的计算机来运行这些辅助工具。该项目将使用单案例研究设计对这些辅助工具进行系统评估，以加深对如何应用技术（包括编译器、人工智能和云计算）向英属维尔京群岛学生教授计算机科学技能的理解。该评估还将提供有关不同演讲风格的有效性的反馈，并为这些辅助工具的未来改进提供额外的反馈。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力优点和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Virtual Summer Camp for High School Students with Disabilities - An Experience Report

残疾高中生虚拟夏令营 - 经验报告

• DOI: 10.1145/3545945.3569818
• 发表时间: 2023-03
• 期刊: Proceedings of the 54th ACM Technical Symposium on Computer Science Education
• 作者: Wang, Wei;Ewoldt, Kathy B.;Xie, Mimi;Mestas;Soderman, Sean;Wang, Jeffrey
• 通讯作者: Wang, Jeffrey