EAGER: Natural Language Processing for Teaching and Research in Engineering Education

EAGER：用于工程教育教学和研究的自然语言处理

• 批准号: 2107008
• 负责人: Andrew Katz
• 金额: $ 29.96万
• 依托单位: Virginia Polytechnic Institute and State University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-03-01 至 2025-02-28
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2107008&HistoricalAwards=false
• 关键词: EAGER; Natural; Language; Processing; Teaching

In ecosystems that form professional engineers, community members produce text through many activities such as end-of-semester feedback to instructors, transcripts of instruction, open-ended survey items, and interviews. In each case, there is abundant text available to educators and researchers that could provide insight into how we form engineers. Unfortunately, while these texts have the potential to provide novel insights, traditional analytic techniques do not scale well. Time investments, bias, interrater reliability, and intrarater reliability each present significant challenges. To address this problem, we aim to develop and characterize approaches for human-in-the-loop (HITL) natural language processing (NLP) systems to augment human analysis, facilitating and enhancing the work of one person (or team). Such systems can help reduce the amount of time needed to analyze texts by grouping similar texts together. The human user can utilize these groupings for further analysis and identify meanings in ways only a human could. The system will also improve consistency by analyzing across the entire collection of texts simultaneously and grouping similar items together. This is in contrast with a single person or a team that would analyze responses sequentially, creating the potential for inconsistencies across time. We will accomplish this work in three phases. In Phase 1, we will conduct a series of experiments to test potential system configurations. The goal will be to identify optimal components and parameter settings for four of the steps in the proposed pipeline. We will use datasets from (i) students’ written responses to an instrument for assessing their systems thinking and (ii) students’ responses to open-ended course feedback surveys. We will measure performance based on consistency of thematic clusters, using standard metrics for homogeneity in text clustering and classification tasks. In Phase 2, we will study system performance on a series of five datasets. These datasets will come from multiple sources: extant NSF-funded projects, longitudinal data from the Virginia Tech College of Engineering, current data in engineering courses, and freshly collected data from online outlets. These represent important areas of the broader ecosystem that supports how we form future engineers. We will test the system for thematic clusters, employing similar metrics as in Phase 1 to identify potential inconsistencies in how different datasets are handled. We will specifically look for homogeneity of texts within a cluster and shared semantic meaning. We will also update the original system designs in the event of systematic differences (e.g., longer texts require a different system configuration). For Phase 3, we will study how it can affect human performance. Since we anticipate significant improvements in human efficiency and consistency, it is important to conduct analyses that can accurately assess the veracity of that proposition. These studies will assess the HITL aspect of this process since many relevant applications of the system will require additional interpretation of the raw output. To accomplish this, we will collect data on differences in human performance when analyzing 1,500 student responses with and without the system’s assistance. We will look at differences when (a) one person alone codes the data and when (b) a team of three researchers codes the data (i.e., we will have two studies: one person with vs one person without and team with vs team without). We will measure differences in coding (whether different themes emerge), reliability (how consistently similar texts are grouped together), time needed to code the data, and differential treatment of student responses associated with student group characteristics. We will host all code on public repositories and notebooks for easy access, copying, and application by other engineering education researchers and teachers along with any new datasets, where appropriate.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

在形成专业工程师的生态系统中，社区成员通过许多活动生成文本，例如向教师提供的学期末反馈、教学记录、开放式调查项目和访谈。在每种情况下，都有丰富的文本可供教育工作者和学生使用。不幸的是，虽然这些文本有可能提供新颖的见解，但传统的分析技术无法很好地扩展时间投入、偏见、评估者间的可靠性和评估者内的可靠性。为了解决这个问题，我们的目标是开发描述人机循环（HITL）自然语言处理（NLP）系统的方法，以增强人类分析，促进和增强一个人（或团队）的工作，这样的系统可以帮助减少分析所需的时间。通过将相似的文本分组在一起来分析文本。人类用户可以利用这些分组以只有人类才能做到的方式来进一步分析和识别含义，该系统还将通过同时分析整个文本集合并将相似的项目分组在一起来提高一致性。与单个人或团队形成对比我们将按顺序分析响应，从而可能会出现不一致的情况。在第一阶段，我们将进行一系列实验来测试潜在的系统配置，目的是确定最佳组件和参数。我们将使用来自以下数据集：（i）学生对评估其系统思维的工具的书面答复；以及（ii）学生对开放式课程反馈调查的答复。基于主题集群的一致性，使用标准在第二阶段，我们将研究一系列五个数据集的系统性能，这些数据集来自多个来源：现有的 NSF 资助的项目、来自弗吉尼亚理工大学工程学院的纵向数据、工程课程中的当前数据以及从在线渠道新收集的数据代表了更广泛的生态系统的重要领域，支持我们如何培养未来的工程师。我们将使用与第一阶段类似的指标来测试系统的主题。在如何处理不同的数据集。我们将特别寻找集群内文本的同质性和共享语义，如果存在系统差异（例如，较长的文本需要不同的系统配置）。 3，我们将研究它如何影响人类绩效，因为我们预计人类效率和一致性将得到显着提高，因此进行能够准确评估该命题准确性的分析非常重要，因为这些研究将评估该过程的 HITL 方面。该系统的许多相关应用程序将为了实现这一目标，我们将在分析 1,500 名学生的回答（无论是否有系统的帮助）时收集人类表现差异的数据。 (b) 由三名研究人员组成的团队对数据进行编码（即，我们将进行两项研究：一个人与一个人没有，以及团队与团队没有），我们将衡量编码方面的差异（是否出现不同的主题）、可靠性（相似文本分组的一致性如何在一起）、编码数据所需的时间以及与学生群体特征相关的学生反应的区别对待。我们将在公共存储库和笔记本上托管所有代码，以便其他工程教育研究人员和教师以及任何人轻松访问、复制和应用。新的数据集（在适当的情况下）。该奖项反映了 NSF 的法定使命，并且通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Students’ Feedback About Their Experiences in EPICS Using Natural Language Processing

学生对他们使用自然语言处理的 EPICS 体验的反馈

• DOI: 10.1109/fie56618.2022.9962557
• 发表时间: 2022-10-08
• 期刊: 2022 IEEE Frontiers in Education Conference (FIE)
• 作者: Isil Anakok;J. Woods;Mark V. Huerta;Jared Schoepf;H. Murzi;Andrew Katz
• 通讯作者: Andrew Katz

Exploring the Impact of Engineering Projects in Community Service on Students' Perspectives About Engineering as a Major

探索社区服务中的工程项目对学生对工程专业的看法的影响

• 发表时间: 2023-10
• 期刊: Frontiers in education
• 作者: Anakok, I;Huerta, M;Katz, A.
• 通讯作者: Katz, A.

Board 65: Work in Progress: Using Natural Language Processing to Facilitate Scoring of Scenario-Based Assessments

Board 65：正在进行的工作：使用自然语言处理促进基于场景的评估评分

• 发表时间: 2023-06
• 期刊: Annual Conference and Exposition of the American Society for Engineering Education
• 作者: Norris, M;Taimoory, H;Katz, A;Grohs, J.
• 通讯作者: Grohs, J.

What engineering employers want: An analysis of technical and professional skills in engineering job advertisements

工程雇主想要什么：工程招聘广告中的技术和专业技能分析

• DOI: 10.1002/jee.20581
• 发表时间: 2024-02
• 期刊: Journal of Engineering Education
• 影响因子: 3.4
• 作者: Fleming, Gabriella Coloyan;Klopfer, Michelle;Katz, Andrew;Knight, David
• 通讯作者: Knight, David

Understanding First-year Engineering Students’ Perceptions of Working with Real Stakeholders on a Design Project: A PBL Approach

了解一年级工科学生对与真正的利益相关者合作设计项目的看法：PBL 方法

• DOI: 10.1109/fie56618.2022.9962395
• 发表时间: 2022-10
• 期刊: IEEE Frontiers in Education Conference
• 作者: Murzi, Homero;Fielding, Lydia;Huerta, Mark;Ortega Alvarez, Juan D.;James, Matthew;Katz, Andrew;Grohs, Jacob
• 通讯作者: Grohs, Jacob