Excellence in Research: Exploring Effectiveness of Automatic Assessment of Cognitive and Metacognitive Processes in Engineering Learning through Natural Language Processing Models

卓越研究：通过自然语言处理模型探索工程学习中认知和元认知过程自动评估的有效性

• 批准号: 2302686
• 负责人: Wei Zheng
• 金额: $ 60万
• 依托单位: Jackson State University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2023
• 资助国家: 美国
• 起止时间: 2023-09-01 至 2026-08-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2302686&HistoricalAwards=false
• 关键词: Excellence; Research; Exploring; Effectiveness; Automatic

Timely assessment of students' learning is crucial for addressing their needs. However, current assessment methods such as multiple-choice or calculation tests may not truly reveal students’ internal thinking processes, because students may guess answers or follow the step-by-step procedures of examples without conceptual understanding. Existing research shows that asking students to write down justifications for their answers, and plan and reflection on their learning can promote them to develop deeper conceptual understanding and apply cognitive and metacognitive strategies in their learning. Nonetheless, this approach is not widely adopted due to the time-consuming nature of assessing free-text responses. This project aims to develop a rubrics-based automatic assessment tool to identify individual students' misconceptions and learning deficiencies from their free-text responses, allowing instructors and students to instantly adjust their teaching and learning, and eventually enabling personalized instructions tailored to individual learners’ needs, which particularly benefits students at HBCUs. The project will also provide the base for delivering research experience for both undergraduates and teachers and outreach various audiences, including public school students, increasing the literacy of the public on artificial intelligence.This research proposes innovative strategies to fine-tune and calibrate the pre-trained language model, through self-supervised learning and few-shot learning, for automatic classification of texts based on user-specified rubrics, with a particular focus on learners' cognitive and metacognitive processes. The proposed model integrates three novel attributes for improving similarity comparison between texts and rubric keywords: (1) across-attention between compared texts for increasing the sensitivity of comparison; (2) joint embedding of words, phrases, and sentences for improving the accuracy of comparison; (3) incorporation of thematic relevance for enhancing the breadth of comparison. Given the assessment rubrics, the model can classify texts to reveal students’ thinking or other traits in multiple finer granular perspectives. It is flexible for adding new assessment perspectives and more transparent than the overall assessment, and allows involving human judgment, leading to more reliable assessment acceptable for practice, and advancing knowledge on adapting pre-trained language models for personalized instructions. The dataset collected from diverse students and specific strategies will be adopted to mitigate the potential biases of the proposed model.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

及时评估学生的学习情况对于满足他们的需求至关重要。然而，现有的多项选择或计算测试等评估方法可能无法真正揭示学生的内部思维过程，因为学生可能会猜测答案或遵循循序渐进的程序。现有的研究表明，要求学生写下他们的答案的理由，并计划和反思他们的学习可以促进他们发展更深入的概念理解并在学习中应用认知和元认知策略。由于耗时，没有被广泛采用该项目旨在开发一种基于量规的自动评估工具，从学生的自由文本回答中识别出他们的误解和学习缺陷，使教师和学生能够立即调整他们的教学和学习。实现针对个人学习者需求的个性化指导，这对 HBCU 的学生尤其有利。该项目还将为向本科生和教师提供研究经验以及向包括公立学校学生在内的各种受众进行推广提供基础，从而提高公众的素养。关于人工智能。这项研究提出创新策略，通过自我监督学习和自动小样本学习来微调和校准预训练的语言模型，根据用户指定的规则对文本进行分类，特别关注学习者的认知和元认知过程该模型集成了三个新的属性来提高文本和标题关键词之间的相似性比较：（1）比较文本之间的交叉注意力，以提高比较的敏感性；（2）单词、短语和句子的联合嵌入，以提高准确性。 （3）纳入主题相关性以增强比较的广度，该模型可以对文本进行分类，以更细粒度的视角揭示学生的思维或其他特征，并且可以灵活地添加新的评估视角。比整体评估更透明，并且允许涉及人类判断，从而导致更可靠的评估可以接受实践，并提高从不同学生收集的数据集和具体策略来适应个性化指导的知识。拟议的潜在偏见该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。