Excellence in Research: Exploring Effectiveness of Automatic Assessment of Cognitive and Metacognitive Processes in Engineering Learning through Natural Language Processing Models

卓越研究：通过自然语言处理模型探索工程学习中认知和元认知过程自动评估的有效性

• 批准号: 2302686
• 负责人: Wei Zheng
• 金额: $ 60万
• 依托单位: Jackson State University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2023
• 资助国家: 美国
• 起止时间: 2023-09-01 至 2026-08-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2302686&HistoricalAwards=false
• 关键词: Excellence; Research; Exploring; Effectiveness; Automatic

Timely assessment of students' learning is crucial for addressing their needs. However, current assessment methods such as multiple-choice or calculation tests may not truly reveal students’ internal thinking processes, because students may guess answers or follow the step-by-step procedures of examples without conceptual understanding. Existing research shows that asking students to write down justifications for their answers, and plan and reflection on their learning can promote them to develop deeper conceptual understanding and apply cognitive and metacognitive strategies in their learning. Nonetheless, this approach is not widely adopted due to the time-consuming nature of assessing free-text responses. This project aims to develop a rubrics-based automatic assessment tool to identify individual students' misconceptions and learning deficiencies from their free-text responses, allowing instructors and students to instantly adjust their teaching and learning, and eventually enabling personalized instructions tailored to individual learners’ needs, which particularly benefits students at HBCUs. The project will also provide the base for delivering research experience for both undergraduates and teachers and outreach various audiences, including public school students, increasing the literacy of the public on artificial intelligence.This research proposes innovative strategies to fine-tune and calibrate the pre-trained language model, through self-supervised learning and few-shot learning, for automatic classification of texts based on user-specified rubrics, with a particular focus on learners' cognitive and metacognitive processes. The proposed model integrates three novel attributes for improving similarity comparison between texts and rubric keywords: (1) across-attention between compared texts for increasing the sensitivity of comparison; (2) joint embedding of words, phrases, and sentences for improving the accuracy of comparison; (3) incorporation of thematic relevance for enhancing the breadth of comparison. Given the assessment rubrics, the model can classify texts to reveal students’ thinking or other traits in multiple finer granular perspectives. It is flexible for adding new assessment perspectives and more transparent than the overall assessment, and allows involving human judgment, leading to more reliable assessment acceptable for practice, and advancing knowledge on adapting pre-trained language models for personalized instructions. The dataset collected from diverse students and specific strategies will be adopted to mitigate the potential biases of the proposed model.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

及时评估学生的学习对于满足他们的需求至关重要。但是，当前的评估方法（例如多项选择或计算测试）可能并不能真正揭示学生的内部思维过程，因为学生可以猜测答案或遵循示例的分步程序而无需概念性理解。现有的研究表明，要求学生写下自己的答案理由，并计划和反思他们的学习可以促进他们在学习中发展更深入的概念理解并应用认知和元认知策略。但是，由于评估自由文本响应的耗时性，这种方法并未被广泛采用。该项目旨在开发一种基于专栏的自动评估工具，以确定学生的失误和从自由文本的回答中学习缺陷，使讲师和学生能够立即调整他们的教学和学习，有时还可以为个人学习者的需求量身定制个性化指导，从而对HBCUS的学生特别有利。该项目还将为本科生和教师以及包括公立学校学生在内的各种受众提供研究经验的基础，提高公众对人工智能的素养。这项研究提出了创新的策略，以通过自动化的学习和刻板的方式来微调和校准预先培训的语言模型，以介绍特定的型号，以求解'ibles of in Texters ob of Sextific'认知和元认知过程。提出的模型集成了三个新的属性，以改善文本和标题关键词之间的相似性比较：（1）比较文本之间的跨注意，以提高比较的敏感性； （2）单词，短语和句子的联合嵌入，以提高比较的准确性； （3）主题相关性对增强比较广度的影响。鉴于评估专栏，该模型可以对文本进行分类，以揭示学生的思维或其他精细粒状观点。与整体评估相比，添加新的评估观点和更透明是灵活的，并允许涉及人类判断，从而导致更可靠的评估可以接受实践，并促进有关调整预训练的语言模型以进行个性化指导的知识。从潜水员学生那里收集的数据集和特定策略将被采用，以减轻拟议模型的潜在偏见。该奖项反映了NSF的法定任务，并使用基金会的知识分子优点和更广泛的影响审查标准来通过评估来评估。