CAREER: Generative Item, Response, and Feedback Models in Assessment and Learning

职业：评估和学习中的生成项目、响应和反馈模型

基本信息

批准号：
2237676
负责人：
Shiting Lan
金额：
$ 64.46万
依托单位：
University of Massachusetts Amherst
依托单位国家：
美国
项目类别：
Standard Grant
财政年份：
2023
资助国家：
美国
起止时间：
2023-05-15 至 2028-04-30
项目状态：
未结题

来源：
https://www.nsf.gov/awardsearch/showAward?AWD_ID=2237676&HistoricalAwards=false
关键词：
CAREER Generative Item Response Feedback

项目摘要

Personalized tutoring and feedback on performance or knowledge mastery, are two instructional strategies that have been shown to be effective at improving student learning outcomes. However, implementing these strategies, especially at scale, is costly in terms of the human resources required to provide them effectively. The use of Artificial Intelligence (AI) to provide students with feedback and personalized tutoring in digital learning platforms has the potential to reduce the human capital required to provide these services and to service growing numbers of learners effectively. This CAREER project will leverage generative language models (GLMs), a recent innovation in AI machine learning, to estimate learner knowledge levels and identify specific errors from open-ended learner responses. The resulting system will then be able to automatically generate personalized items and feedback, to support teachers and learners. Primarily grounded in middle-school math education with data collection and evaluation supported by ASSISTments and OpenStax, this CAREER project has the potential to benefit many teachers and learners. Other potential outcomes of his CAREER project include activities that expand the access of minority learners to real-world applications of AI and a new course on AI for education. This CAREER project includes three major research threads. First, the project team will develop a family of open-ended item response theory and knowledge tracing frameworks for open-ended math items. The key technical challenge will be to inject learner knowledge states to steer GLMs towards generating personalized response predictions according to each learner’s knowledge on different skills. These models will power teacher dashboard tools and learner error detection tools during tutoring activities. Second, the project team will develop GLM-based automated math item generation methods to meet the needs and interests of each learner and evaluate them in a randomized controlled trial. The key technical challenge will be to control the generated items according to human specifications on item context and both mathematical and language complexity. Third, the project team will develop a GLM-based automated feedback generation framework and explore its usage in both common wrong answer feedback and tutoring dialogue turn generation. The key technical challenge will be to learn how to leverage effective teacher-written feedback messages and use them as input examples for GLMs. The team will also explore learning-from-teacher-edit methods to constantly improve the quality of generated feedback over time.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

对性能或知识掌握的个性化辅导和反馈是两种教学策略，这些策略已被证明有效地改善了学生的学习成果。但是，就有效提供它们提供的人力资源而言，实施这些策略，尤其是规模上是昂贵的。使用人工智能（AI）在数字学习平台中为学生提供反馈和个性化辅导，有可能减少提供这些服务所需的人力资本，并有效地为越来越多的学习者提供服务。该职业项目将利用通用语言模型（GLMS），这是AI机器学习中最近的创新，以估算学习者知识水平并从开放式学习者的响应中确定特定的错误。然后，最终的系统将能够自动生成个性化的项目和反馈，以支持教师和学习者。该职业项目主要基于中学数学教育，并在辅助和OpenStax支持的数据中收集数据和评估，有可能使许多教师和学习者受益。他的职业生涯项目的其他潜在结果包括将少数学习者访问AI的现实应用程序和AI新课程的活动的活动。这个职业项目包括三个主要的研究线程。首先，项目团队将开发一个开放式项目响应理论和开放式数学项目的知识追踪框架的家庭。关键的技术挑战将是注入学习者知识状态，以根据每个学习者对不同技能的知识来产生个性化的响应预测。这些模型将在辅导活动期间为教师仪表板工具和学习者错误检测工具提供动力。其次，项目团队将开发基于GLM的自动数学项目生成方法，以满足每个学习者的需求和兴趣，并在随机对照试验中对其进行评估。关键的技术挑战将是根据人体规范在项目上下文以及数学和语言复杂性上控制生成的项目。第三，项目团队将开发一个基于GLM的自动反馈生成框架，并在常见的错误答案反馈和辅导对话中探索其用法。关键的技术挑战将是学习如何利用有效的教师写的反馈消息，并将其用作GLM的输入示例。该团队还将探索从教师编辑方法中的学习方法，以不断提高随着时间的推移的反馈质量。该奖项反映了NSF的法定使命，并使用基金会的知识分子优点和更广泛的影响评估标准，被认为是通过评估而被视为珍贵的支持。

项目成果

期刊论文数量（8）

专著数量（0）

科研奖励数量（0）

会议论文数量（0）

专利数量（0）

Automated Distractor and Feedback Generation for Math Multiple-choice Questions via In-context Learning

DOI：
发表时间：
2023-08
期刊：
影响因子：
0
作者：
Hunter McNichols;Wanyong Feng;Jaewook Lee;Alexander Scarlatos;Digory Smith;Simon Woodhead;Andrew S. Lan
通讯作者：
Hunter McNichols;Wanyong Feng;Jaewook Lee;Alexander Scarlatos;Digory Smith;Simon Woodhead;Andrew S. Lan

Balancing Test Accuracy and Security in Computerized Adaptive Testing

平衡计算机化自适应测试中的测试准确性和安全性