AitF: Mechanism Design and Machine Learning for Peer Grading

AitF：同行评分的机制设计和机器学习

• 批准号: 1733860
• 负责人: Jason Hartline
• 金额: $ 70万
• 依托单位: Northwestern University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2017
• 资助国家: 美国
• 起止时间: 2017-09-01 至 2022-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1733860&HistoricalAwards=false
• 关键词: AitF; Mechanism; Design; Machine; Learning

This project explores the design and analysis of peer grading technology. A peer grading system is an online tool that collects student submissions, assigns review tasks to the students and graders, and aggregates reviews to produce assessments of both the submissions and the peer reviews. The PIs have developed a prototype system and have collected preliminary evidence that suggests that peer review has important potential benefits:1. Learning by reviewing: Students learn from critical assessment of other students' work. In the PIs' prototype at Northwestern, 60% of the students reported that peer review helped them learn course material and 55% of the students reported that peer review helped them to prepare better homework solutions themselves.2. Reduced grading staff: Peer grading reduces the grading load on course staff and allows for effective teaching with larger classes. This is especially important currently, as interest in computer science classes increases at a faster pace than teaching resources. In the PIs' prototype at Northwestern, the course staff graded 1/5 of the student submissions.3. Promptness of feedback: Reduced teacher grading enables prompt feedback to students. In the PIs' prototype at Northwestern, peer reviews were available within three days and final assessment of both the submission and peer reviews were available within five days. Prior to introducing peer review, assessments took one to two weeks.A peer grading system is comprised of three main components:1. The review matching algorithm determines which peers should review which submissions and which submissions should be reviewed by the teacher.2. The submission grading algorithm aggregates the reviews of the peers and the submissions and assigns grades to the submissions.3. The review grading algorithm compares the peer reviews with the teacher reviews and assigns grades to the peer reviews. Without this algorithm, peers may not put effort into providing quality reviews, and the reviews will be neither accurate for grading nor beneficial for the peer.The details of these algorithms are crucial for the proper working of the peer review system. A main research effort of this project is to identify the algorithms to use for each of these components. The review matching algorithm affects the accuracy of the subsequent grading algorithms and the grading load of the teacher. The submission grading algorithm determines which peer reviews are accurate and which are inaccurate and uses this understanding to assign grades to the submissions that are representative of the submission quality. The review grading algorithm incentivizes the peers to put in sufficient effort to determine whether a submission is good or bad and it is calibrated so that good reviews and bad reviews get the appropriate review grades. The PIs have implemented prototypes of these algorithms as part of a peer grading system that has been prototyped in Northwestern computer science classes. However, the space of possible algorithms is large and the PIs' work on the prototype has yet to determine the algorithms that combine to give the best education outcomes. A main focus of this project will be improving the understanding of which algorithms lead to the best education outcomes.Theoretical work in algorithms and machine learning provides a starting point for the project's study of good algorithms for peer grading systems. A key endeavor of the project is translating and applying these theoretical algorithms to the peer grading domain. As one example, proper scoring rules are a natural approach for grading the peer reviews. However, test runs of the PIs' prototype implementation suggest that these rules might not be so good in practice. Both new models and algorithms are needed in theory, and these new algorithms need to work in practice.

该项目探索同行评分技术的设计和分析。 同行评分系统是一种在线工具，它收集学生提交的内容，向学生和评分者分配审阅任务，并汇总评论以对提交的内容和同行评审进行评估。 PI 开发了一个原型系统，并收集了初步证据，表明同行评审具有重要的潜在好处：1． 通过复习来学习：学生通过对其他学生作业的批判性评估来学习。 在西北大学的 PI 原型中，60% 的学生表示同行评审帮助他们学习课程材料，55% 的学生表示同行评审帮助他们自己准备更好的作业解决方案。2． 减少评分人员：同伴评分减少了课程人员的评分负担，并允许在更大的班级中进行有效的教学。 目前这一点尤其重要，因为人们对计算机科学课程的兴趣增长速度快于教学资源。 在西北大学的 PI 原型中，课程工作人员对学生提交的 1/5 进行评分。3． 反馈的及时性：教师评分的降低可以使学生得到及时的反馈。 在西北大学的 PI 原型中，同行评审可在三天内完成，提交内容和同行评审的最终评估可在五天内完成。 在引入同行评审之前，评估需要一到两周的时间。同行评分系统由三个主要组成部分组成：1。 评审匹配算法决定哪些同学应该评审哪些提交内容以及哪些提交内容应该由老师评审。2． 提交评分算法汇总了同行的评论和提交的内容，并对提交的内容进行评分。 3． 评审评分算法将同行评审与教师评审进行比较，并对同行评审进行评分。 如果没有这种算法，同行可能不会努力提供高质量的评论，并且评论对于评分既不准确，也不对同行有利。这些算法的细节对于同行评审系统的正常工作至关重要。 该项目的主要研究工作是确定用于每个组件的算法。 复习匹配算法影响后续评分算法的准确性和老师的评分负担。 提交评分算法确定哪些同行评审是准确的，哪些是不准确的，并使用这种理解来为代表提交质量的提交评分。 评论评分算法激励同行付出足够的努力来确定提交的内容是好还是坏，并且对其进行校准，以便好的评论和差的评论获得适当的评论等级。 PI 已经实现了这些算法的原型，作为同行评分系统的一部分，该系统已在西北大学计算机科学课程中原型化。 然而，可能的算法空间很大，PI 的原型工作尚未确定能够结合起来提供最佳教育成果的算法。 该项目的主要重点将是提高对哪些算法可以带来最佳教育成果的理解。算法和机器学习方面的理论工作为该项目研究同行评分系统的良好算法提供了起点。 该项目的一个关键工作是将这些理论算法转化并应用到同行评分领域。 举个例子，适当的评分规则是对同行评审进行评分的一种自然方法。 然而，PI 原型实施的测试运行表明这些规则在实践中可能不太好。 理论上需要新的模型和算法，而这些新的算法需要在实践中发挥作用。

项目成果

Optimization of Scoring Rules

评分规则优化

• DOI: 10.1145/3490486.3538338
• 发表时间: 2022-07
• 期刊: ACM Conference on Economics and Computation
• 作者: Li, Yingkai;Hartline, Jason D.;Shan, Liren;Wu, Yifan
• 通讯作者: Wu, Yifan

Practical Methods for Semi-automated Peer Grading in a Classroom Setting

课堂环境中半自动同伴评分的实用方法

• DOI: 10.1145/3340631.3394878
• 发表时间: 2020-07
• 期刊: Adaptation and Personalization
• 作者: Yuan, Zheng;Downey, Doug
• 通讯作者: Downey, Doug