NSF-BSF: RI: Small: Mechanisms and Algorithms for Improving Peer Selection

NSF-BSF：RI：小型：改进同行选择的机制和算法

• 批准号: 2134857
• 负责人: Nicholas Mattei
• 金额: $ 30.89万
• 依托单位: Tulane University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-01-01 至 2024-12-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2134857&HistoricalAwards=false
• 关键词: NSF; BSF; RI; Small; Mechanisms

The process of peer review, evaluation, and selection is a fundamental aspect of modern science. Funding bodies and academic publications around the world employ experts to review and select the best science for funding and publication. The process of evaluating and selecting the best from among a group of peers is much more general problem. For example, a professional society may want to give a subset of its members awards based on the opinions of all members or an instructor for a Massive Open Online Course (MOOC) may want to crowdsource grading or a marketing company may select ideas from group brainstorming sessions based on peer evaluation. In all of these settings, we wish to select a small set of winners that are judged to be the best by the community itself -- which includes those who are competing and who may have conflict of interests. This problem, known as the peer selection problem, is the focus of this research. Within a peer selection setting there may be competing priorities and inherent biases amongst the set of reviewers, and it is necessary to develop methods and algorithms that align the individual incentives of reviewers with the overall goal of selecting the best set. The intellectual merit of this project lies in expanding our understanding and developing novel algorithms for the process of peer evaluation and peer selection. Within the fields that use peer review, conflict of interest and peer selection bias have been cited as an impediment for broader participation in the science. This project will have broad impact through making the peer review process more robust to equitable selection by filtering some reviewers’ unconscious biases and conflict of interest thus resulting in a better infrastructure for research and education.The project will achieve its goal of expanding our knowledge and building mechanisms for peer evaluation and selection through four specific aims. The first aim is to develop novel metrics for the evaluation of peer selection mechanisms by defining both normative and quantitative properties that allow to precisely describe features of the peer evaluation and selection process. The second aim is to develop distributed peer selection mechanisms that are able to be used without requiring a centralized controller. This project will develop tools to understand how these mechanisms behave in this distributed setting as well as opportunities to create novel mechanisms for the unique challenges this setting poses. The third aim is to develop our understanding of multi-stage peer evaluation for peer selection. Motivated by the rolling review cycle of many academic conferences, journals, and even some NSF programs, there is a need to investigate the properties of peer evaluation and selection mechanisms when reviews (evaluations) may propagate between specific selection settings. The final aim is to incentivize effort in peer selection: There is a fundamental tension between the classic social choice properties of impartiality, i.e., an agent may not affect their own probability of getting accepted, and provide incentives for reviewers to invest effort in the peer evaluation process. This project will develop a tool kit of mechanisms that allow system designers to rationally choose tradeoffs between the amount of information an agent knows, incentives for effort, and potential for malicious behavior.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

同行评审，评估和选择的过程是现代科学的一个基本方面。全球员工专家的资助机构和学术出版物审查和选择资金和出版物的最佳科学。在一组同行中评估和选择最好的过程是更普遍的问题。例如，专业社会可能希望根据所有成员的意见或大规模开放在线课程的教练（MOOC）授予其成员的一部分奖项，或者可能希望对众群落进行评分，或者营销公司可能会从基于同行评估的小组集思广益会议中选择思想。在所有这些环境中，我们希望选择一小部分获奖者，这些获奖者被社区本身认为是最好的，其中包括那些正在竞争和可能有利益冲突的人。这个被称为同伴选择问题的问题是这项研究的重点。在同行选择设置中，可能存在竞争优先级，并在审阅者集中继承偏见，并且有必要开发将审稿人的个体激励措施与选择最佳集合的总体目标相一致的方法和算法。该项目的智力优点在于扩大我们的理解和开发新颖的算法，以进行同伴评估和同伴选择。在使用同行评审的领域中，人们认为利益冲突和同伴选择偏见是为了更广泛参与科学的障碍。通过过滤一些审稿人的无意识偏见和利益冲突，该项目将通过使同伴审查过程更加强大，从而产生广泛的影响，从而为研究和教育提供更好的基础架构。该项目将实现其扩展我们的知识和建立机制，以通过四个特定目标来扩展我们的知识和建立机制。第一个目的是通过定义允许精确描述同伴评估和选择过程的特征来开发新的指标来评估同伴选择机制。第二个目的是开发能够使用的分布式同行选择机制，而无需集中式控制器。该项目将开发工具，以了解这些机制在这种分布式环境中的行为以及为这种设置所带来的独特挑战创造新颖机制的机会。第三个目的是发展我们对多阶段同行评估以进行同伴选择的理解。在许多学术会议，期刊甚至某些NSF计划的滚动审查周期中，需要调查同行评估和选择机制的属性，当评论（评估）可能在特定选择设置之间传播。最终目的是激励同伴选择的努力：公正的经典社会选择属性之间存在基本张力基金会的智力优点和更广泛的影响评论标准。

项目成果

PeerNomination: A novel peer selection algorithm to handle strategic and noisy assessments

• DOI: 10.1016/j.artint.2022.103843
• 发表时间: 2022-12
• 期刊: Artif. Intell.
• 作者: Omer Lev;Nicholas Mattei;P. Turrini;Stanislav Zhydkov

Who Reviews The Reviewers? A Multi-Level Jury Problem

• 发表时间: 2022-11
• 作者: Ben Abramowitz;Nicholas Mattei

Mitigating Skewed Bidding for Conference Paper Assignment

• DOI: 10.48550/arxiv.2303.00435
• 发表时间: 2023-03
• 作者: Inbal Rozencweig;R. Meir;Nick Mattei;Ofra Amir