IGF::OT::IGF : An Innovative Approach for Peer-Review. Period of Performance: September 15, 2017 - March 14, 2018. N43DA-17-1214. BASE AWARD.

IGF::OT::IGF：同行评审的创新方法。

• 批准号: 9582275
• 负责人: YINFENG ZHANG
• 金额: $ 22.5万
• 依托单位: PEVALS, LLC
• 依托单位国家: 美国
• 财政年份: 2017
• 资助国家: 美国
• 起止时间: 2017-09-15 至 2018-03-14
• 项目状态: 已结题
• 来源: https://reporter.nih.gov/project-details/9582275
• 关键词: Advocacy; Agreement; Algorithms; Award; Bioinformatics; Biological Sciences; Contractor; Decision Making; Disease; Educational workshop; Effectiveness; Electronics; Engineering; Evaluation; Florida; Foundations; Future; Goals; Individual; Institutes; Legal patent; Methodology; Methods; Modification; Online Systems; Paper; Participant; Peer Review; Performance; Phase; Pilot Projects; Policy Maker; Process; Reporting; Research; Science; Small Business Innovation Research Grant; Solid; Students; Testing; Universities; Work; Workload; base; design; essays; graduate student; improved; indexing; innovation; instructor; novel; novel strategies; programs; simulation; symposium

Under this Small Business Innovation Research (SBIR) Phase I project, Research Topic 162, the contractor will investigate the effectiveness of new peer-review methods, Concordance Index based Global Ranking (CIGR) and Modified Borda Count (MCB), to enhance professional evaluation and decision making in life sciences management and administration. The intended result is a novel set of metrics that can be used by NGOs/disease foundations, advocacy groups, research funders, policy makers, and by academic institutional bodies. This SBIR phase I project has the following objectives: 1. Simulation studies to investigate the effectiveness of the method used in the NSF pilot study and the contractor's new method, CIGR. The contractor will perform simulation studies to investigate the effectiveness of the new review method in terms of how it can recover the true ranking of proposals or papers. The contractor will also study which parameter settings are optimal for the new review method. 2. Testing the new approach in real evaluation settings The contractor has implemented the method used in the NSF’s pilot study as an online platform at http://pevals.com. The contractor will implement our new ranking method, CIGR, in the phase I project and test the new peer-review approach in graduate student course project/essay evaluation at Florida State University. In these tests, the true rankings and scores of the project reports or essays will be obtained from the course instructor and used as the true rankings. After collecting the ranking from individuals, we will use both CIGR and the modified Borda Count (MBC) score, proposed originally for the new peer-review method and also used in the NSF’s pilot study, to obtain the final rankings and compare the accuracy of the two different ranking methods based on their agreement with the true rankings. This test is unique in the sense that it has the true rankings obtained from the course instructors, which will help us to evaluate the effectiveness of different ranking algorithms. This test of using graduate student course projects will establish solid basis for our future tests on scientific papers and proposals. The contractor will also test our method for a conference paper evaluation for a workshop in IEEE BIBM 2017 conference.

在小型企业创新研究 (SBIR) 第一阶段项目的研究主题 162 下，承包商将调查新的同行评审方法、基于一致性指数的全球排名 (CIGR) 和改进的博尔达计数 (MCB) 的有效性，以加强专业评估预期结果是一套新颖的指标，可供非政府组织/疾病基金会、倡导团体、研究资助者、政策制定者和学术机构机构使用。项目有以下目标： 1. 模拟研究，调查 NSF 试点研究中使用的方法和承包商的新方法 CIGR 的有效性。 承包商将进行模拟研究，以调查新评审方法在如何恢复提案或论文的真实排名方面的有效性。承包商还将研究哪些参数设置对于新评审方法来说是最佳的。 2. 在实际评估环境中测试新方法 承包商已将 NSF 试点研究中使用的方法作为在线平台实施，网址为 http://pevals.com。承包商将在第一阶段项目中实施我们的新排名方法 CIGR，并在 2017 年测试新的同行评审方法。佛罗里达州立大学研究生课程项目/论文评估在这些测试中，项目报告或论文的真实排名和分数将从课程导师处获得，并在收集个人排名后用作真实排名。同时使用 CIGR 和修改后的 Borda计数（MBC）分数，最初是为新的同行评审方法提出的，也用于 NSF 的试点研究，以获得最终排名，并根据它们与真实排名的一致性来比较两种不同排名方法的准确性。其独特之处在于它具有从课程讲师处获得的真实排名，这将有助于我们评估不同排名算法的有效性，这种使用研究生课程项目的测试将为我们将来的科学论文和测试奠定坚实的基础。提案。 承包商还将在 IEEE BIBM 2017 会议的研讨会上测试我们的会议论文评估方法。