Using Scientific Materials to Compare Disaggregated Versus Holistic Ratings

使用科学材料比较分类评级与整体评级

基本信息

批准号：
0109250
负责人：
Hal Arkes
金额：
$ 8.44万
依托单位：
Ohio State University Research Foundation -DO NOT USE
依托单位国家：
美国
项目类别：
Standard Grant
财政年份：
2001
资助国家：
美国
起止时间：
2001-08-01 至 2003-07-31
项目状态：
已结题

来源：
https://www.nsf.gov/awardsearch/showAward?AWD_ID=0109250&HistoricalAwards=false
关键词：
Using Scientific Materials Compare Disaggregated

项目摘要

The National Institutes of Health (NIH), the National Science Foundation (NSF), and other agencies spend billions of dollars each year evaluating scientific proposals. Only the most meritorious should receive taxpayer dollars. But how should these agencies decide which proposals are the most meritorious?There are two general ways to evaluate proposals, candidates, strategies, and nearly any other entity. The first way is called "holistic." Holistic ratings are characterized by a single overall number which reflects one's opinion. The second way is called "disaggregated." Disaggregated ratings are characterized by one rating being given to each of several criteria. The criteria's ratings are then summed or averaged to obtain a single number, which reflects one's opinion. Some research suggests that disaggregated ratings are superior to holistic ones. However in the late 1990s, both NIH and NSF refused to adopt disaggregated ratings in their own proposal evaluation procedures, because no prior research comparing holistic and disaggregated ratings used materials such as scientific proposals. The proposed research will fill that gap. Attendees at a scientific convention will be asked to rate oral presentations in either a holistic or disaggregated fashion. Two holistic panels of five persons each and two disaggregated panels of five persons each will rate every presentation. The correlation between the two holistic panels' ratings of the presentations will be compared to the correlation between the two disaggregated panels' ratings of the presentations. If prior research applies to scientific presentations, then the correlation between the disaggregated panels' ratings should be higher than the correlation between the holistic panels' ratings. This will provide evidence that disaggregated ratings of scientific materials yields evaluations of higher reliability than do holistic ratings. This result should in turn provide guidance to the scientific agencies as to one way in which their evaluation processes might be improved.

美国国立卫生研究院（NIH），国家科学基金会（NSF）和其他机构每年花费数十亿美元评估科学建议。只有最有功的人才能收到纳税人的美元。但是，这些机构应该如何决定哪些建议是最有功的？有两种一般方法可以评估建议，候选人，策略以及几乎任何其他实体。第一种方式称为“整体”。整体评级的特征是单个总数，反映了一个人的意见。第二种方式称为“分类”。分解评级的特征是给出了几个标准的每个评分。然后将标准的评级求和或平均以获取单个数字，这反映了一个人的意见。一些研究表明，分解评级优于整体评级。但是，在1990年代后期，NIH和NSF均拒绝在自己的提案评估程序中采用分类的评级，因为没有先前的研究比较了整体和分类评级的使用材料，例如科学建议。拟议的研究将填补这一空白。科学公约的与会者将被要求以整体或分类的方式对口头介绍进行评分。每个五个人和两个分组的面板的两个整体面板各五人将评分每个演示文稿。两个整体面板的演示评分之间的相关性将与两个分类面板的演示评分之间的相关性进行比较。如果先前的研究适用于科学演示，那么分解面板评级之间的相关性应高于整体面板评级之间的相关性。这将提供证据表明，与整体评级相比，分解科学材料的评级可产生更高的可靠性评估。此结果反过来应该为科学机构提供指导，以提高其评估过程的一种方式。