Understanding the relative valuations of research impact: Applying best-worst scaling experiments to survey the public & biomedical/health researchers

了解研究影响的相对评估：应用最佳-最差尺度实验来调查公众

• 批准号: MR/L010569/2
• 负责人: Jonathan Grant
• 金额: $ 17.55万
• 依托单位: King's College London
• 依托单位国家: 英国
• 项目类别: Research Grant
• 财政年份: 2014
• 资助国家: 英国
• 起止时间: 2014 至 无数据
• 项目状态: 已结题
• 来源: https://gtr.ukri.org/projects?ref=MR%2FL010569%2F2
• 关键词: Understanding; relative; valuations; research; impact

Research Councils "encourage researchers to consider the potential contribution that their research can make to the economy and society from the outset, and the resources required to carry out appropriate and project specific knowledge exchange impact activities". As part of their funding applications, researchers must submit a 'Pathway to Impact' section which is peer reviewed by referees and panel members. Similarly the Funding Council will be assessing impact using a case study approach as part of the Research Excellence Framework. Case studies will be reviewed by academic peers and non-academic experts providing a private, public and third sector perspective. However, in assessing the adequacy/quality of these impact claims reviewers cannot currently draw on comprehensive evidence of the views of beneficiaries (i.e. the general population) or the producers of research (i.e. biomedical and health researchers) to qualify or justify their recommendations. It remains unclear how beneficiaries value research impact, how such values vary across different groups and more fundamentally if the UK population understands research impact. In the absence of knowledge about - and methods for assessing - values of beneficiaries and funders of research, policy that directs researchers to estimate impact and research funders to value such impacts rests on weak empiric foundations. This study aims to address this gap by refining and adapting a survey-based approach known as Best-Worst Scaling (BWS) to analyse the relative valuations of research impact as perceived by the general population and researchers. This study will focus on biomedical and health research and will contribute to the MRC's objective to fund studies that "Deliver user/beneficiary views on the contribution of academic research and MRC input to this".The proposed study comprises 5 phases:- Desk and qualitative research in the identification and classification of impacts. A thorough desk based review will catalogue various types of impacts and models for classifying them. We will then refine and develop this classification in four focus groups with the general public and 24 interviews with biomedical and health researchers. The output of this phase will be a conceptual framework with different types of biomedical and health related impact and different intensities of each impact category for use in the BWS survey.- Development of quantitative survey. Based on the first phase we will design and test a survey instrument to elicit relative ranks of different types of impact. The questionnaire will be tested through cognitive interviews and the full fieldwork procedures through a pilot with the public and researchers. The output of this phase will be the final survey instrument.- Fieldwork with researchers and the general public. We will use an online panel to survey 1000 members of the public and the list of existing MRC grantholders to survey 500 health and biomedical scientists. The output of this phase will be a dataset for analysis. - Analysis of researcher and public preferences. The analysis will involve two steps. First we will conduct a review and exploratory statistical analysis to identify associations and trends that require consideration in the modelling work. This would include sample composition across key socioeconomic and demographic variables and checking respondents' understanding of the survey (whether they were able to respond to the questions). Second we will develop a BWS model to (i) derive a full ranking of the impact statements by the general public and health and biomedical scientists, and (ii) explore the extent to which preferences are consistent across the two stakeholder groups. The output of this phase will be the study's main findings.- Reporting and dissemination. Findings will be widely disseminated to policy and research communities using a number of channels, including peer reviewed publications, policy briefs and videos.

研究委员会“鼓励研究人员从一开始就考虑他们的研究可以对经济和社会做出的潜在贡献，以及开展适当和项目具体知识交流影响活动所需的资源”。作为资助申请的一部分，研究人员必须提交“影响之路”部分，该部分由审稿人和小组成员进行同行评审。同样，资助委员会将使用案例研究方法来评估影响，作为卓越研究框架的一部分。案例研究将由学术同行和非学术专家进行审查，提供私营、公共和第三部门的视角。然而，在评估这些影响声明的充分性/质量时，审查人员目前无法利用受益人（即普通大众）或研究生产者（即生物医学和健康研究人员）观点的全面证据来证明其建议的资格或合理性。目前尚不清楚受益者如何评价研究的影响力、这些价值观在不同群体之间有何不同，以及更根本的是英国民众是否理解研究的影响力。在缺乏关于研究受益者和资助者价值的知识和评估方法的情况下，指导研究人员估计影响和研究资助者评估此类影响的政策依赖于薄弱的经验基础。本研究旨在通过改进和调整基于调查的方法（称为“最佳-最差尺度”（BWS））来分析普通大众和研究人员对研究影响的相对评估，从而弥补这一差距。这项研究将重点关注生物医学和健康研究，并将有助于实现 MRC 资助研究的目标，即“就学术研究的贡献和 MRC 对此的贡献提供用户/受益人的看法”。拟议的研究包括 5 个阶段：- 案头和定性研究影响识别和分类的研究。彻底的案头审查将对各种类型的影响和分类模型进行分类。然后，我们将通过与公众的四个焦点小组以及对生物医学和健康研究人员的 24 次访谈来完善和发展这种分类。此阶段的输出将是一个概念框架，其中包含不同类型的生物医学和健康相关影响以及每个影响类别的不同强度，供 BWS 调查使用。- 开展定量调查。基于第一阶段，我们将设计并测试一个调查工具，以得出不同类型影响的相对排名。该问卷将通过认知访谈和通过公众和研究人员试点的完整实地工作程序进行测试。此阶段的输出将是最终的调查工具。- 与研究人员和公众进行实地调查。我们将使用在线小组对 1000 名公众进行调查，并利用现有 MRC 资助者名单对 500 名健康和生物医学科学家进行调查。此阶段的输出将是用于分析的数据集。 - 研究人员和公众偏好的分析。分析将涉及两个步骤。首先，我们将进行审查和探索性统计分析，以确定建模工作中需要考虑的关联和趋势。这将包括关键社会经济和人口变量的样本构成，并检查受访者对调查的理解（他们是否能够回答问题）。其次，我们将开发一个 BWS 模型，以 (i) 得出公众以及健康和生物医学科学家的影响陈述的完整排名，以及 (ii) 探索两个利益相关者群体偏好的一致程度。此阶段的输出将是研究的主要发现。- 报告和传播。研究结果将通过多种渠道广泛传播给政策和研究界，包括同行评审出版物、政策简报和视频。

项目成果

Understanding the relative valuation of research impact: a best-worst scaling experiment of the general public and biomedical and health researchers.

了解研究影响的相对评估：公众以及生物医学和健康研究人员的最佳-最差规模实验。

• DOI: http://dx.10.1136/bmjopen-2015-010916
• 发表时间: 2016
• 期刊: BMJ open
• 影响因子: 2.9
• 作者: Pollitt A