Knowledge Production Processes: An Analysis of Research Perseverance and the File Drawer Bias in Social Science Survey Experiments

The scientific process is difficult to evaluate because many of its stages typically evade observation. This includes whether one has success in obtaining funding for data collection, whether one perseveres should their funding application fail, whether one writes up the results of data analyses and submits a manuscript to a journal, and of those submissions whether publication occurs. Using data from applicants to a unique grant program to fund probability-sample survey experiments in the U.S. (Time-sharing Experiments for the Social Sciences), Moniz, Druckman, and Freese identify factors that influence each step. They find that research time, and not resources, plays a substantial role in determining whether the grant is funded, and, if not, whether the applicant proceeds with the project. The latter result likely reflects the availability of cheaper non-probability sample data sources that still require time to collect. Additionally, they document the substantial influence of obtaining statistically significant results in determining whether a scholar writes up and submits a paper (a variation of file drawer bias). Once a manuscript is submitted, however, statistical significance does not influence publication likelihood at all. Thus, file drawer bias emerges from researcher rather than editorial choices. The bias also is substantially smaller than it was a decade ago (Franco et al. 2014), suggesting increased recognition of the importance of null results. Overall, the researchers’ findings identify how research time and statistical significance shape science, at least in the broad domain of survey experimental research, providing guidance for potential interventions in the scientific process.

科学过程很难评估，因为它的许多阶段通常逃避观察。发表的手稿，以及是否出版的那些提交。美国（社会科学的时间共享实验），莫尼兹（Moniz），德鲁克（Druckman）和弗里斯（Freese）都会发现他们发现该研究时间而不是资源的因素，在确定赠款是否是资助的情况下起着重要的作用不是，适用的收益是否可以反映出仍然需要时间收集的廉价的非概率样本数据源的可用性。在确定学者是否撰写并提交论文时，获得统计上有意义的结果（文件抽屉偏见的变化）。从研究人员而不是编辑选择中出现。研究人员的发现提出了对无效结果的重要性的认识，研究时间和统计学意义如何影响了调查实验研究的广泛领域，从而为科学过程中的潜在干预提供了指导。