Collaborative Research: SOS-DCI / HNDS-R: Advancing Semantic Network Analysis to Better Understand How Evaluative Exchanges Shape Scientific Arguments

合作研究：SOS-DCI / HNDS-R：推进语义网络分析，以更好地理解评估性交流如何塑造科学论证

• 批准号: 2244805
• 负责人: Andrew McCallum
• 金额: $ 22.5万
• 依托单位: University of Massachusetts Amherst
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2023
• 资助国家: 美国
• 起止时间: 2023-04-01 至 2025-03-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2244805&HistoricalAwards=false
• 关键词: Collaborative; Research; SOS; DCI; HNDS

Peer review is central to the scientific process, but much of it is hidden from those who engage in it. What issues do reviewers focus on? Do authors rebut and/or implement some of these comments more than others? Peer review is a social process where reviewers and authors exchange competing arguments about the merits of new scientific work. In these exchanges, authors advance new arguments in their submissions, which are critically evaluated by peer reviewers who contest some of these arguments and affirm others. Authors then respond to these reviews with counter-arguments, sometimes conceding and other times refuting. This process of peer review - the exchange of argument, evaluation, and response – is a social, institutional means of shaping what is published as scientific knowledge. This project will systematically depict this exchange process and its variation, revealing how different fields establish socially accepted knowledge claims. We intend to do this by: representing the hierarchical logical dependencies of scientific arguments; discerning distinct expressions of epistemological value; and developing a method that identifies where and when such evaluations are intertextually represented. When these patterns of exchange and evaluation are viewed in the aggregate, they will offer authors, reviewers, editors, and field participants a means by which to observe these hidden epistemological deliberations, to reflect on their merits, and to potentially help participants advocate for improvements in peer review. The techniques we develop should also enable social scientists to systematically study evaluative exchanges more generally, and this approach can be used to see how evaluative norms are practiced in different social and institutional contexts.Using tens of thousands of evaluative exchanges and nearly 100,000 scholarly texts of OpenReview and JSTOR, we will systematically study evaluative exchanges among peer scholars over the course of three interrelated phases: how authors advance arguments, how peers evaluate and review them, and how authors counterargue and revise their manuscript in response to those reviews. We will apply natural language processing techniques including discourse parsing, rhetorical structural theory, and network analysis to investigate evaluative exchanges in both private peer reviews and public review-and-responses. The work has three aims: first, to discover the semantic structure of scientific arguments and how they vary within and across distinct scholarly fields. This entails understanding the kinds of claims, warrants, and evidence scientists use to debate; how these are interrelated in network forms or structures; and how certain forms in turn are specific to different scientific cultures. Second, we aim to discover the latent inter-textual structure of scientific arguments and, in particular, how responding scientists forge arguments that rearrange, omit, or even emphasize the semantic relations of other scientists’ arguments as laid out in texts. This entails innovating “text as data” methods in order to observe and measure discrete connections across disparate texts. Finally, we aim to understand the conditions under which scientists change their minds: how their arguments get revised and how resolution is achieved. This entails understanding whether and to what extent argument structures are updated or aligned by counter- or responding arguments. The project is co-funded by the Science of Science: Discovery, Communications, and Impact and the Human Networks and Data Science Programs.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

同行评审是科学过程的核心，但是其中大部分对参与其中的人隐藏了。审阅者专注于哪些问题？作者是否比其他评论反驳和/或实施其中一些评论？同行评审是一个社会过程，审稿人和作者在其中交流了有关新科学工作优点的竞争论点。在这些交流中，作者在提交的提出中提出了新的论点，这些论点是由同伴审稿人对这些论点进行了批判性评估并影响了其他论点的。然后，作者用反对意见回应了这些评论，有时会承认，而有时会驳斥这些评论。同行评审的这一过程 - 论证，评估和回应的交流 - 是一种塑造作为科学知识出版的内容的社会机构手段。该项目将系统地描述此交换过程及其变化，揭示不同领域如何建立社会接受的知识主张。我们打算通过：代表科学论证的等级逻辑依赖性；辨别认识论价值的不同表达；并开发一种识别互文表示此类评估的何时何地的方法。当这些交换和评估模式在总体中查看时，它们将为作者，审稿人，编辑和现场参与者提供一种观察这些隐藏的认识论审议，反思其优点的手段，并有可能帮助参与者主张改进同行评审。 The techniques we develop should also enable social scientists to systematically study evaluated exchanges more generally, and this approach can be used to see how evaluated norms are practiced in different social and institutional contexts.Using tens of thousands of evaluation exchanges and almost 100,000 scientific texts of OpenReview and JSTOR, we will systematically study evaluated exchanges among peer scholars over The course of three interrelated phases: how authors advance arguments, how同行评估和审查它们，以及作者如何反对和修改他们的手稿，以响应这些评论。我们将应用自然语言处理技术在内，包括话语解析，修辞结构理论和网络分析来调查私人同行评审和公众审查和响应中的评估交流。这项工作有三个目的：首先，发现科学论证的语义结构以及它们在不同的科学领域中的变化。这需要了解科学家用来辩论的主张，认股权证和证据。它们如何在网络形式或结构中相互关联；以及某些形式又如何特异性地针对不同的科学文化。其次，我们旨在发现科学论证的潜在跨文本结构，尤其是回应科学家如何伪造的论点，即重新安排，忽略甚至强调其他科学家论证的语义关系，如文本中所述。这需要创新的“文本作为数据”方法，以观察和衡量跨不同文本的离散连接。最后，我们的目标是了解科学家改变主意的条件：他们的论点如何得到修改以及如何实现解决方案。这需要理解是否以及在何种程度上通过反词或响应论点更新或对齐。该项目由科学科学：发现，沟通和影响以及人类网络和数据科学计划共同资助。该奖项反映了NSF的法定使命，并使用基金会的知识分子优点和更广泛的影响评估审查标准，被认为是珍贵的支持。