Total Recall via Keyqueries: A Case Study for Systematic Reviews

Researchers conduct systematic reviews to gain and build a comprehensive un-derstanding of a studied (cid:28)eld. During the screening of documents, researchers aim for total recall to ensure that all relevant documents are covered in their systematic review. Creating a systematic review is time-consuming and can take several years. Systems for technology-assisted systematic reviews incorpo-rate user feedback, whether a document was relevant or not, to learn presenting more yet unknown potential relevant documents. We propose a system that automatically creates so-called keyqueries which rank the known relevant documents in the top results of a reference search engine. This keyquery approach is motivated by research on related work search, where keyqueries retrieve additional related work for a given set of documents. Therefore, we construct keyqueries for the documents labeled as relevant in the systematic review to identify new, potentially relevant documents. We compare our keyquery-based approach with four classical machine-learning approaches and the state-of-the-art approach on three simulated systematic reviews with biological research topics. The Evaluation shows that our keyquery-based approach outperforms our implementation of logistic regression and decision table, and is comparable to a random forest approach. The state-of-the-art and our naive Bayes approach both outperform our keyquery-based approach.

研究人员进行系统综述以获得并建立对所研究领域的全面理解。在文献筛选过程中，研究人员力求全面回忆，以确保系统综述涵盖所有相关文献。创建系统综述耗时且可能需要数年时间。技术辅助系统综述系统纳入用户反馈（即文献是否相关），以便学习呈现更多未知的潜在相关文献。我们提出一种系统，它能自动创建所谓的关键查询，这些关键查询能将已知相关文献在参考搜索引擎的结果顶部进行排序。这种关键查询方法是受相关工作搜索研究的启发，在相关工作搜索中，关键查询可为给定的一组文献检索额外的相关工作。因此，我们为系统综述中标记为相关的文献构建关键查询，以识别新的、潜在相关的文献。我们在三个具有生物学研究主题的模拟系统综述中，将我们基于关键查询的方法与四种经典机器学习方法以及最先进的方法进行比较。评估表明，我们基于关键查询的方法优于我们实现的逻辑回归和决策表方法，并且与随机森林方法相当。最先进的方法和我们的朴素贝叶斯方法都优于我们基于关键查询的方法。