Using CrowdLogger for in situ information retrieval system evaluation

A major hurdle faced by many information retrieval researchers---especially in academia---is evaluating retrieval systems in the wild. Challenges include tapping into large user bases, collecting user behavior, and modifying a given retrieval system. We outline several options available to researchers to overcome these challenges along with their advantages and disadvantages. We then demonstrate how CrowdLogger, an open-source browser extension for Firefox and Google Chrome, can be used as an in situ evaluation platform.

许多信息检索研究人员（尤其是在学术界）面临的一个主要障碍是在实际环境中评估检索系统。挑战包括利用大量用户群体、收集用户行为以及修改给定的检索系统。我们概述了研究人员为克服这些挑战可采用的几种选择及其优缺点。然后，我们展示了CrowdLogger（一款适用于火狐和谷歌浏览器的开源浏览器扩展）如何能够用作一个原位评估平台。