Lessons Learned from Using an Online Platform to Conduct Large-Scale, Online Controlled Security Experiments with Software Developers

Security and privacy researchers are increasingly conducting controlled experiments focusing on IT professionals, such as software developers and system administrators. These professionals are typically more difficult to recruit than general end-users. In order to allow for distributed recruitment of IT professionals for security user studies, we designed Developer Observatory, a browser-based virtual laboratory platform that enables controlled programming experiments while retaining most of the observational power of lab studies. The Developer Observatory can be used to conduct largescale, reliable online programming studies with reasonable external validity. We report on our experiences and lessons learned from two controlled programming experiments (n>200) conducted using Developer Observatory.

安全和隐私研究人员越来越多地针对信息技术专业人员（如软件开发人员和系统管理员）进行对照实验。这些专业人员通常比普通终端用户更难招募。为了能够分布式招募信息技术专业人员进行安全用户研究，我们设计了“开发者观察站”，这是一个基于浏览器的虚拟实验室平台，它能够进行对照编程实验，同时保留实验室研究的大部分观察能力。“开发者观察站”可用于开展具有合理外部效度的大规模、可靠的在线编程研究。我们报告了从使用“开发者观察站”进行的两次对照编程实验（样本量n>200）中获得的经验和教训。