Complications for Computational Experiments from Modern Processors

In this paper, we revisit the approach to empirical experiments for combinatorial solvers. We provide a brief survey on tools that can help to make empirical work easier. We illustrate origins of uncertainty in modern hardware and show how strong the influence of certain aspects of modern hardware and its experimental setup can be in an actual experimental evaluation. More specifically, there can be situations where (i) two different researchers run a reasonable-looking experiment comparing the same solvers and come to different conclusions and (ii) one researcher runs the same experiment twice on the same hardware and reaches different conclusions based upon how the hardware is configured and used. We investigate these situations from a hardware perspective. Furthermore, we provide an overview on standard measures, detailed explanations on effects, potential errors

在本文中，我们重新审视组合求解器的实证实验方法。我们对有助于使实证工作更轻松的工具进行了简要综述。我们阐述了现代硬件中不确定性的来源，并展示了现代硬件的某些方面及其实验设置在实际实验评估中的影响有多大。更具体地说，可能存在以下情况：（i）两位不同的研究人员进行一项看起来合理的实验，对相同的求解器进行比较，但得出不同的结论；（ii）一位研究人员在相同的硬件上两次进行相同的实验，根据硬件的配置和使用方式得出不同的结论。我们从硬件角度对这些情况进行了研究。此外，我们提供了标准度量的概述、对影响的详细解释以及潜在的错误。