Scheduling functional regression tests for IBM DB2 products

Functional Regression Testing (FRT) is performed to ensure that a new version of a product functions properly as designed. In a corporate environment, the large numbers of test jobs and the complexity of scheduling the jobs on different platforms make performance of this testing an important issue. A grid provides an infrastructure for applications to use shared heterogeneous resources. Such an infrastructure may be used to solve large-scale testing problems or to improve application performance. FRT is a good candidate application for running on a grid because each test job can run separately, in parallel. However, experience indicates that such applications may suffer performance problems without a proper cost-based grid scheduling strategy.The Database Technology (DBT) Regression Test Team at IBM conducts the FRT for IBM® DB2® Universal DatabaseTM (DB2 UDB) products. As a case study, we examined the current test scheduling approach for the DB2 products. We found that the performance of the test scheduler suffers because it does not incorporate cost-dependent selection of jobs and slaves (testing IDs). Therefore, we have replaced the DB2 test scheduler with one that estimates jobs' run times, and then chooses slaves using those times. Although knowing a job's actual run time is difficult, we can use case-based reasoning to estimate it based on past experience. We create a case base to store historical data, and design an algorithm to estimate new jobs' run times by identifying cases that have executed in the past. The performance evaluation of our new scheduler shows a significant performance benefit over the original scheduler. In this paper, we also examine how machine specifications, such as the number of slaves running on a machine and the machine speed, affect application performance and run time estimation accuracy.

执行功能回归测试（FRT）是为了确保新版本的产品按设计正常运行。在公司环境中，大量的测试作业和在不同平台上调度作业的复杂性使得测试的性能成为一个重要问题。网格为应用程序提供了使用共享异构资源的基础设施。这样的基础结构可用于解决大规模测试问题或提高应用程序性能。FRT是在网格上运行的一个很好的候选应用程序，因为每个测试作业可以单独并行运行。然而，经验表明，如果没有适当的基于成本的网格调度策略，这些应用程序可能会遇到性能问题。IBM的数据库技术（DBT）回归测试团队对IBM®DB2®Universal DatabaseTM （DB2 UDB）产品进行FRT。作为一个案例研究，我们研究了DB2产品的当前测试调度方法。我们发现测试调度器的性能会受到影响，因为它没有包含与成本相关的作业和从（测试id）选择。因此，我们将DB2测试调度器替换为估算作业运行时间的调度器，然后使用这些时间选择从机。虽然知道作业的实际运行时间是困难的，但是我们可以使用基于案例的推理，根据过去的经验来估计它。我们创建了一个案例库来存储历史数据，并设计了一种算法，通过识别过去执行过的案例来估计新作业的运行时间。我们的新调度器的性能评估显示，与原来的调度器相比，它具有显著的性能优势。在本文中，我们还研究了机器规范（例如在一台机器上运行的从机数量和机器速度）如何影响应用程序性能和运行时估计精度。