Scheduling functional regression tests for IBM DB2 products

Functional Regression Testing (FRT) is performed to ensure that a new version of a product functions properly as designed. In a corporate environment, the large numbers of test jobs and the complexity of scheduling the jobs on different platforms make performance of this testing an important issue. A grid provides an infrastructure for applications to use shared heterogeneous resources. Such an infrastructure may be used to solve large-scale testing problems or to improve application performance. FRT is a good candidate application for running on a grid because each test job can run separately, in parallel. However, experience indicates that such applications may suffer performance problems without a proper cost-based grid scheduling strategy.The Database Technology (DBT) Regression Test Team at IBM conducts the FRT for IBM® DB2® Universal DatabaseTM (DB2 UDB) products. As a case study, we examined the current test scheduling approach for the DB2 products. We found that the performance of the test scheduler suffers because it does not incorporate cost-dependent selection of jobs and slaves (testing IDs). Therefore, we have replaced the DB2 test scheduler with one that estimates jobs' run times, and then chooses slaves using those times. Although knowing a job's actual run time is difficult, we can use case-based reasoning to estimate it based on past experience. We create a case base to store historical data, and design an algorithm to estimate new jobs' run times by identifying cases that have executed in the past. The performance evaluation of our new scheduler shows a significant performance benefit over the original scheduler. In this paper, we also examine how machine specifications, such as the number of slaves running on a machine and the machine speed, affect application performance and run time estimation accuracy.

执行功能回归测试（FRT）是为了确保产品的新版本能按设计正常运行。在企业环境中，大量的测试任务以及在不同平台上安排任务的复杂性使得这种测试的性能成为一个重要问题。网格为应用程序使用共享的异构资源提供了一种基础设施。这种基础设施可用于解决大规模测试问题或提高应用程序性能。FRT是一个适合在网格上运行的应用程序，因为每个测试任务都可以并行地单独运行。然而，经验表明，如果没有适当的基于成本的网格调度策略，这类应用程序可能会出现性能问题。 IBM的数据库技术（DBT）回归测试团队为IBM® DB2®通用数据库（DB2 UDB）产品进行FRT。作为一个案例研究，我们研究了DB2产品当前的测试调度方法。我们发现测试调度器的性能不佳，因为它没有结合对任务和从节点（测试标识）的基于成本的选择。因此，我们用一个能估计任务运行时间，然后根据这些时间选择从节点的调度器取代了DB2测试调度器。尽管了解一个任务的实际运行时间很困难，但我们可以使用基于案例的推理，根据过去的经验来估计它。我们创建一个案例库来存储历史数据，并设计一种算法，通过识别过去执行过的案例来估计新任务的运行时间。对我们新调度器的性能评估表明，它比原来的调度器有显著的性能优势。在本文中，我们还研究了机器规格，如一台机器上运行的从节点数量和机器速度，如何影响应用程序性能和运行时间估计的准确性。