[Back] [PDF

Improving the Performance of I/O-Intensive Applications on Clusters of Workstations

 Xiao Qin1    Hong Jiang2    Yifeng Zhu2    David R. Swanson2

1 Department of Computer Science
   New Mexico Institute of Mining and Technology
   801 Leroy Place, Socorro, New Mexico 87801
2 Department of Computer Science and Engineering
   University of Nebraska-Lincoln
   Lincoln, NE 68588-0115 

Load balancing in a workstation-based cluster system has been investigated extensively, mainly focusing on the effective usage of global CPU and memory resources. However, if a significant portion of applications running in the system is I/O-intensive, traditional load balancing policies that focus on CPU and memory usage may ignore I/O load imbalance, thereby causing system performance to decrease substantially. In this paper, two I/O-aware load-balancing schemes, referred to as IOCM and WAL-PM, are presented to improve the overall performance of a cluster system with a general and practical workload including I/O activities. The proposed schemes dynamically detect I/O load imbalance on nodes of a cluster, and determine whether to migrate some I/O load or running jobs from overloaded nodes to other less- or under-loaded nodes. The current running jobs are eligible to be migrated in WAL-PM only if it overall performance improves. Besides balancing I/O load, the scheme judiciously takes into account both CPU and memory load sharing in the system, thereby maintaining the same level of performance as existing schemes when I/O load is low or well balanced. Extensive trace-driven simulations for both synthetic and real I/O-intensive applications show that: (1) Compared with existing schemes that only consider CPU and memory, the proposed schemes improve performance with respect to mean slowdown by up to a factor of 20; (2) When compared to the existing approaches that only consider I/O with non-preemptive job migrations, the proposed schemes achieve improvement in mean slowdown by up to a factor of 10; (3) Under CPU-memory intensive workload, our scheme improves the performance over the existing approaches that only consider I/O by up to 47.5%.

Appeared in Cluster Computing: The Journal of Networks, Software Tools and Applications, Special Issue on Cluster Computing in Science and Engineering, vol. 9, no. 3, pp. 297-311, July 2006.