Distributed Offline Load Balancing in MapReduce Networks

Charalambous, T., Kalyvianaki, E., Hadjicostis, C. N. & Johansson, M. (2013). Distributed Offline Load Balancing in MapReduce Networks. Paper presented at the 2013 IEEE 52nd Annual Conference on Decision and Control (CDC), 10-12-2013 - 13-12-2013, Florence, Italy.

[img]
Preview
Text - Accepted Version
Download (497kB) | Preview

Abstract

In this paper we address the problem of balancing the processing load of MapReduce tasks running on heterogeneous clusters, i.e., clusters composed of nodes with different capacities and update cycles. We present a fully decentralized algorithm, based on ratio consensus, where each mapper decides the amount of workload data to handle for a single user job using only job specific local information, i.e., information that can be collected from directly connected neighboring mappers, regarding their current workload usage and capacity. In contrast to other algorithms in the literature, the proposed algorithm can be deployed in heterogeneous clusters and can operate asynchronously in both directed and undirected communication topologies. The performance of the proposed algorithm is demonstrated via simulation experiments on large-scale strongly connected topologies.

Item Type: Conference or Workshop Item (Paper)
Additional Information: © 2013 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.
Subjects: Q Science > QA Mathematics > QA75 Electronic computers. Computer science
Divisions: School of Informatics > Department of Computing
URI: http://openaccess.city.ac.uk/id/eprint/8182

Actions (login required)

View Item View Item

Downloads

Downloads per month over past year

View more statistics