ReHRS: A Hybrid Redundant System for Improving MapReduce Reliability and Availability

MapReduce is a parallel programming framework proposed by Google. Recently, it has become a popular technology for solving data-intensive applications. However, current MapReduce implementations provide insufficient redundant mechanisms for their master servers, consequently causing the fact that the master servers' services cannot continue and all jobs cannot proceed and complete when the master servers unexpectedly fail. To solve this problem, this chapter proposes a master server redundant mechanism called the Reliable Hybrid Redundant System (ReHRS for short), in which a hot-standby server is employed to maintain the latest metadata of the master sever so as to achieve a fast takeover, and a warm-standby server is employed to further enhance system reliability and extend the operation of MapReduce when both the master server and hot-standby server cannot work properly. We proposed a failure detection algorithm to detect the failure of the master server and hot-standby server, and provided appropriate takeover processes to continue their operations. Additionally, we introduced a dynamic warmup mechanism for the warm-standby server to warm itself up such that it can quickly act as the hot-standby server when necessary. The extensive simulation and experiment results show that the ReHRS significantly speeds up the takeover process as compared with three state-of-the-art schemes.