Data Balancing Scheme for Multi-node Heterogeneous Hadoop Cluster

Indresh B. Rajwade, Er. Prateek Singh

Abstract: Big data encompasses huge amount of information from multiple internal and external resources such as transactions, social media, enterprise content, sensors and mobile devices. It is characterized as volume, velocity, variety and veracity. MapReduce is a parallel computing framework which meets the tremendous needs for large scale data processing. Due to its simplicity, robustness and scalability MapReduce has been widely used by the companies such as Amazon, Facebook and Yahoo! to process large volumes of data on a daily basis. The MapReduce framework simplifies the complexity of running distributed data processing functions across multiple nodes in a cluster. It automatically handles the gathering of results across the multiple nodes and returns a single result or a set. Hadoop is an open source implementation of MapReduce which balances the load in a cluster by distributing data to multiple nodes based on disk space availability and processing efficiency. In this dissertation, the evaluation of data placement mechanism in a heterogeneous Hadoop cluster is performed using Grep tool and WordCount program. These are two MapReduce applications running on Hadoop clusters. A comparison has been done with Grep and WordCount through Ubuntu 14.04 LTS for three nodes in a Hadoop cluster and it is observed that the computing ratios of a Hadoop cluster are application dependent and size independent. This means that if the configuration of a cluster is updated, computing ratios must be determined again.

Keywords: Big Data, Hadoop, MapReduce, HDFS, Grep, WordCount, Heterogeneous cluster