Lightning Fast Distributed Machine Learning Framework

Madhu Sudhan H V

Abstract: According to IBM, Big Data can be expressed with 4 Vs, namely Volume, Velocity, Variety and Veracity. Lots of companies are incorporating Big Data in their business model to derive insights from the unstructured data. Big Data is analyzed using Statistical Methods and Machine Learning. Limiting factor using traditional technologies is the incompetence to use huge amounts of data to learn or train algorithms within a practical time. This problem can be handled by using in-memory and distributed machine learning techniques with the help of distributed data sets and by allocating learning to various workstations. A distributed machine learning framework is developed with Spark, Hadoop and Python to scale the machine learning algorithm and to reduce the intensive computation.

Keywords: Big Data, Distributed Machine Learning, Apache Spark, Python