A comprehensive performance analysis of Apache Hadoop and Apache Spark for large scale data sets using HiBench

Комплексный анализ производительности Apache Hadoop и Apache Spark для наборов данных большого объёма с использованием HiBench
Nasim Ahmed, Teo Sušnjak, Andre L. C. Barczak, Mohammad A. Rashid
2020-12-01

Apache HadoopApache SparkHiBenchTeraSortparameter tuning
Abstract Big Data analytics for storing, processing, and analyzing large-scale datasets has become an essential tool for the industry. The advent of distributed computing frameworks such as Hadoop and Spark offers efficient solutions to analyze vast amounts of data. Due to the application programming interface (API) availability and its performance, Spark becomes very popular, even more popular than the MapReduce framework. Both these frameworks have more than 150 parameters, and the combination of these parameters has a massive impact on cluster performance. The default system parameters help the system administrator deploy their system applications without much effort, and they can measure their specific cluster performance with factory-set parameters. However, an open question remains: can new parameter selection improve cluster performance for large datasets? In this regard, this study investigates the most impacting parameters, under resource utilization, input splits, and shuffle, to compare the performance between Hadoop and Spark, using an implemented cluster in our laboratory. We used a trial-and-error approach for tuning these parameters based on a large number of experiments. In order to evaluate the frameworks of comparative analysis, we select two workloads: WordCount and TeraSort. The performance metrics are carried out based on three criteria: execution time, throughput, and speedup. Our experimental results revealed that both system performances heavily depends on input data size and correct parameter selection. The analysis of the results shows that Spark has better performance as compared to Hadoop when data sets are small, achieving up to two times speedup in WordCount workloads and up to 14 times in TeraSort workloads when default parameter values are reconfigured.
1
A trial-and-error tuning process identifies parameter configurations that substantially affect cluster performance across WordCount and TeraSort workloads.
2
After reconfiguring default parameters, Spark outperforms Hadoop on small datasets, achieving up to twofold speedup for WordCount and up to 14-fold speedup for TeraSort.
3
Both Hadoop and Spark performance depend strongly on input data size and appropriate parameter selection.
4
Performance is evaluated using execution time, throughput, and speedup on a laboratory-implemented cluster.
5
The study compares Apache Hadoop and Apache Spark using HiBench, focusing on resource utilization, input splits, and shuffle parameters.

Apache Hadoop and Apache Spark distributed computing frameworks processing large-scale datasets

The effects of parameter selection, input data size, resource utilization, input splits, and shuffle on execution time, throughput, and speedup in WordCount and TeraSort workloads

Publication Details
Publication Date
2020-12-01
Journal
Publisher
ISSN
Cited by
125
Access Type
Author Information
Authors
Nasim Ahmed
Teo Sušnjak
Andre L. C. Barczak
Mohammad A. Rashid
Explore further
Open the scid.ai AI chat with a ready-made request: it will find papers on a similar topic and help build a literature review.
Find similar papers in the chat →
Make a presentation
100%