Comparative analysis of large data processing in Apache Spark using Java, Python and Scala
Preprint
Abstract
During the study, the results of a comparative analysis of the process of handling large datasets using the Apache Spark platform in Java, Python, and Scala programming languages were obtained. Although prior works have focused on individual stages, comprehensive comparisons of full ETL workflows across programming languages using Apache Iceberg remain limited. The analysis was performed by executing several operations, including downloading data from CSV files, transforming and loading it into an Apache Iceberg analytical table. It was found that the performance of the Spark algorithm varies significantly depending on the amount of data and the programming language used. When processing a 5-megabyte CSV file, the best result was achieved in Python: 6.71 seconds, which is superior to Scala's score of 9.13 seconds and Java's time of 9.62 seconds. For processing a large CSV file of 1.6 gigabytes, all programming languages demonstrated similar results: the fastest performance was showed in Python: 46.34 seconds, while Scala and Java showed results of 47.72 and 50.56 seconds, respectively. When performing a more complex operation that involved combining two CSV files into a single dataset for further loading into an Apache Iceberg table, Scala demonstrated the highest performance, at 374.42 seconds. Java processing was completed in 379.8 seconds, while Python was the least efficient, with a runtime of 398.32 seconds. It follows that the programming language significantly affects the efficiency of data processing by the Apache Spark algorithm, with Scala and Java being more productive for processing large amounts of data and complex operations, while Python demonstrates an advantage in working with small amounts of data. The results obtained can be useful for optimizing data handling processes depending on specific performance requirements and the amount of information being processed.Keywords
All Related Versions
This publication has 14 references indexed in Scilit:
- A large-scale lychee image parallel classification algorithm based on spark and deep learningComputers and Electronics in Agriculture, 2025
- Data Lakehouse: A survey and experimental studyInformation Systems, 2024
- BigLake: BigQuery's Evolution toward a Multi-Cloud Lakehouse2024
- The Lakehouse: State of the Art on Concepts and TechnologiesSN Computer Science, 2024
- Spatial big data architecture: From Data Warehouses and Data Lakes to the LakeHouseJournal of Parallel and Distributed Computing, 2023
- Framing Apache Spark in life sciencesHeliyon, 2023
- Distributed big data analysis using spark parallel data processingBulletin of Electrical Engineering and Informatics, 2022
- Towards Low-Latency Big Data Infrastructure at Sangfor2022
- A comprehensive performance analysis of Apache Hadoop and Apache Spark for large scale data sets using HiBenchJournal of Big Data, 2020
- Performance Analysis of Distributed Computing Frameworks for Big Data Analytics: Hadoop Vs SparkComputación y Sistemas, 2020