An efficient hybrid optimization of ETL process in data warehouse of cloud architecture
Authors:
Open Access
- 8 January 2024
- journal article
- research article
- Published by Springer Nature in Journal of Cloud Computing
- Vol. 13 (1) , 1-12
Abstract
In big data, analysis data is collected from different sources in various formats, transforming into the aspect of cleansing the data, customization, and loading it into a Data Warehouse. Extracting data in other formats and transforming it to the required format requires transformation algorithms. This transformation stage has redundancy issues and is stored across any location in the data warehouse, which increases computation costs. The main issues in big data ETL are handling high-dimensional data and maintaining similar data for effective data warehouse usage. Therefore, Extract, Transform, Load (ETL) plays a vital role in extracting meaningful information from the data warehouse and trying to retain the users. This paper proposes hybrid optimization of Swarm Intelligence with a tabu search algorithm for handling big data in a cloud-based architecture-based ETL process. This proposed work overcomes many issues related to complex data storage and retrieval in the data warehouse. Swarm Intelligence algorithms can overcome problems like high dimensional data, dynamical change of huge data and cost optimization in the transformation stage. In this work for the swarm intelligence algorithm, a Grey-Wolf Optimizer (GWO) is implemented to reduce the high dimensionality of data. Tabu Search (TS) is used for clustering the relevant data as a group. Clustering means the segregation of relevant data accurately from the data warehouse. The cluster size in the ETL process can be optimized by the proposed work of (GWO-TS). Therefore, the huge data in the warehouse can be processed within an expected latency.Keywords
This publication has 45 references indexed in Scilit:
- An automated data verification approach for improving data quality in a clinical registryComputer Methods and Programs in Biomedicine, 2019
- H∞ Consensus for Multiagent-Based Supply Chain Systems Under Switching Topology and Uncertain DemandsIEEE Transactions on Systems, Man, and Cybernetics: Systems, 2018
- Big Data Quality Assessment Model for Unstructured Data2018
- Device-Free Motion & Trajectory Detection via RFIDACM Transactions on Embedded Computing Systems, 2018
- Situation-Aware Dynamic Service Coordination in an IoT EnvironmentIEEE/ACM Transactions on Networking, 2017
- Taming Big Data: Implementation of a Clinical Use-Case Driven ArchitectureInternational Journal of Radiation Oncology - Biology - Physics (IJROBP), 2016
- Performance optimization for extraction, transformation, loading and reporting of data2015
- A Survey of Extract–Transform–Load TechnologyInternational Journal of Data Warehousing and Mining, 2009
- Extraction, Transformation, and Loading2009
- Designing ETL processes using semantic web technologies2006