A Model for Enhancing Unstructured Big Data Warehouse Execution Time
Open Access
- 6 February 2024
- journal article
- research article
- Published by MDPI AG in Big Data and Cognitive Computing
- Vol. 8 (2) , 17
Abstract
Traditional data warehouses (DWs) have played a key role in business intelligence and decision support systems. However, the rapid growth of the data generated by the current applications requires new data warehousing systems. In big data, it is important to adapt the existing warehouse systems to overcome new issues and limitations. The main drawbacks of traditional Extract–Transform–Load (ETL) are that a huge amount of data cannot be processed over ETL and that the execution time is very high when the data are unstructured. This paper focuses on a new model consisting of four layers: Extract–Clean–Load–Transform (ECLT), designed for processing unstructured big data, with specific emphasis on text. The model aims to reduce execution time through experimental procedures. ECLT is applied and tested using Spark, which is a framework employed in Python. Finally, this paper compares the execution time of ECLT with different models by applying two datasets. Experimental results showed that for a data size of 1 TB, the execution time of ECLT is 41.8 s. When the data size increases to 1 million articles, the execution time is 119.6 s. These findings demonstrate that ECLT outperforms ETL, ELT, DELT, ELTL, and ELTA in terms of execution time.Keywords
This publication has 49 references indexed in Scilit:
- SETL: A programmable semantic extract-transform-load framework for semantic data warehousesInformation Systems, 2017
- Data Warehouse with Big Data Technology for Higher EducationProcedia Computer Science, 2017
- Defining Big Data2016
- P-ETL: Parallel-ETL based on the MapReduce paradigm2014
- CloudETL2014
- ELTA: New Approach in Designing Business Intelligence Solutions in Era of Big DataProcedia Technology, 2014
- On-Demand ELT Architecture for Right-Time BIInternational Journal of Data Warehousing and Mining, 2013
- Transactions on Large-Scale Data- and Knowledge-Centered Systems VIII2013
- Integration of Data Warehouse and Unstructured Business Documents2012
- Automatic Training Data Cleaning for Text Classification2011