Streaming Big Data with Open-source: A Comparative Study and Architectural Recommendations
Authors:
- 23 March 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1420-1425
Abstract
This study focusses on providing state-of-the-art infrastructure for data pipelines in e-Commerce sector, especially for online stores. With people going digital and also latest impact of Covid-19, daily e-Commerce companies are dealing with large amount of data (terabytes to petabytes). With growing Internet of Things, systems of computing devices which are interrelated. The inter-relation may be between mechanical and digital machines, objects or people. The interrelated objects will be provided with unique identifiers and the ability to transfer data over a network without requiring human-to-human or human-to-computer interaction. Growth of big data poses several challenges and opportunities in every field of its usage. Realtime analysis of data and its inference gives a competitive edge over its partners in every business field especially in e-commerce. Recent advances in technology and tools have exposed new opportunities to get actionable insights from historical data like market data, customer demographics, along with real-time data. Advancement in distributed streaming technology makes it important to investigate existing streaming data pipeline capabilities in eCommerce sector with a focus on online stores. This study analyzes the published research works on streaming data pipelines in e-commerce sector also to facilitate e-commerce’s variety of data streaming applications requirement. A state-of-the-art lambda architecture for streaming is proposed completely based on open-source technologies. Challenge in proprietary owned streaming platforms are vendor lock-in, limited ability to customize, cost, limited innovation & support. Proposed reference architecture will address many streaming use cases compared to its competitors, it has support of large open-source community in providing the inter-operability between streaming & related technologies like connectors, apart from providing better performance apart from other open-source based product advantages.Keywords
This publication has 17 references indexed in Scilit:
- Secure Data Storage and Sharing Techniques for Data Protection in Cloud Environments: A Systematic Review, Analysis, and Future DirectionsIEEE Access, 2022
- SPR Based Biosensing Chip for COVID-19 Diagnosis—A ReviewIEEE Sensors Journal, 2022
- Comparing AWS Streaming Services: A Use Case on ECG Data Streams2022
- Network Traffic Impact on Cloud Usage at Different Providers2022
- Managing Multi-Cloud Deployments on Kubernetes with Istio, Prometheus and Grafana2022
- A Critical Review of Cloud Computing Environment for Big Data Analytics2022
- Apache Spark based analysis on word count application in Big Data2022
- A Risk Model Based Heart Disease Prediction Using Data Lake Architecture2022
- A Survey on High-Throughput Non-Binary LDPC Decoders: ASIC, FPGA, and GPU ArchitecturesIEEE Communications Surveys & Tutorials, 2021
- The Challenges and Opportunities in the Digitalization of Companies in a Post-COVID-19 WorldIEEE Engineering Management Review, 2020