Real Time Anomaly Detection Techniques Using PySpark Frame Work
Author:
- 31 March 2020
- journal article
- research article
- Published by Inventive Research Organization in Journal of Artificial Intelligence and Capsule Networks
- Vol. 2 (1) , 20-30
Abstract
The identification of anomaly in a network is a process of observing keenly the minute behavioral changes from the usual pattern followed. These are often referred with different names malware, exceptions, and anomaly or as outlier according to the dominion of the application. Though many works have emerged for the detection of the outlier, the identification of the abnormality in the multiple source data stream structure is still under research. To identify the abnormalities in the cloud data center that is encompassed with the multiple-source VMWare, by observing the behavioral changes in the load of the CPU, utilization of the memory etc. consistently the paper has developed a real time identification process. The procedure followed utilizes the PySpark to compute the batches of data and make predictions, with minimized delay. Further a flat-increment based clustering is used to frame the normal attributes in the PySpark Structure. The latencies in computing the tuple while clustering and predicting, was compared for PySpark, Storm and other dispersed structure that were used in processing the batches of data and was experimentally found that the processing time of tuple in a PySpark was much lesser compared to the other methods.Keywords
This publication has 11 references indexed in Scilit:
- Network Traffic Anomaly Detection based on Apache Spark2019
- Development of Scalable On-Line Anomaly Detection System for Autonomous and Adaptive Manufacturing ProcessesApplied Sciences, 2019
- PERFORMANCE EVALUATION OF ROUTING ALGORITHM FOR MANET BASED ON THE MACHINE LEARNING TECHNIQUESJournal of Trends in Computer Science and Smart Technology, 2019
- DDOS ATTACK DETECTION IN TELECOMMUNICATION NETWORK USING MACHINE LEARNINGJournal of Ubiquitous Computing and Communication Technologies, 2019
- Secured Self Organizing Network Architecture in Wireless Personal NetworksWireless Personal Communications, 2017
- Anomaly Detection2016
- Anonymization in Social Ne tworks: A Survey on the issues of Data Privacy in Social Network SitesInternational Journal Of Engineering And Computer Science, 2016
- Anomaly detection in network traffic using K-mean clustering2016
- Discretized Streams: A Fault-Tolerant Model for Scalable Stream Processing2012
- Incremental Clustering and Dynamic Information RetrievalSIAM Journal on Computing, 2004