Distributed pattern matching and document analysis in big data using Hadoop MapReduce model
Authors:
- 1 December 2014
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 312-317
Abstract
Sequential pattern mining and Document analysis is an important data mining problem in Big Data with broad applications. This paper investigates a specific framework for managing distributed processing in dataset pattern match and document analysis context. MapReduce programming model on a Hadoop cluster is highly scalable and works with commodity machines with integrated mechanisms for fault tolerance. In this paper, we propose a Knuth Morris Pratt based sequential pattern matching in distributed environment with the help of Hadoop Distributed File System as efficient mining of sequential patterns. It also investigates the feasibility of partitioning and clustering of text document datasets for document comparisons. It simplifies the search space and acquires a higher mining efficiency. Data mining tasks has been decomposed to many Map tasks and distributed to many Task trackers. The map tasks find the intermediate results and send to reduce task which consolidates the final result. Both theoretical analysis and experimental result with data as well as cluster of varying size shows the effectiveness of MapReduce model primarily based on time requirements.Keywords
This publication has 9 references indexed in Scilit:
- A comparison of join algorithms for log processing in MaPreduce2010
- MapReduce and parallel DBMSsCommunications of the ACM, 2010
- A taxonomy based semantic similarity of documents using the cosine measure2009
- A comparison of approaches to large-scale data analysis2009
- Application of PrefixSpan* Algorithm in Malware Detection Expert System2009
- SCOPEProceedings of the VLDB Endowment, 2008
- Pig latin2008
- MapReduceCommunications of the ACM, 2008
- Mining sequential patterns by pattern-growth: the PrefixSpan approachIEEE Transactions on Knowledge and Data Engineering, 2004