A Semantic and Optimized Focused Crawler Based on Semantic Graph and Genetic Algorithm

Authors:
Open Access
Publisher website
Google Scholar
Abstract
A focused crawler automatically retrieves, organizes, and extracts specific topic-related information from the internet for analysis and application. Currently, most focused crawlers assess the relevance of web pages to a given topic through methods such as keyword matching, semantic analysis, and link structures. However, these existing focused crawlers suffer from issues such as misleading directions and reduced accuracy due to the lack of semantic analysis of topic terms, as well as biased computation of topic relevance caused by the absence of effective weighting factors. To solve the above-mentioned problems, this study proposes a semantic and optimized focused crawler based on Semantic Graph and Genetic Algorithm. The proposed crawler eliminates ambiguous terms by constructing a semantic graph, optimizes the weighting factors of topic relevance with asymmetry by using a genetic algorithm, and combines both above two points to predict the priority of each unvisited hyperlink. The experiment results indicate that the proposed SG-GA Crawler improves the evaluation indicators compared with the other three focused crawlers, including VSM Crawler, SSRM Crawler, and SG Crawler. More specifically, the percentage improvement achieved by the proposed method exceeds 19%, 19%, and 13% in terms of three evaluation indicators, including the number of relevant web pages, acquisition rate, and average relevance, respectively. In conclusion, the proposed focused crawler can grab more quantity and higher quality topic-related web pages from the Internet.
Funding Information
  • National Natural Science Foundation of China (61070015, 2022GCZX013, ZNJW2024KFQN012, SDNY-2024001, D202405310814162131)
  • Guangdong Provincial Key Technology Research Center for Object Digitization and Epidemic Intelligence Prevention and Control, Guangdong Provincial Engineering Technology Research Center for General Colleges and Universities, Guangdong Provincial Departmen (61070015, 2022GCZX013, ZNJW2024KFQN012, SDNY-2024001, D202405310814162131)
  • Intelligent Policing Key Laboratory of Sichuan Province (61070015, 2022GCZX013, ZNJW2024KFQN012, SDNY-2024001, D202405310814162131)
  • Open Research Subject of Sichuan Provincial Engineering Research Center of Hydroelectric Energy Power Equipment Technology (61070015, 2022GCZX013, ZNJW2024KFQN012, SDNY-2024001, D202405310814162131)
  • College Student Innovation and Entrepreneurship Training Project of Sichuan Province (61070015, 2022GCZX013, ZNJW2024KFQN012, SDNY-2024001, D202405310814162131)

This publication has 32 references indexed in Scilit: