Wenjun Liu

Entity disambiguation is an important task in natural language processing that addresses situations where entity names mentioned in text may correspond to multiple real-world entities, with the goal of disambiguation by mapping entity names mentioned in text to unique and unambiguous entity identifiers. At present, most researches focus on entity disambiguation based on the concept level, which mainly uses contextual information to achieve entity disambiguation. However, the above methods still have some shortcomings: the word vector representation of the traditional model can not solve the polysemy problem, and the disambiguation terms can not consider the context globally to achieve entity disambiguation. In order to solve the above problems, this paper proposes a short text entity disambiguation method based on BERT model and shortest path algorithm. In this method, BiLSTM-CRF model is used in the ambiguous entity recognition stage, cosine similarity is used to perform optimal segmentation of short texts in the text segmentation stage, BERT pre-trained model is used to construct word vectors in the semantic vector construction stage, and the shortest path algorithm (SPA) is used to screen out the correct meaning of ambiguous entities in the current context in the entity disambiguation stage. The experimental results show that the three evaluation indicators of the proposed method, namely accuracy rate, recall rate and F1 score, have increased by an average of 23.37%, 26.47% and 24.98% respectively compared with the other three entity disambiguation methods. In conclusion, the proposed method can further improve the performance of entity disambiguation.

Text summary is an information processing technology that aims to extract the important information in the text and filter out the useless information. In the research literature, text summary methods generate a text summary by clustering, supervised-based, and unsupervised-based methods. However, the K value selection of K-means clustering algorithms is manually specified, and the improper selection of the K value will lead to a poor clustering effect. At the same time, most automatic text summary methods have high redundancy. To solve the above problems, this paper proposes an automatic text summary method based on an optimized K-means clustering algorithm with symmetry and the Maximal-Marginal-Relevance (MMR) algorithm. This method uses the Genetic Algorithm with symmetry to optimize the K value selection of the K-means clustering algorithm and reduces the sentence redundancy of the text summary by using the Maximal-Marginal-Relevance algorithm. The experimental results show that the three evaluation indicators of the proposed method, namely, ROUGE-1, ROUGE-2, and ROUGE-L, have increased by an average of 96.81%, 2.39 times, and 10.15%, respectively, compared with the other three automatic text summary methods, including Lead-3, Text-Rank, and KM-MMR. In conclusion, the proposed method in this paper can obtain better-quality text summaries.

The goal of a focused crawler is to selectively fetch pages that are relevant to a given topic. Previous crawlers use text content to determine text topic relevance and manually determined weighting factors to predict the priority of unvisited URLs. However, there are still some problems in the above focused crawler methods, the calculation formula of semantic similarity between words is flawed. The weighting factor for the priority of unvisited URLs is determined arbitrarily. In order to solve the above problems, this paper proposes a semantic and intelligent focused crawler based on BERT semantic vector space model and hybrid algorithm. This method used BERT semantic vector space model to calculate the topic relevance of documents, and used a hybrid algorithm to optimize the weighting factor of unvisited URL priority. The experimental results show that the proposed BSVSM-HA crawler can obtain better evaluation indicators compared with the other three crawlers including Word2vec crawler, ELMO crawler and BSVSM crawler. In conclusion, the semantic and intelligent crawler proposed in this paper makes the semantic similarity between terms more accurate, and improves the topic relevance of the text, and the optimized weighting factor makes the priority evaluation of unvisited URLs more accurate.

A focused crawler automatically retrieves, organizes, and extracts specific topic-related information from the internet for analysis and application. Currently, most focused crawlers assess the relevance of web pages to a given topic through methods such as keyword matching, semantic analysis, and link structures. However, these existing focused crawlers suffer from issues such as misleading directions and reduced accuracy due to the lack of semantic analysis of topic terms, as well as biased computation of topic relevance caused by the absence of effective weighting factors. To solve the above-mentioned problems, this study proposes a semantic and optimized focused crawler based on Semantic Graph and Genetic Algorithm. The proposed crawler eliminates ambiguous terms by constructing a semantic graph, optimizes the weighting factors of topic relevance with asymmetry by using a genetic algorithm, and combines both above two points to predict the priority of each unvisited hyperlink. The experiment results indicate that the proposed SG-GA Crawler improves the evaluation indicators compared with the other three focused crawlers, including VSM Crawler, SSRM Crawler, and SG Crawler. More specifically, the percentage improvement achieved by the proposed method exceeds 19%, 19%, and 13% in terms of three evaluation indicators, including the number of relevant web pages, acquisition rate, and average relevance, respectively. In conclusion, the proposed focused crawler can grab more quantity and higher quality topic-related web pages from the Internet.

The focused crawler grabs continuously web pages related to the given topic according to priorities of unvisited hyperlinks. In many previous studies, the focused crawlers predict priorities of unvisited hyperlinks based on the text similarity models. However, the representation terms of the web page ignore the phenomenon of polysemy, and the topic similarity of the text cannot combine the cosine similarity and the semantic similarity effectively. To address these problems, this paper proposes a focused crawler based on semantic disambiguation vector space model (SDVSM). The SDVSM method combines the semantic disambiguation graph (SDG) and the semantic vector space model (SVSM). The SDG is used to remove the ambiguation terms irrelevant to the given topic from representation terms of retrieved web pages. The SVSM is used to calculate the topic similarity of the text by constructing text and topic semantic vectors based on TF × IDF weights of terms and semantic similarities between terms. The experiment results indicate that the SDVSM method can improve the performance of the focused crawler by comparing different evaluation indicators for four focused crawlers. In conclusion, the proposed method can make the focused crawler grab the higher quality and more quantity web pages related to the given topic from the Internet.