Text Classification Using Novel Term Weighting Scheme-Based Improved TF-IDF for Internet Media Reports
Open Access
- 5 March 2021
- journal article
- research article
- Published by Wiley in Mathematical Problems in Engineering
- Vol. 2021, 1-30
Abstract
With the rapid development of the internet technology, a large amount of internet text data can be obtained. The text classification (TC) technology plays a very important role in processing massive text data, but the accuracy of classification is directly affected by the performance of term weighting in TC. Due to the original design of information retrieval (IR), term frequency-inverse document frequency (TF-IDF) is not effective enough for TC, especially for processing text data with unbalanced distributions in internet media reports. Therefore, the variance between the DF value of a particular term and the average of all DFs , namely, the document frequency variance (ADF), is proposed to enhance the ability in processing text data with unbalanced distribution. Then, the normal TF-IDF is modified by the proposed ADF for processing unbalanced text collection in four different ways, namely, TF-IADF, TF-IADF+, TF-IADFnorm, and TF-IADF+norm. As a result, an effective model can be established for the TC task of internet media reports. A series of simulations have been carried out to evaluate the performance of the proposed methods. Compared with TF-IDF on state-of-the-art classification algorithms, the effectiveness and feasibility of the proposed methods are confirmed by simulation results.Keywords
Funding Information
- National Natural Science Foundation of China (61973024, 61703027, JD1914, XK1802-4)
- Fundamental Research Funds for the Central Universities (61973024, 61703027, JD1914, XK1802-4)
This publication has 33 references indexed in Scilit:
- Ensemble of keyword extraction methods and classifiers in text classificationExpert Systems with Applications, 2016
- An Improved Focused Crawler: Using Web Page Classification and Link Priority EvaluationMathematical Problems in Engineering, 2016
- Entropy Optimized Feature-Based Bag-of-Words Representation for Information RetrievalIEEE Transactions on Knowledge and Data Engineering, 2016
- Combining supervised term-weighting metrics for SVM text classification with extended term representationKnowledge and Information Systems, 2016
- Hybridized term-weighting method for Dark Web classificationNeurocomputing, 2016
- Classification of Chinese Texts Based on Recognition of Semantic TopicsCognitive Computation, 2015
- ForesTexter: An efficient random forest algorithm for imbalanced text categorizationKnowledge-Based Systems, 2014
- Hybrid decision tree and naïve Bayes classifiers for multi-class classification tasksExpert Systems with Applications, 2014
- A Survey of Text Classification Algorithms2012
- Supervised Term Weighting for Automated Text Categorization2004