On entropy-based term weighting schemes for text categorization

Tao WANG, Yi CAI*, Ho-fung LEUNG, Raymond Y. K. LAU, Haoran XIE, Qing LI

*Corresponding author for this work

Research output: Journal PublicationsJournal Article (refereed)peer-review

5 Citations (Scopus)


In text categorization, Vector Space Model (VSM) has been widely used for representing documents, in which a document is represented by a vector of terms. Since different terms contribute to a document’s semantics in various degrees, a number of term weighting schemes have been proposed for VSM to improve text categorization performance. Much evidence shows that the performance of a term weighting scheme often varies across different text categorization tasks, while the mechanism underlying variability in a scheme’s performance remains unclear. Moreover, existing schemes often weight a term with respect to a category locally, without considering the global distribution of a term’s occurrences across all categories in a corpus. In this paper, we first systematically examine pros and cons of existing term weighting schemes in text categorization and explore the reasons why some schemes with sound theoretical bases, such as chi-square test and information gain, perform poorly in empirical evaluations. By measuring the concentration that a term distributes across all categories in a corpus, we then propose a series of entropy-based term weighting schemes to measure the distinguishing power of a term in text categorization. Through extensive experiments on five different datasets, the proposed term weighting schemes consistently outperform the state-of-the-art schemes. Moreover, our findings shed new light on how to choose and develop an effective term weighting scheme for a specific text categorization task.
Original languageEnglish
Pages (from-to)2313-2346
Number of pages34
JournalKnowledge and Information Systems
Issue number9
Early online date7 Jul 2021
Publication statusPublished - Sept 2021

Bibliographical note

Funding Information:
This work was supported by the Fundamental Research Funds for the Central Universities, SCUT (No. D2200150, D2201300), the Science and Technology Programs of Guangzhou (Nos. 201704030076, 201802010027, 201902010046), National Natural Science Foundation of China (No. 62076100) and National Key Research and Development Program of China (Standard knowledge graph for epidemic prevention and production recovering intelligent service platform and its applications).

Publisher Copyright:
© 2021, The Author(s), under exclusive licence to Springer-Verlag London Ltd., part of Springer Nature.


  • Entropy
  • Normalization
  • Smoothing
  • Term weighting
  • Text categorization


Dive into the research topics of 'On entropy-based term weighting schemes for text categorization'. Together they form a unique fingerprint.

Cite this