Learning From High-Cardinality Categorical Features in Deep Neural Networks
- 23 June 2022
- journal article
- research article
- Published by Canakkale Onsekiz Mart University in Journal of Advanced Research in Natural and Applied Sciences
- Vol. 8 (2) , 222-236
Abstract
Some machine learning algorithms expect the input variables and the output variables to be numeric. Therefore, in an early stage of modelling, feature engineering is required when categorical variables present in the dataset. As a result, we must encode those attributes into an appropriate feature vector. However, categorical variables having more than 100 unique values are considered to be high-cardinality and there exists no straightforward methods to handle them. Besides, the majority of the work on categorical variable encoding in the literature assumes that the categories is limited, known beforehand, and made up of mutually-exclusive elements, inde-pendently from the data, which is not necessarily true for real-world applications. Feature engineering typically practices to tackle the high cardinality issues with data-cleaning techniques which they are time-consuming and often needs human intervention and domain expertise which are major costs in data science projects The most common methods of transform categorical variables is one-hot encoding and target encoding. To address the issue of encoding categorical variables in environments with a high cardinality, we also seek a general-purpose approach for statistical analysis of categorical entries that is capable of handling a very large number of catego-ries, while avoiding computational and statistical difficulties. Our proposed approach is low dimensional; thus, it is very efficient in processing time and memory, it can be computed in an online learning setting. Even though for this paper, we opt to utilize it in the input layer, dictionaries are typically architecture-independent and may be moved between different architectures or layers.Keywords
This publication has 17 references indexed in Scilit:
- Deep learningNature, 2015
- Including high-cardinality attributes in predictive models: A case study in churn prediction in the energy sectorDecision Support Systems, 2015
- Glove: Global Vectors for Word Representation2014
- A few useful things to know about machine learningCommunications of the ACM, 2012
- Practical Recommendations for Gradient-Based Training of Deep Architectures2012
- Distribution-based aggregation for relational learning with identifier attributesMachine Learning, 2006
- A preprocessing scheme for high-cardinality categorical attributes in classification and prediction problemsACM SIGKDD Explorations Newsletter, 2001
- Statistical Classification Methods in Consumer Credit Scoring: A ReviewJournal of the Royal Statistical Society Series A: Statistics in Society, 1997
- Learning representations by back-propagating errorsNature, 1986
- Use of Dummy Variables in Regression EquationsJournal of the American Statistical Association, 1957