Dialectal Detective: Leveraging Feature Selection Techniques to Unearth Moroccan Dialect Arabic
Preprint
- 5 February 2024
- preprint
- Published by Springer Nature in Research Square
Abstract
In the evolving context of automated language processing, unraveling the complex fabric of Moroccan Arabic (Darija) in a multilingual environment is a major challenge. This study embarks on the arduous task of detecting the nuances of Moroccan Arabic within a binary dataset that consists of both Standard Arabic and Darija expressions. Using a comprehensive methodology, we interweave a sophisticated set of feature selection techniques, including well-known extraction techniques in natural language processing (NLP) such as TF-idf, CBOW, and Word2Vec. By leveraging the capabilities of machine learning techniques via LASSO decision and regression trees, we navigate the labyrinth of linguistic diversity, relying on semantic methods that consist of using mostly advanced encoders to deepen our understanding of the distinctive linguistic fabric. We also look at static methods of feature selection, such as ANOVA, Pearson correlation coefficient, and mutual information, in order to add strata of analysis. Finally, by emphasizing the paramount importance of dimensionality reduction through principal component analysis (PCA) and singular value decomposition (SVD), our methodology not only preserves the essential structures in the high-dimensional linguistic space, but also strives to contribute significantly to the accurate detection of Moroccan Arabic dialects. The stability of our results is achieved with the XGBOOST algorithm using classical extraction methods and SVD, with a reasonable execution time. This research aspires not only to unveil the specific subtleties of Arabic dialects, but also to open up new horizons in the field of natural language processing within diverse and multilingual societies.Keywords
All Related Versions
This publication has 12 references indexed in Scilit:
- Sentiment analysis in Arabic: A review of the literatureAin Shams Engineering Journal, 2018
- Detecting Novel Associations in Large Data SetsScience, 2011
- Principal component analysisWIREs Computational Statistics, 2010
- Analysis of Messy Data Volume 12009
- The Elements of Statistical Learning2009
- Light Stemming for Arabic Information Retrieval2007
- Machine learning: a review of classification and combining techniquesArtificial Intelligence Review, 2006
- Regression Shrinkage and Selection Via the LassoJournal of the Royal Statistical Society Series B: Statistical Methodology, 1996
- A vector space model for automatic indexingCommunications of the ACM, 1975
- A STATISTICAL INTERPRETATION OF TERM SPECIFICITY AND ITS APPLICATION IN RETRIEVALJournal of Documentation, 1972