Sentiment Analysis of Acceptance TVET Online Courses on the Skill Academy App from Google Play: Leveraging Text Mining with Comparison Machine Learning Model
Open Access
- 24 September 2026
- journal article
- Published by F1000 Research Ltd in F1000Research
- Vol. 15, 899
Abstract
Background Online Technical and Vocational Education and Training (TVET) has expanded rapidly in Indonesia through platforms such as Skill Academy. User reviews provide valuable insights into user acceptance, but their volume makes manual analysis impractical. Therefore, this study applies text mining to analyze user sentiment and compare eight machine learning models for sentiment classification. Methods The study used 3,000 reviews collected via web scraping using the google-play-scraper library. The data was then anonymized, cleaned, translated into English, and automatically sentiment-labeled using VADER. The validity of the labeling was verified by comparing it with TextBlob and the original star-rating categories using Cohen’s Kappa. Eight classification algorithms Naive Bayes, SVM, Logistic Regression, Random Forest, Decision Tree, KNN, XGBoost, and LightGBM were trained using a TF-IDF pipeline with GridSearchCV and five-fold cross-validation. Results The labeling validation results showed moderate agreement, with a Cohen’s Kappa of 0.500 for TextBlob and 0.533 for the original star rating categories. On the test data, Logistic Regression achieved the best performance with an accuracy of 85.48%, a macro-F1 score of 76.08%, a balanced accuracy of 80.86%, and a macro-AUC of 0.9486, followed by SVM and XGBoost. The sentiment distribution was dominated by positive reviews at 75.31%, followed by negative reviews at 15.57%, and neutral reviews at 9.12%. The SMOTETomek experiment improved balanced accuracy and recall for the minority class but reduced macro-precision. Word cloud and word frequency analysis revealed that the main complaints were related to the app being slow, resource-intensive, and prone to errors, while user appreciation centered on ease of learning and the quality of the material. Conclusions Research shows that logistic regression achieved the best classification performance. While user sentiment was largely positive, improving application stability remains essential for sustaining user acceptance, enhancing learning experiences, and supporting long-term adoption of online TVET platforms.Keywords
Funding Information
- Lembaga Pengelola Dana Pendidikan (IndonesiaEndowmentFundforEducation(LPDP))
This publication has 77 references indexed in Scilit:
- Don’t push the button! Exploring data leakage risks in machine learning and transfer learningArtificial Intelligence Review, 2025
- Sentiment analysis of news: unveiling AI’s role in sustainability and no poverty (SDG1)Discover Sustainability, 2025
- The impact of technology frustration and consumer passion on consumer evaluation shift in case of mobile appsActa Psychologica, 2025
- Aspect-based sentiment analysis for software requirements elicitation using fine-tuned Bidirectional Encoder Representations from Transformers and Explainable Artificial IntelligenceEngineering Applications of Artificial Intelligence, 2025
- Sentiment analysis of movie reviews based on NB approaches using TF–IDF and count vectorizerSocial Network Analysis and Mining, 2024
- Interpreting Black-Box Models: A Review on Explainable Artificial IntelligenceCognitive Computation, 2023
- Does technology assist to continue learning during pandemic? A sentiment analysis and topic modeling on online learning in south asian regionSocial Network Analysis and Mining, 2022
- Using Machine Learning to Predict the Sentiment of Online Reviews: A New Framework for Comparative AnalysisArchives of Computational Methods in Engineering, 2021
- What predicts student satisfaction with MOOCs: A gradient boosting trees supervised machine learning and sentiment analysis approachComputers & Education, 2020
- Text categorization with Support Vector Machines: Learning with many relevant features1998