Future-Ready Skills Across Big Data Ecosystems: Insights from Machine Learning-Driven Human Resource Analytics
Open Access
- 22 May 2025
- journal article
- research article
- Published by MDPI AG in Applied Sciences
- Vol. 15 (11) , 5841
Abstract
This study aims to analyze online job postings using machine learning-based, semantic approaches and to identify the expertise roles and competencies required for big data professions. The methodology of this study employs latent Dirichlet allocation (LDA), a probabilistic topic modeling technique, to reveal hidden semantic structures within a corpus of big data job postings. As a result of our analysis, we have identified seven expertise roles, six proficiency areas, and 32 competencies (knowledge, skills, and abilities) necessary for big data professions. These positions include “developer”, “engineer”, “architect”, “analyst”, “manager”, “administrator”, and “consultant”. The six essential proficiency areas for big data are “big data knowledge”, “developer skills”, “big data analytics”, “cloud services”, “soft skills”, and “technical background”. Furthermore, the top five skills emerged as “big data processing”, “big data tools”, “communication skills”, “remote development”, and “big data architecture”. The findings of our study indicated that the competencies required for big data careers cover a broad spectrum, including technical, analytical, developer, and soft skills. Our findings provide a competency map for big data professions, detailing the roles and skills required. It is anticipated that the findings will assist big data professionals in assessing and enhancing their competencies, businesses in meeting their big data labor force needs, and academies in customizing their big data training programs to meet industry requirements.Keywords
This publication has 42 references indexed in Scilit:
- Student engagement research trends of past 10 years: A machine learning-based analysis of 42,000 research articlesEducation and Information Technologies, 2023
- Integrating big data and cloud computing topics into the computing curricula: A modular approachJournal of Parallel and Distributed Computing, 2021
- What skills do IT companies look for in new developers? A study with Stack Overflow jobsInformation and Software Technology, 2021
- How Big Data Creates New Job Opportunities: Skill Profiles of Emerging Professional Roles2019
- Human resources for Big Data professions: A systematic classification of job roles and required skill setsInformation Processing & Management, 2018
- Comparing Business Intelligence and Big Data SkillsBusiness & Information Systems Engineering, 2014
- Data-intensive applications, challenges, techniques and technologies: A survey on Big DataInformation Sciences, 2014
- The impact of preprocessing on text classificationInformation Processing & Management, 2014
- Balancing software engineering education and industrial needsJournal of Systems and Software, 2012
- Data Mining2011