A novel hybrid machine learning model for credit card fraud detection using anomaly detection and optimised classification
Authors:
- Maram AlamriMAMaram Alamri
- Mourad YkhlefMYMourad Ykhlef
Open Access
- 17 April 2026
- journal article
- Published by PeerJ in PeerJ Computer Science
- Vol. 12, e3748
Abstract
Credit card fraud poses a significant challenge to financial institutions, necessitating models that can detect rare and complex fraudulent patterns in highly imbalanced datasets. This study introduced a hybrid approach that integrated one-class support vector machine (OCSVM) for anomaly removal, random forest (RF) for classification and genetic algorithm (GA) for hyperparameter optimisation. The method was evaluated on the publicly available PaySim dataset, which simulates real-world financial transactions. Model performance was assessed using F1-score, accuracy, precision, recall, average precision-recall curve and area under the receiver operating characteristic curve. The proposed model achieved superior results compared to baseline methods, with an F1-score of 90.74% and accuracy of 99.95%, along with notable improvements in precision and recall. These findings demonstrate that combining anomaly detection with optimised classification significantly enhances fraud detection performance. The proposed framework can serve as a robust tool for improving fraud prevention in financial applications.Keywords
Funding Information
- Ongoing Research Funding Program, King Saud University, Riyadh, Saudi Arabia (ORFFT-2025)
This publication has 51 references indexed in Scilit:
- Optimizing Hyperparameters for Fraud Detection: A Comparative Analysis of Machine Learning Algorithms2024
- Explainable Fraud Detection with Deep Symbolic Classification2024
- Enhanced particle swarm optimization-based hyperparameter optimized stacked autoencoder for credit card fraud detectionInternational Journal of Data Science and Analytics, 2024
- Credit card fraud detection using ensemble data mining methodsMultimedia Tools and Applications, 2023
- Fraud Detection in Mobile Payment Systems using an XGBoost-based FrameworkInformation Systems Frontiers, 2022
- Performance analysis of metaheuristics based hyperparameters optimization for fraud transactions detectionEvolutionary Intelligence, 2022
- Machine Learning-Based Algorithmic Approach for Enhanced Anomaly Detection in Financial Transactions2022
- Deep Neural Networks Hyperparameter Optimization Using Particle Swarm Optimization for Detecting Frauds Transactions2021
- Hyperparameter Optimization Using Genetic Algorithms to Detect Frauds Transactions2021
- Hybrid Method in Identifying the Fraud Detection in the Credit Card2020