Driver distraction is a major cause of road accidents globally. To address this, there has been significant research into using machine learning to monitor driver behavior. However, many existing reviews are limited in scope. They often focus only on specific algorithms or overlook the critical role of different data types. This paper fills that gap by providing an in-depth systematic review of both the data and the algorithms in machine learning-based driver distraction detection. By providing a clear overview of the current state of the field, this paper aims to improve the understanding of driver distraction detection and guide future research. To ensure a rigorous and unbiased selection process, the PRISMA methodology was used to select studies. To provide a fine-grained view, both topics (data and algorithms) are systematically categorized into specific subdomains. For each category, key concepts are explained, significant contributions are summarized, and their respective strengths and weaknesses are critically analyzed. In addition, this article provides a comparative and critical analysis of widely used driving-related datasets. Finally, challenges, research gaps, and possible directions for future research are analyzed and discussed. The main results of this review show that deep learning methods outperform older models, although they require more computational power. Issues such as model interpretability and monitoring systems with driving automation have received less attention. Also, systems that use multimodal data (end to end multimodal systems) have great potential for the future.