Humza Naveed

Deep learning has evolved from early perceptron models to modern foundation models through a series of breakthroughs that addressed fundamental limitations in representation learning, optimization, computational efficiency, and scalability. While numerous surveys review specific architectures, learning paradigms, or application domains, few examine the field from a historical and evolutionary perspective. This paper traces the evolution of deep learning by identifying the key discoveries that shaped its development, the bottlenecks that motivated each advance, and the innovations that enabled subsequent progress. Without an understanding of the historical motivations behind major breakthroughs, researchers may inadvertently revisit previously resolved challenges or overlook design principles that have shaped modern deep learning architectures. Understanding why particular architectures emerged and how they addressed existing limitations can help researchers avoid revisiting previously solved problems and make more informed design decisions for future models. Beginning with the origins of artificial neural networks and the introduction of backpropagation, we examine the resurgence of deep learning driven by large-scale datasets, GPU computing, and architectural innovations, followed by developments in convolutional, recurrent, attention-based, graph, generative, and foundation models. Rather than providing an exhaustive review, this paper focuses on the seminal methods that established new research directions and highlights representative applications in which these advances have been successfully adopted. By connecting major breakthroughs with the challenges they addressed, this survey equips new researchers with a principled understanding of deep learning’s evolution and a foundation for developing novel approaches to emerging problems.

Large Language Models (LLMs) have recently demonstrated remarkable capabilities in natural language processing tasks and beyond. This success of LLMs has led to a large influx of research contributions in this direction. These works encompass diverse topics such as architectural innovations, better training strategies, context length improvements, fine-tuning, multimodal LLMs, robotics, datasets, benchmarking, efficiency, and more. With the rapid development of techniques and regular breakthroughs in LLM research, it has become considerably challenging to perceive the bigger picture of the advances in this direction. Considering the rapidly emerging plethora of literature on LLMs, it is imperative that the research community is able to benefit from a concise yet comprehensive overview of the recent developments in this field. This article provides an overview of the literature on a broad range of LLM-related concepts. Our self-contained comprehensive overview of LLMs discusses relevant background concepts along with covering the advanced topics at the frontier of research in LLMs. This review article is intended to provide not only a systematic survey but also a quick, comprehensive reference for the researchers and practitioners to draw insights from extensive, informative summaries of the existing works to advance the LLM research.

Neural networks are prone to overfitting and memorizing data patterns. To avoid over-fitting and enhance their generalization and performance, various methods have been suggested in the literature, including dropout, regularization, label smoothing, etc. One such method is augmentation which introduces different types of corruption in the data to prevent the model from overfitting and to memorize patterns present in the data. A sub-area of data augmentation is image mixing and deleting. This specific type of augmentation either deletes image regions or mixes two images to hide or make particular characteristics of images confusing for the network, forcing it to emphasize the overall structure of the object in an image. Models trained with this approach have proven to perform and generalize well compared to those trained without image mixing or deleting. An added benefit that comes with this method of training is robustness against image corruption. Due to its low computational cost and recent success, researchers have proposed many image mixing and deleting techniques. We furnish an in-depth survey of image mixing and deleting techniques and provide categorization via their most distinguishing features. We initiate our discussion with some fundamental relevant concepts. Next, we present essentials, such as each category's strengths and limitations, describing their working mechanism, basic formulations, and applications. We also discuss the general challenges and recommend possible future research directions for image mixing and deleting data augmentation techniques. Datasets and codes for evaluation are publicly available here.