Yi Wang

Article Yu Wei ^1,2, Yi Wang ^1,*, Baofeng Zhu ¹, Chi Lin ¹, Dan Wu ¹, Xinwei Xue ¹, and Ruili Wang ^{3,4 1} School of Software Technology, Dalian University of Technology, Dalian 116620, China ² Harbin Boiler Co., Ltd, Harbin, 150000, China ³ School of Mathematical and Computational Sciences, Massey University, Auckland 0632, New Zealand ⁴ School of Computer Science, University of Nottingham Ningbo China, Ningbo 315100, China ^* Correspondence: dlutwangyi@dlut.edu.cn Received: 27 June 2023 Accepted: 25 April 2024 Published: 25 December 2024 Abstract: Underwater detection poses significant challenges due to the unique characteristics of the underwater environment, such as light attenuation, scattering, water turbidity, and the presence of small or camouflaged objects. To gain a clearer understanding of these challenges, we first review two common detection tasks: object detection (OD) and salient object detection (SOD). Next, we examine the difficulties of adapting existing OD and SOD techniques to underwater settings. Additionally, we introduce a new Underwater Object Multitask (UOMT) dataset, complete with benchmarks. This survey, along with the proposed dataset, aims to provide valuable resources to researchers and practitioners to develop more effective techniques to address the challenges of underwater detection. The UOMT dataset and benchmarks are available at https://github.com/yiwangtz/UOMT. Marine resources are extremely valuable to humans. Underwater detection has broad applications in many areas, such as oceanography, marine navigation, and fish farming [1]. However, the complexities of underwater environments and ecosystems present formidable obstacles to effective underwater detection. For example, underwater environments exhibit considerable variability in lighting conditions, depending on factors such as depth and quality [2−3]. Intricate and multifaceted shapes characterize diverse organisms. Certain organisms also evolved a camouflage mechanism. Furthermore, these organisms often face obfuscation or disruption due to the integration of sand and gravel [4]. Therefore, it calls for the integration of related technologies in computer vision and artificial intelligence to address these challenges in underwater detection. In this context, Object Detection (OD) [5−7] and Salient Object Detection (SOD) [8−10] are crucial to understanding and analyzing underwater scenes. OD is designed to detect and classify objects accurately while providing their precise spatial locations. SOD focuses on the localization and segmentation of salient objects that conform to the human visual system (HVS) [11]. Several studies and surveys have been conducted in the fields of underwater OD and SOD, highlighting the significance of these tasks in underwater image analysis [12−19]. Our work begins with a brief overview of recent breakthroughs and advances within the OD and SOD fields. Different from existing surveys on these two topics [5−10, 12−14, 20−21], we classify OD and SOD models into two primary categories: (i) Convolutional Neural Networks (CNNs) [22] based approaches and (ii) Transformer [23] based approaches. Furthermore, we delve into the unique challenges posed by underwater detection tasks, as well as the strengths and weaknesses of these approaches. A comprehensive dataset entitled Underwater Object Multitask (UMOT) is also proposed, which contains various underwater scenes with more than 7K instances encompassing three organism types. UOMT provides COCO dataset [24] format labels for OD and binary segmentation masks for SOD, as illustrated in Figure 1. Evaluations of state-of-the-art OD and SOD models are conducted on the UMOT dataset. Figure 1. Examples in the proposed Underwater Object Multitask (UOMT) dataset: (a) Original images; (b) Annotations for Object Detection; (c) Annotations for Salient Object Detection. The rest of this article is presented accordingly. Section II briefly reviews OD methods. Section III briefly reviews SOD methods. Section IV explains underwater OD and SOD challenges. Section V describes the evaluation metrics for OD and SOD. Section VI describes the proposed UOMT dataset. Section VII presents benchmarks for OD and SOD for the UOMT dataset through quantitative and qualitative experiments. Section VIII discusses the future development of underwater OD and SOD. Section IX summarizes the main points of this work. Object detection (OD) endeavors to accurately detect and classify objects, denoting their positions by rectangular boxes [25]. Object detection encompasses various applications, including face detection, instance segmentation, autonomous driving, surveillance systems, and sports analytics [5]. In recent years, many object detection surveys have been conducted. In 2019, Jiao et al. [5] conducted a comparative analysis of different deep learning-based OD methods. It also introduced some commonly used public datasets, analyzed their characteristics, and highlighted their strengths and weaknesses. In 2020, Wu et al. [6] concluded techniques useful in OD, such as attention-based models [26−28], end-to-end models [29−30], depth separable convolutions [31−32], etc. This survey also presented challenges and requirements for practical applications, including detecting small objects, object tracking, real-time performance, and other related topics. Li et al. [7] extensively explored various aspects of OD in images captured by optical sensors or satellites. The survey also proposed an optical remote sensing image dataset with benchmarks. Padilla et al. [12] provided a detailed and comprehensive exposition of the commonly used performance evaluation metrics in object detection methods. This survey provided details on the principles, advantages, and disadvantages of different indicators and their applicability to different application scenarios. In 2022, Cheng et al. [13] provided a comprehensive study and summary of the problem of small object detection. In 2023, Zou et al. [14] systematically reviewed the development of OD over the past 20 years, offering valuable insights into the field of SOD. Our review of OD categorizes its methods into CNN-based and Transformer-based models. Table 1 lists the models discussed in this section. Convolutional Neural Networks (CNNs) [22] exhibit profound influences on object detection [14] by using mechanisms such as ReLU activation, Dropout, Anchor, and GPU acceleration. Among them, the anchor mechanism [27] is a key technique in object detection. We elaborate on OD models based on CNNs by classifying them into two groups: anchor-based detectors and anchor-free detectors, in the following. 1) Anchor-Based Approaches: Anchor [5] mechanism was proposed in 2014. It first generates a series of predefined bounding boxes (anchors) at different locations, and then each anchor is matched to a real object during detection. In the following years, many OD models were proposed using anchors. These methods can be further categorized into two-stage and one-stage detection approaches. Two-Stage Detectors: Girshick et al. [33] proposed R-CNN in 2014, which revolutionized OD by introducing anchors. R-CNN converts OD into a two-step process: candidate region extraction and classification. R-CNN exhibits remarkable progress compared to conventional OD methods. He et al. [34] introduced SPP-Net, a significant OD breakthrough. SPP-Net facilitates the conversion of an input image with varying dimensions into a feature map with a consistent size, enabling holistic image comprehension. Nonetheless, it faces limitations as it cannot be trained end-to-end. In 2015, Girshick et al. introduced Fast R-CNN [26] and Faster R-CNN [27]. Fast R-CNN uses the entire image as input, avoiding redundant feature computations, and employs a multitask loss function to integrate classification and regression tasks, improving OD efficiency and accuracy. However, it still relies on a region proposal algorithm to generate potential regions, which has speed limitations and compromises the accuracy of the selected regions. The Faster R-CNN, therefore, generates candidate regions using a Region Proposal Network (RPN) instead of a selective search algorithm. Dai et al. [35] proposed the R-FCN, which features a fully convolutional network instead of the region of interest (ROI) pooling operation, avoiding the traditional ROI operation, and can be trained end-to-end. In 2017, Mask-RCNN [28] improved the previous versions with increased detection speed and accuracy. One-Stage Detectors: In 2015, Redmon et al. [36] proposed YOLOv1, a one-stage detector with faster speed and better real-time performance than two-stage detectors. This method divides an image into several grids, each predicting a bounding box and a category probability. However, it has some disadvantages, such as low detection accuracy, poor detection of small objects, and easy occlusion of dense objects. In 2016, Liu et al. [32] proposed the SSD, which features only one forward propagation to complete the object detection process. In 2017, YOLOv2 [37] was introduced, which improved YOLOv1 with increased detection speed and accuracy. Various excellent methods emerged after 2017. RefineDet [38] and YOLOv3 [39] were proposed in 2018. RefineDet is the first real-time method to achieve detection accuracy greater than 80% \( mAP \) on PASCAL VOC 2007 [60]. In YOLOv3, performance was further enhanced by a more efficient backbone network, multiple anchors, and spatial pyramid pooling. In 2019, Tan et al. [31] introduced EfficientDet, incorporating a revolutionary integrated scaling technique to improve detection accuracy and computational efficiency. In 2020, Alexey et al. [40] proposed YOLOv4, and Cai et al. [41] proposed YOLOv5. In YOLOv4, mosaic data augmentation, improved anchor-free detection head, and a new loss function were introduced. YOLOv5 features hyperparameter optimization, integrated experiment tracking, and automatic export to popular export formats. In 2021, YOLOF [42] was proposed, which used a Dilated Encoder and Uniform Matching that improved detection speed and accuracy without FPN [61]. PP-YOLOv2 [43] is an efficient real-time object detection method that uses a compact foundational architecture. It introduced an adaptive weighted loss function and adaptive label smoothing techniques to better handle objects of different sizes and difficulties. PP-YOLOv2 significantly improved detection performance while maintaining faster speed and practicality. In 2022, YOLOv7 [45] was introduced using the Extended Efficient Layer Aggregation Network (E-ELAN). YOLOv7 also added additional tasks, such as pose estimation, and surpassed the previous state-of-the-art for real-time applications. DAB-DETR [51] utilizes coil coordinates in Transformer decoders and performs soft ROI pooling. Anchor-based object detection approaches have certain limitations that can impact training and inference. Objects with small sizes and aspect ratios may be missed or falsely detected due to their size and aspect ratio limitations. Additionally, multiple detection frames may overlap due to anchor overlap, increasing follow-up difficulties and computational effort. 2) Anchor-Free Approaches: In 2018, Zhou et al. [5] introduced the concept of Anchor-Free detection, eliminating the need for predefined bounding boxes. This makes the model simpler, faster to train and infer, and more adaptable to object shapes and sizes. Law and Deng et al. [62] proposed CornerNet in 2018, a CNN-based Anchor-Free detector, that detects the object bounding box as two key points, the upper left and lower right. CornerNet also proposed an effective corner pooling operation that captures boundary information. In 2019, Fei et al. [61] proposed NAS-FPN, which introduces a Neural Architecture Search (NAS) technique that automatically searches the structure of a neural network to obtain a better feature pyramid network. It also combines the advantages of NAS and FPN to automatically search for the optimal pyramid structure of characteristics. That year, it achieved first place in the COCO Object Detection Challenge [24]. In 2020, CircleNet [47] was developed, which treats an object as a circular region consisting of points. It detects the object by predicting the center and radius of the circle. CircleNet detects spherical biomedical objects accurately. In 2021, YOLOX [48] was proposed as an anchor-free version of YOLO, with a simpler design but better performance. YOLOv6 [49] was developed in 2022 and used in many autonomous delivery robots. YOLOv8 [29] improves YOLOv5 by separating classification and detection heads and using Distribution Focal Loss. Anchor-free methods, however, require more training data and longer training times to achieve the same accuracy as anchor-based detectors, and cluttered images may further hinder Anchor-Free Detectors. In 2020, DETR [55] (Detection Transformer) first applied the Transformer [23] to object detection. Unlike traditional object detection methods, DETR does not need prior frames or anchor points. Instead, it predicts the positions and classes of objects directly in the image. Specifically, DETR transforms the object detection problem into an ensemble matching problem. YOLOR [57] leverages Transformer to boost feature representation without increasing computation. YOLOR has a unified network that integrates explicit and implicit knowledge to learn a unified representation capable of performing multiple tasks. YOLOS [56] utilizes a Swin Transformer and Focal Loss that improve detection without convolutional layers. In 2022, Detic [58] proposed using the image classification dataset to train the classification head of the target detector. A strong end-to-end object detector, DINO [52] was developed based on DN-DETR [59], DAB-DETR [51], and Deformable-DETR [44]. DINO improves over previous DETR-like models in performance and efficiency by using a contrastive way to denoize training, a look forward twice scheme for box prediction, and a mixed query selection method for anchor initialization. In 2023, a unified object detection and segmentation framework, Mask-DINO [53], was developed. The DINO Mask extends DINO by adding a mask prediction branch that supports all image segmentation tasks (instance, panorama, and semantic). In the same year, another improved DINO model, Grounding DINO [54], was proposed. Grounding DINO can detect specified targets based on text descriptions. Salient object detection (SOD) [87] is a computer vision task that aims to identify visually distinctive objects or regions within an image. This task is crucial in various applications, including image editing, visual tracking, image retrieval, and scene understanding [9−10]. Several surveys have focused on detecting salient objects in the last few years. For example, Borji et al. [8] reviewed SOD methods using CNNs in 2019. In 2020, Kumar et al. [9] investigated weakly supervised/pseudo-supervised and adversarial training learning approaches. In 2021, Zhour et al. [88] reviewed RGB-D-based SOD models, as well as related benchmark datasets, using traditional and deep learning methods. In 2022, Fu et al. [20] provided a review and benchmarks for light-field SOD. In 2022, Zhou et al. [21] summarized the latest SOD models and pointed out that different implementation details may affect performance. In this section, our review focuses on SOD using CNNs and Transformers. Table 2 shows the models presented in this section. 2014 was the year of pioneering deep learning applied in SOD [89]. In many SOD models, CNNs are used to extract high-level features from images and combine local and global contextual information. These methods capture semantic information and the salient features of images in complex scenes more effectively than traditional methods. In 2015, FCN [90] networks revolutionized the field using a semantic segmentation architecture at the pixel level. From then on, many SOD models have been developed based on FCNs. Liu et al. [91] used multiscale deep features to express saliency comparisons and prior knowledge. He et al. [63] developed an exemplar-driven top-down saliency detection model that employed a deep association network to learn similarities between exemplars and images. Lee et al. [64] used encoding distance maps, high-level features, and an encoder-decoder structure to generate saliency maps. Wang et al. [65] proposed a weakly supervised SOD model, which reduces the costs of manual annotation. Zhang et al. [66] proposed a progressive attention-guided mechanism to improve the quality of the prediction. Many excellent SOD algorithms have been proposed since 2019. Most of them employ VGG [92] or ResNet [93] as the backbone network, such as CPD [67], PoolNet [68], BASNet [69], EGNet [70], etc. CPD [67] employs a bidirectional feature pyramid network and a feedback optimization module to generate high-quality saliency maps. PoolNet [68] proposed a Global Guidance Module (GGM) and a Feature Aggregation Module (FAM) based on pooling techniques that combine multiple layers of features at different scales. Contextual information about the image and edge information are also considered. BASNet [69] is a boundary-aware method that uses a prediction-optimization architecture and a hybrid loss function to improve the precision of boundary delineation on saliency maps. EGNet [70] extracted the features of the salient objects through progressive fusion, integrating the local edge information with the global location information and utilizing complementary features to detect objects at various resolutions. The representative models proposed in 2020 were LDF [71], MINet [72], and UCNet [73]. LDF [71] separated a GT (Ground-Truth) map into a body map and a detailed map. It then performs feature learning on the inner and boundary regions in a two-branch decoder, respectively. UC-Net [73] improved the generalization and robustness of the model by introducing hidden variables to represent the uncertainty of the input data. SAC [74] adaptively propagates, and aggregates image context features with different attenuation factors through a spatially attenuated context (SAC) module and attention mechanisms. The representative models proposed in 2021 were KRN [75], HQSOD [76], and DCN [77]. KRN [75] introduced a progressive approach to replicate the human visual system restoration process, with the coarse localization module and the fine segmentation module developed. HQSOD [76] extended the SOD for high-resolution images by a low-resolution saliency classification network (LRSCN) to capture semantics at low resolution and a high-resolution refinement network (HRRN) to refine the salient values of pixels in uncertain regions. DCN [77] proposed a multitasking network to predict salient maps, edges, and skeleton maps simultaneously at first. Then it designed cross-task aggregation and cross-layer aggregation modules to integrate multi-level and multi-tasking features in the final results. In 2022, EDN [78] effectively exploited multiscale learning while improving collaboration between high- and low-level features. TNet [79] used thermal infrared images to get effective object localization and integrity information for RGB decoded features by controlling the interaction between RGB images and thermal images. Using attention-guided tracing modules, Lee et al. [80] proposed a model called TRACER, which eliminates multi-decoder structures and minimizes the use of learning parameters. In 2023, Zhuge et al. [86] proposed ICON-R, which facilitates the extraction of multiscale feature maps through the integrity...