Current Organizations / Institutes

The organizations/institutes from the scholar's latest article


Rui Gao has a total of 11 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    Pseudo-LiDAR With Two-Dimensional Instance for Monocular Three-Dimensional Object Tracking

    by Rui Gao, Junoh Kim, Phuong Minh Chu, Kyungeun Cho

    Establishing a framework capable of performing 3D object tracking is crucial for various applications, such as autonomous driving and robot navigation. Monocular cameras offer economic and flexible advantages over LiDAR sensors; however, monocular camera-based 3D object detection is fraught with uncertainty owing to difficulty in depth estimation, visual occlusion, and ambiguous object appearance. This uncertainty poses additional challenges to object tracking as well, making performing stable multiple-object tracking on monocular 3D detection results more complex and difficult. To address these challenges in monocular 3D multiple-object detection and tracking, we propose the innovative framework pseudo-LiDAR-MOT, which accurately infers complete 3D bounding box information from 2D image sequences captured by a monocular camera and efficiently correlates and tracks moving objects in the temporal dimension. This framework increases the object detection accuracy and tracking stability of objects in dynamic environments via the following three methods. Monocular depth prediction and 2D object detection networks are leveraged to estimate pixelwise depth and object boundaries. Subsequently, a pseudo-LiDAR approach is employed to generate a 3D point cloud for each object, where 3D point clouds are transformed into 2D representations. Furthermore, a novel 3D detection network based on image-based detection is introduced, simplifying the complexity of conventional 3D point-cloud-based detection while increasing the accuracy of monocular image-based 3D object detection. A specialized association network is designed to enhance object trajectory tracking across frames, thereby improving 3D tracking performance. The effectiveness of our approach was validated through extensive experiments conducted on the KITTI dataset. Our proposed framework exhibits excellent performance in 3D object detection and tracking, outperforming conventional methods. This study introduces innovative ideas for 3D object detection and tracking and establishes a basis for advancing autonomous driving systems.


  • Open Access

    2D Instance-Guided Pseudo-LiDAR Point Cloud for Monocular 3D Object Detection

    by Rui Gao, Junoh Kim, Kyungeun Cho

    Monocular three-dimensional (3D) scene understanding tasks, e.g., object size angle and 3D position, estimation are challenging to perform. More successful current methods usually require data from 3D sensors, which limits their performance if only monocular images are used because of the lack of distance information. In recent years, 3D object detection methods based on pseudo-LiDAR have effectively improved the accuracy of 3D prediction. However, most pseudo-LiDAR methods directly use LiDAR-based 3D detection networks, which also limits their performance. In this paper, a new monocular 3d object detection framework is proposed. By redesigning the representation of pseudo-LiDAR, a 2d detection mask channel is introduced as a guide layer to guide 3D object detection. A new 3D object detection network is designed for the newly designed multi-channel data. The pillar feature encoding module is optimized by using 2d detection mask and height histogram, so that it can more effectively convert point clouds into pillar features by using 2d detection mask and height characteristics of different objects. A new feature extraction network was designed using the transformer module to extract pillar features. The transformer head was optimized using three independent heads to categorize the position, properties and specific parameters of the object. An evaluation on the challenging KITTI dataset demonstrates that the proposed method significantly improves the performance of state-of-the-art monocular methods.


  • Open Access

    Switchable-Encoder-Based Self-Supervised Learning Framework for Monocular Depth and Pose Estimation

    by Junoh Kim, Rui Gao, Jisun Park, Jinsoo Yoon, Kyungeun Cho

    Monocular depth prediction research is essential for expanding meaning from 2D to 3D. Recent studies have focused on the application of a newly proposed encoder; however, the development within the self-supervised learning framework remains unexplored, an aspect critical for advancing foundational models of 3D semantic interpretation. Addressing the dynamic nature of encoder-based research, especially in performance evaluations for feature extraction and pre-trained models, this research proposes the switchable encoder learning framework (SELF). SELF enhances versatility by enabling the seamless integration of diverse encoders in a self-supervised learning context for depth prediction. This integration is realized through the direct transfer of feature information from the encoder and by standardizing the input structure of the decoder to accommodate various encoder architectures. Furthermore, the framework is extended and incorporated into an adaptable decoder for depth prediction and camera pose learning, employing standard loss functions. Comparative experiments with previous frameworks using the same encoder reveal that SELF achieves a 7% reduction in parameters while enhancing performance. Remarkably, substituting newly proposed algorithms in place of an encoder improves the outcomes as well as significantly decreases the number of parameters by 23%. The experimental findings highlight the ability of SELF to broaden depth factors, such as depth consistency. This framework facilitates the objective selection of algorithms as a backbone for extended research in monocular depth prediction.


  • Open Access

    CAGNet: A Multi-Scale Convolutional Attention Method for Glass Detection Based on Transformer

    by Xiaohang Hu, Rui Gao, Seungjun Yang, Kyungeun Cho

    Glass plays a vital role in several fields, making its accurate detection crucial. Proper detection prevents misjudgments, reduces noise from reflections, and ensures optimal performance in other computer vision tasks. However, the prevalent usage of glass in daily applications poses unique challenges for computer vision. This study introduces a novel convolutional attention glass segmentation network (CAGNet) predicated on a transformer architecture customized for image glass detection. Based on the foundation of our prior study, CAGNet minimizes the number of training cycles and iterations, resulting in enhanced performance and efficiency. CAGNet is built upon the strategic design and integration of two types of convolutional attention mechanisms coupled with a transformer head applied for comprehensive feature analysis and fusion. To further augment segmentation precision, the network incorporates a custom edge-weighting scheme to optimize glass detection within images. Comparative studies and rigorous testing demonstrate that CAGNet outperforms several leading methodologies in glass detection, exhibiting robustness across a diverse range of conditions. Specifically, the IOU metric improves by 0.26% compared to that in our previous study and presents a 0.92% enhancement over those of other state-of-the-art methods.


  • Open Access

    TGSNet: Multi-Field Feature Fusion for Glass Region Segmentation Using Transformers

    by Xiaohang Hu, Rui Gao, Seungjun Yang, Kyungeun Cho

    Glass is a common object in living environments, but detecting it can be difficult because of the reflection and refraction of various colors of light in different environments; even humans are sometimes unable to detect glass. Currently, many methods are used to detect glass, but most rely on other sensors, which are costly and have difficulty collecting data. This study aims to solve the problem of detecting glass regions in a single RGB image by concatenating contextual features from multiple receptive fields and proposing a new enhanced feature fusion algorithm. To do this, we first construct a contextual attention module to extract backbone features through a self-attention approach. We then propose a VIT-based deep semantic segmentation architecture called MFT, which associates multilevel receptive field features and retains the feature information captured by each level of features. It is shown experimentally that our proposed method performs better on existing glass detection datasets than several state-of-the-art glass detection and transparent object detection methods, which fully demonstrates the better performance of our TGSNet.


  • Open Access

    Heterogeneous Feature Fusion Module Based on CNN and Transformer for Multiview Stereo Reconstruction

    by Rui Gao, Jiajia Xu, Yipeng Chen, Kyungeun Cho

    For decades, a vital area of computer vision research has been multiview stereo (MVS), which creates 3D models of a scene using photographs. This study presents an effective MVS network for 3D reconstruction utilizing multiview pictures. Alternative learning-based reconstruction techniques work well, because CNNs (convolutional neural network) can extract only the image’s local features; however, they contain many artifacts. Herein, a transformer and CNN are used to extract the global and local features of the image, respectively. Additionally, hierarchical aggregation and heterogeneous interaction modules were used to improve these features. They are based on the transformer and can extract dense features with 3D consistency and global context that are necessary to provide accurate matching for MVS.


  • Open Access

    Reflective Noise Filtering of Large-Scale Point Cloud Using Transformer

    by Rui Gao, Mengyu Li, Seung-Jun Yang, Kyungeun Cho

    Point clouds acquired with LiDAR are widely adopted in various fields, such as three-dimensional (3D) reconstruction, autonomous driving, and robotics. However, the high-density point cloud of large scenes captured with Lidar usually contains a large number of virtual points generated by the specular reflections of reflective materials, such as glass. When applying such large-scale high-density point clouds, reflection noise may have a significant impact on 3D reconstruction and other related techniques. In this study, we propose a method that uses deep learning and multi-position sensor comparison method to remove noise due to reflections from high-density point clouds in large scenes. The proposed method converts large-scale high-density point clouds into a range image and subsequently uses a deep learning method and multi-position sensor comparison method for noise detection. This alleviates the limitation of the deep learning networks, specifically their inability to handle large-scale high-density point clouds. The experimental results show that the proposed algorithm can effectively detect and remove noise due to reflection.


  • Open Access

    Reflective Noise Filtering of Large-Scale Point Cloud Using Multi-Position LiDAR Sensing Data

    by Rui Gao, Jisun Park, Xiaohang Hu, Seungjun Yang, Kyungeun Cho

    Signals, such as point clouds captured by light detection and ranging sensors, are often affected by highly reflective objects, including specular opaque and transparent materials, such as glass, mirrors, and polished metal, which produce reflection artifacts, thereby degrading the performance of associated computer vision techniques. In traditional noise filtering methods for point clouds, noise is detected by considering the distribution of the neighboring points. However, noise generated by reflected areas is quite dense and cannot be removed by considering the point distribution. Therefore, this paper proposes a noise removal method to detect dense noise points caused by reflected objects using multi-position sensing data comparison. The proposed method is divided into three steps. First, the point cloud data are converted to range images of depth and reflective intensity. Second, the reflected area is detected using a sliding window on two converted range images. Finally, noise is filtered by comparing it with the neighbor sensor data between the detected reflected areas. Experiment results demonstrate that, unlike conventional methods, the proposed method can better filter dense and large-scale noise caused by reflective objects. In future work, we will attempt to add the RGB image to improve the accuracy of noise detection.


  • Open Access

    Human Mesh Reconstruction with Generative Adversarial Networks from Single RGB Images

    by Rui Gao, Mingyun Wen, Jisun Park, Kyungeun Cho

    Applications related to smart cities require virtual cities in the experimental development stage. To build a virtual city that are close to a real city, a large number of various types of human models need to be created. To reduce the cost of acquiring models, this paper proposes a method to reconstruct 3D human meshes from single images captured using a normal camera. It presents a method for reconstructing the complete mesh of the human body from a single RGB image and a generative adversarial network consisting of a newly designed shape–pose-based generator (based on deep convolutional neural networks) and an enhanced multi-source discriminator. Using a machine learning approach, the reliance on multiple sensors is reduced and 3D human meshes can be recovered using a single camera, thereby reducing the cost of building smart cities. The proposed method achieves an accuracy of 92.1% in body shape recovery; it can also process 34 images per second. The method proposed in this paper approach significantly improves the performance compared with previous state-of-the-art approaches. Given a single view image of various humans, our results can be used to generate various 3D human models, which can facilitate 3D human modeling work to simulate virtual cities. Since our method can also restore the poses of the humans in the image, it is possible to create various human poses by given corresponding images with specific human poses.