Tao Yang

Integrating neural rendering with simultaneous localization and mapping (SLAM) has shown great promise for achieving high-precision localization and photorealistic scene reconstruction. However, the dynamic perspectives of uncrewed aerial vehicles (UAVs), compounded by Global Navigation Satellite System (GNSS) signal interference and the accumulation of localization errors, pose significant challenges to existing monocular SLAM methods in complex environments. To address these limitations, we propose LD3DGS-SLAM, a novel framework that integrates monocular SLAM, GNSS, and 3-D Gaussian splatting (3DGS) to enhance UAV-based localization and mapping. The system integrates traditional geometric feature-based localization with the multiview synthesis capability of 3-D Gaussian rendering, achieving a breakthrough in accuracy while enhancing robustness. First, we construct a multisensor fusion graph optimization model that tightly integrates GNSS and monocular vision data, effectively mitigating the cumulative drift typical of traditional SLAM pipelines. Afterward, we introduce a 3-D Gaussian-based mapping strategy that incrementally refines a dense scene representation using SLAM-generated point clouds, fusing geometric and texture information to reconstruct highly accurate and detailed maps. To further enhance localization performance in GNSS-denied scenarios, we develop a dynamic frame interpolation and tracking approach based on 3-D Gaussian rendering. By synthesizing novel viewpoints, this method improves observation matching and strengthens loop closure detection, enabling robust relocalization in large-scale environments. To validate the effectiveness of the proposed framework, we implement a UAV test system and evaluate its performance on both the publicly available TUM dataset and a self-collected aerial dataset spanning several kilometers. Experimental results show that LD3DGS-SLAM achieves a state-of-the-art average localization error of only 0.8 m using just a monocular camera, even during long-range flight missions exceeding tens of kilometers and altitudes up to 500 m. Overall, LD3DGS-SLAM effectively addresses the limitations of monocular SLAM in aerial scenarios, providing a robust, accurate, and cost-effective localization solution for urban air mobility (UAM) and Aerial Internet of Things (IoT) applications.

Simultaneous localization and mapping (SLAM) has attracted tremendous interest from members of the research community in recent years because of its ability to make the robot truly independent in navigation. The integration of neural rendering and the SLAM system showed promising results in joint localization and photorealistic view reconstruction. However, currently available methods that fully rely on implicit representations are resource hungry. Recent work has shown that 3D Gaussians enable high-quality reconstruction and real-time rendering of scenes using multiple posed cameras. Owing to the increasing popularity of Gaussian splatting (GS) and its expanding research area, we present a comprehensive survey of GS based SLAM papers. This survey addresses the challenges of neural based systems by exploring the application of GS within SLAM frameworks. We present a comprehensive analysis of Gaussian Splatting for scene representation in SLAM, focusing on the past three years of research. Our survey delves into the advantages of GS based SLAM, including efficiency, unified representation for various tasks, suitability for resource-constrained devices, and high accuracy and robustness for dense scene reconstruction. Additionally, the survey includes a broader range of research, including semantic GS techniques, to provide a holistic understanding of this approach. By offering an in-depth exploration of Gaussian Splatting in SLAM, a benchmark comparison of performance, and a focus on 3D Gaussian Splatting (3DGS) based SLAM advantages, this survey aims to be a valuable resource for researchers and developers seeking to leverage Gaussian Splatting for efficient and robust SLAM.

Anti-aliasing is a crucial research topic in computer graphics, which can significantly enhance the rendering quality of neural radiance fields (NeRF). Recent studies have introduced effective anti-aliasing NeRF methods, utilizing cone-casting to replace ray-casting and modeling the 3D observation area of pixels as circular cones. The cone-casting strategy has successfully reduced blurring and aliasing in novel view rendering. However, we have observed that the light cones are not standard circular cones because the camera projection model distorts them into elliptic cones of diverse sizes and shapes. This finding motivates us to model pixel light cones as anisotropic elliptic cones and propose an elliptic cone-casting-based anti-aliasing NeRF method called “ECC-NeRF". Specifically, we first derive the elliptic cone models for common pinhole, fisheye, and panoramic cameras based on their camera projection models. Then, we integrate the proposed elliptic cone-casting into two representative cone-casting-based anti-aliasing NeRF methods: Mip-NeRF and Zip-NeRF. Our experimental evaluations on multiple datasets demonstrate that our method can achieve more accurate multi-scale anisotropic representation and better novel view rendering quality with negligible additional computation cost.

This paper presents a novel panoramic stitching solution for real-time monitoring of longwall mining faces, addressing significant parallax issues inherent in underground environments. Using a depth camera array and leveraging a local point cloud registration method, our method enhances real-time monitoring, ensuring precise scene alignment measurement and object placement in dynamic environments. Furthermore, we introduce depth completion to resolve issues related to incomplete depth data and propose a scan projection method to maintain image integrity in constrained, tunnel-like spaces. Our system has been validated in both real and simulated environments. The experimental results demonstrate superior performance compared to existing state-of-the-art methods, with an operational speed of 25 fps.

In this paper, we present an end-to-end holographic video conferencing system that enables real-time high-quality free-viewpoint rendering of participants in different spatial regions, placing them in a unified virtual space for a more immersive display. Our system offers a cost-effective, complete holographic conferencing process, including multiview 3D data capture, RGB-D stream compression and transmission, high-quality rendering, and immersive display. It employs a sparse set of commodity RGB-D cameras that capture 3D geometric and textural information. We then remotely transmit color and depth maps via standard video encoding and transmission protocols. We propose a GPU-parallelized rendering pipeline based on an image-based virtual view synthesis algorithm to achieve real-time and high-quality scene rendering. This algorithm uses an on-the-fly Truncated Signed Distance Function (TSDF) approach, which marches along virtual rays within a computed precise search interval to determine surface intersections. We then design a multiweight projective texture mapping method to fuse color information from multiple views. Furthermore, we introduce a method that uses a depth confidence map to weight the rendering results from different views, which mitigates the impact of sensor noise and inaccurate measurements on the rendering results. Finally, our system places conference participants from different spaces into a virtual conference environment with a global coordinate system through coordinate transformation, which simulates a real conference scene in physical space, providing an immersive remote conferencing experience. Experimental evaluations confirm our system’s real-time, low-latency, high-quality, and immersive capabilities.

Achieving precise synchronization is critical for multi-camera systems in various applications. Traditional methods rely on hardware-triggered synchronization, necessitating significant manual effort to connect and adjust synchronization cables, especially with multiple cameras involved. This not only increases labor costs but also restricts scene layout and incurs high setup expenses. To address these challenges, we propose a novel subframe synchronization technique for multi-camera systems that operates without the need for additional hardware triggers. Our approach leverages a time-calibrated video featuring specific markers and a uniformly moving ball to accurately extract the temporal relationship between local and global time systems across cameras. This allows for the calculation of new timestamps and precise frame-level alignment. By employing interpolation algorithms, we further refine synchronization to the subframe level. Experimental results validate the robustness and high temporal precision of our method, demonstrating its adaptability and potential for use in demanding multi-camera setups.

Free-viewpoint videos offer audiences a more immersive and liberated way to watch sports. The rendering of sports scenes encompasses two essential elements: dynamic targets and static scenes. While much current research focuses on achieving high-quality rendering of human bodies, rendering large-scale sports scenes presents various challenges. Sports arenas are characterized by large spatial extents, restricted camera placement, uncontrollable lighting, weak textures, and repetitive patterns, all of which pose significant obstacles to achieving high-quality scene rendering. In this work, we propose a neural radiance field rendering method based on scene-prior geometric constraints. We introduce prior 3D geometric dimensions and 2D semantic masks, to derive high-precision ground plane depth maps from camera imaging parameters. This is a geometry-based method that does not rely on visual features, and thus it is unaffected by insufficient textures, repetition, and reflections. Subsequently, we apply ground depth maps as geometric consistency constraints to optimize the neural rendering network, thereby reducing the impact of color inconsistencies across viewpoints. The visual sensor network we build can synchronously capture static fields and dynamic targets in sports scenes. Based on the visual sensor network, we collected multiviewpoint datasets of large-scale sports scenes at Invengo and Xidian Gymnasium for performance evaluation. Experimental results demonstrate that our method can generate high-precision and cross-viewpoint scale-consistent depth constraints, and helps reduce holes and artifacts in synthesized views. Our method outperforms the state-of-the-art for novel view rendering for challenging large-scale sports scenes.

We introduce GS-SFS, a method that utilizes a camera array with wide baselines for high-quality multiple human mesh reconstruction in large-scale sports scenes. Traditional human reconstruction methods in sports scenes, such as Shape-from-Silhouette (SFS), struggle with sparse camera setups and small human targets, making it challenging to obtain complete and accurate human representations. Despite advances in differentiable rendering, including 3D Gaussian Splatting (3DGS), which can produce photorealistic novel-view renderings with dense inputs, accurate depiction of surfaces and generation of detailed meshes is still challenging. Our approach uniquely combines 3DGS's view synthesis with an optimized SFS method, thereby significantly enhancing the quality of multiperson mesh reconstruction in large-scale sports scenes. Specifically, we introduce body shape priors, including the human surface point clouds extracted through SFS and human silhouettes, to constrain 3DGS to a more accurate representation of the human body only. Then, we develop an improved mesh reconstruction method based on SFS, mainly by adding additional viewpoints through 3DGS and obtaining a more accurate surface to achieve higher-quality reconstruction models. We implement a highdensity scene resampling strategy based on spherical sampling of human bounding boxes and render new perspectives using 3D Gaussian Splatting to create precise and dense multi-view human silhouettes. During mesh reconstruction, we integrate the human body's 2D Signed Distance Function (SDF) into the computation of the SFS's implicit surface field, resulting in smoother and more accurate surfaces. Moreover, we enhance mesh texture mapping by blending original and rendered images with different weights, preserving high-quality textures while compensating for missing details. The experimental results from real basketball game scenarios demonstrate the significant improvements of our approach for multiple human body model reconstruction in complex sports settings