Shunbo Zhou

Generating dynamic scenes from images has gained increasing attention. Existing methods have two major limitations: (1) They can hardly handle sparse images which exhibit limited geometry constraints and insufficient motion; (2) They struggle to maintain spatial-temporal consistency when rendering multi-view videos. To address these limitations, we propose SCSV, a spatial-temporal consistent dynamic scene generation method from sparse views. Our method consists of two stages: scene reconstruction and scene expansion, both of which decouple background and foreground. In the scene reconstruction stage, we first interpolate a set of images between the input images based on a video generation model, followed by the optimization of the scene Gaussian from the interpolated and input images. To improve the spatial-temporal consistency of the reconstructed scene, we propose an uncertainty-aware Gaussian training approach, which introduces adaptive weights of images and pixels. In the scene expansion stage, for background, we render novel views and refine them with a geometry-aware diffusion process. These refined images are then used to incrementally add the Gaussians. As to foreground, we generate human motion according to previous motion, enabling temporal coherent generation of motion. To further enhance the physical plausibility, we integrate the expanded foreground into the background using a gravity-aware alignment. Experiments on NeuMan, Bonn, and EMDB datasets demonstrate that our SCSV achieves superior performance compared to state-of-the-art methods. The code will be released upon acceptance.

This paper focuses on the Light Detection and Ranging (LiDAR)–Inertial Measurement Unit (IMU) simultaneous localization and mapping (SLAM) problem: How to fuse the sensor measurement from the LiDAR and IMU to online estimate robot's poses and build a consistent map of the environment. This paper presents LTA‐OM: an efficient, robust, and accurate LiDAR SLAM system. Employing fast direct LiDAR‐inertial odometry (FAST‐LIO2) and Stable Triangle Descriptor as LiDAR–IMU odometry and the loop detection method, respectively, LTA‐OM is implemented to be functionally complete, including loop detection and correction, false‐positive loop closure rejection, long‐term association (LTA) mapping, and multisession localization and mapping. One novelty of this paper is the real‐time LTA mapping, which exploits the direct scan‐to‐map registration of FAST‐LIO2 and employs the corrected history map to provide direct global constraints to the LIO mapping process. LTA mapping also has the notable advantage of achieving drift‐free odometry at revisit places. Besides, a multisession mode is designed to allow the user to store the current session's results, including the corrected map points, optimized odometry, and descriptor database for future sessions. The benefits of this mode are additional accuracy improvement and consistent map stitching, which is helpful for life‐long mapping. Furthermore, LTA‐OM has valuable features for robot control and path planning, including high‐frequency and real‐time odometry, driftless odometry at revisit places, and fast loop closing convergence. LTA‐OM is versatile as it is applicable to both multiline spinning and solid‐state LiDARs, mobile robots and handheld platforms. In experiments, we exhaustively benchmark LTA‐OM and other state‐of‐the‐art LiDAR systems with 18 data sequences. The results show that LTA‐OM steadily outperforms other systems regarding trajectory accuracy, map consistency, and time consumption. The robustness of LTA‐OM is validated in a challenging scene—a multilevel building having similar structures at different levels. To demonstrate our system, we created a video which can be found on https://youtu.be/DVwppEKlKps.

LiDAR Mapping has been a long-standing problem in robotics. Recent progress in neural implicit representation has brought new opportunities to robotic mapping. In this letter, we propose the multi-volume neural feature fields, called NF-Atlas, which bridge the neural feature volumes with pose graph optimization. By regarding the neural feature volume as pose graph nodes and the relative pose between volumes as pose graph edges, the entire neural feature field becomes both locally rigid and globally elastic . Locally, the neural feature volume employs a sparse feature Octree and a small MLP to encode the signed distance function (SDF) of the submap with an option of semantics. Learning the map using this structure allows for end-to-end solving of maximum a posteriori (MAP) based probabilistic mapping. Globally, the map is built volume by volume independently, avoiding catastrophic forgetting when mapping incrementally. Furthermore, when a loop closure occurs, with the elastic pose graph based representation, only updating the origin of neural volumes is required without remapping. Finally, these functionalities of NF-Atlas are validated. Thanks to the sparsity and the optimization based formulation, NF-Atlas shows competitive performance in terms of accuracy, efficiency and memory usage on both simulation and real-world datasets.

This brief presents real-time configuration estimation and motion planning for the industrial tractor–trailers vehicle composed of a full-scale car-like tractor and multiple full trailers. For the real-life vehicle, determining the configuration is challenging. With only on-tractor sensors, we solve this problem by fusing information from system dynamics propagation, geometrical constraints among the articulated units, and matching with the prebuilt environment map. The solution is efficiently achieved by formulating and solving a maximum a posterior (MAP) estimation problem in the pose-graph optimization framework. With the complicated tractor–trailers’ structure, small mismatch between actual and planned trajectories is crucial to inherit obstacle-free guarantee from planning to execution. We consider the dynamics and focus on facilitating the reduction of the mismatch by proposing a controller-based smoothing method to perform online motion planning. The given waypoint path is smoothed by forward propagation using a deliberately designed controller to generate the trajectory. The efficiently computed trajectories are also obstacle-free and dynamically feasible. The controller is also applied in execution to precisely reproduce the planned system evolvement. We demonstrate the performance with the real-life industrial tractor–trailers’ vehicle.

Most of the previously proposed vehicle controllers are developed with the no-skid assumption, also presume that full states of the vehicle are available, which are usually violated for industrial tractors. This article dedicates to building a practical vision-based antiskid tracking control framework for industrial tractors. For that, we first present a novel nonlinear visual–inertial estimator (VIE) to estimate the real-time position and velocity of the tractor in unstructured environments (GPS-denied, no prior feature maps, etc.). Subsequently, VIE-based kinematic and dynamic controllers are designed for trajectory tracking. The proposed kinematic controller is easy to implement. It works well even if the vehicle is towing payloads since the sideslip angle can be reactively estimated and compensated. Besides, by fully considering vehicle dynamics, the VIE-based dynamic controller can be compatible with occasions that require higher control performance. It is proved by Lyapunov theory that the asymptotic stability of the closed-loop estimation-control system is guaranteed. Extensive full-scale experiments on an autonomous tractor are implemented to show the validity of the VIE-based control system in unstructured environments.

This letter presents a practical method for parameter estimation of a full-size industrial car-like tractor. With only on-vehicle sensors including three encoders, an IMU and a 3D LiDAR, we estimate the effective radius of rear wheels, ratio between the steering wheel angle and road wheel angle, the longitudinal position of center of gravity (COG), the yaw moment of inertia and the longitudinal and cornering tire stiffnesses. This letter innovatively introduces nonlinear vehicle-dynamics constraints into a factor-graph estimation framework that also fuses IMU and LiDAR measurements, thus achieving an easy-to-tune and highly-accurate parameter estimation solution. The batch maximum a posterior (MAP) problem is formulated and solved efficiently for both states and parameters. To make it even easier to tune, we perform specifically-designed motion patterns to simplify the required description model, and we estimate as fewer parameters as possible for one motion pattern. The dynamic constraints are also formulated according to the motion properties. Performance and results of the proposed method are validated in detail with experiments on the industrial car-like tractor.

This article presents the dynamic modeling and trajectory tracking control of the industrial tractor-trailers vehicle composed of a full-size car-like tractor and multiple full trailers. To eliminate the high uncertainty of the kinematic and dynamic parameters of the trailers in the model, we propose to install a force sensor to monitor the forces exerted on the tractor in real time. With the force measurements, the tractor can be considered separately and the dynamic model is significantly simplified. The inaccurate parameters and varying configurations of the trailers can all be ignored, whereas their dynamic effects on the tractor are still precisely captured. A Lyapunov-based trajectory tracking controller is proposed to online compensate the forces and drive the tractor onto the desired trajectory. The controller is developed using backstepping techniques and the stability is rigorously proved. The performance is illustrated with practical implementations on a full-scale industrial tractor towing several full trailers. The robustness and accuracy of the controller with force compensation are verified with different trailer settings. Implementation results show that applying the simplified model has generally superior performance than using the whole-vehicle model.

Automated driving of a heavy tractor-trailers vehicle using only low-cost onboard sensors is challenging. Complex system dynamics and towing configuration changes degrade the performance of conventional model-based estimation and control methods. In this paper, we propose an onboard-sensor-based automatic control system for a heavy tractor-trailers vehicle used in low-speed industrial applications. First, by introducing a new hitch force sensor, a tractor dynamic state estimator (TDSE) is proposed. It integrates the kinematic and dynamic information sources such that accurate and reactive dynamic states and model parameters of the lead tractor can be obtained simultaneously. Second, building on top of the TDSE, a highly practical integrated dynamic controller (IDC) is developed. It reliably enforces the heavy tractor-trailers vehicle to track the reference path and velocity under any towing situation. The performance of the proposed TDSE and IDC is proved theoretically. TDSE and IDC are also integrated into a full-size tractor-trailers vehicle, and their feasibility and superiority are validated through extensive experiments.