Shunkun Liang

Despite the rapid advancements in event-based motion estimation, current geometric methods primarily focus on velocity estimation. However, absolute pose estimation, which is equally crucial for key applications such as robotic navigation and augmented reality, remains relatively underexplored. Consequently, the simultaneous recovery of absolute pose and velocity from event streams remains an open and challenging problem. To address this gap, we propose a geometric framework for absolute pose and velocity estimation by leveraging 3D lines in the scene and the events they trigger. At the core of the framework lie two key geometric constraints: the orthogonality between a 3D line and the normal vector of its corresponding event plane, and the collinearity of an event with the 2D projection of its associated line. Based on these constraints, we present both linear and polynomial solvers for absolute pose estimation. The former enables efficient computation, while the latter provides a globally optimal solution for rotation. For velocity estimation, we develop an efficient linear solver and a more accurate optimization-based solver to recover both angular and linear velocities. Notably, our methods require a minimum of three event-line correspondences to determine the 6-DoF absolute pose or velocities independently. Extensive experiments in simulation and on real-world datasets demonstrate that our methods achieve state-of-the-art performance, with significant improvements in accuracy and computational efficiency compared to existing methods. The demo code is publicly available at https://github.com/Zibin6/EventPoseVelocity.

Multi-camera systems offer rich observation capabilities for visual navigation and 3D scene reconstruction; however, the resulting feature redundancy often compromises computational efficiency. This challenge is particularly pronounced during bundle adjustment, where the non-linear optimization of both system poses and scene points incurs substantial computational overhead. To address this challenge, this paper introduces a pose-only geometric constraint for multi-camera systems and proposes a corresponding pose adjustment algorithm. Specifically, we use generalized camera model to establish a unified representation of the multi-camera system. Building upon this model, we formulate the multi-camera pose-only constraint, which implicitly represents a 3D scene point using two base observations and their associated poses, thereby achieving a pose-only representation of the projection geometry. Subsequently, we introduce a multi-camera pose adjustment algorithm that eliminates 3D points from the parameter space, thereby achieving efficient and focused pose optimization. Experimental results on both synthetic and real-world datasets demonstrate that the proposed algorithm outperforms baseline bundle adjustment methods in computational efficiency, while maintaining or even improving pose estimation accuracy

In this paper, we aim to estimate the relative pose and focal length between two views with known intrinsic parameters except for an unknown focal length from two affine correspondences (ACs). Cameras are commonly used in combination with inertial measurement units (IMUs) in applications such as self-driving cars, smartphones, and autonomous aerial vehicles. The vertical direction of camera views can be obtained by IMU measurements. The relative pose between two cameras is reduced from 5DOF to 3DOF. We propose a new solver to estimate the 3DOF relative pose and focal length. First, we establish constraint equations from two affine correspondences when the vertical direction is known. Then, based on the properties of the equation system with nontrivial solutions, four equations can be derived. These four equations only involve two parameters: the focal length and the relative rotation angle. Finally, the polynomial eigenvalue method is utilized to solve the problem of focal length and relative rotation angle. The proposed solver is evaluated using synthetic and real-world datasets. The results show that our solver performs better than the existing state-of-the-art solvers.

Vision-based pose estimation plays a crucial role in the autonomous navigation of flight platforms. However, the field of view and spatial resolution of the camera limit pose estimation accuracy. This paper designs a divergent multi-aperture imaging system (DMAIS), which is equivalent to a single imaging system capable of simultaneously achieving a large field of view and high spatial resolution observations. The DMAIS overcomes traditional observation limitations, allowing accurate pose estimation for the flight platform. Before conducting pose estimation, the DMAIS must be calibrated. To this end, we propose a calibration method for DMAIS based on the 3D calibration field. The calibration process determines the imaging parameters of the DMAIS, which allows us to model the DMAIS as a generalized camera. Subsequently, a new algorithm for accurately determining the pose of flight platform is introduced. We transform the absolute pose estimation problem into a nonlinear minimization problem. Using Lagrange multipliers, new optimality conditions are established for solving this problem. Finally, real calibration experiments show the effectiveness and accuracy of the proposed method. Results from real flight experiments validate the system’s ability to achieve centimeter-level positioning accuracy and arc-minute-level orientation accuracy.

Object pose tracking is one of the pivotal technologies in multimedia, attracting ever-growing attention in recent years. Existing methods employing traditional cameras encounter numerous challenges such as motion blur, sensor noise, partial occlusion, and changing lighting conditions. The emerging bio-inspired sensors, particularly event cameras, possess advantages such as high dynamic range and low latency, which hold the potential to address the aforementioned challenges. In this work, we present an optical flow-guided 6DoF object pose tracking method with an event camera. A 2D-3D hybrid feature extraction strategy is firstly utilized to detect corners and edges from events and object models, which characterizes object motion precisely. Then, we search for the optical flow of corners by maximizing the event-associated probability within a spatio-temporal window, and establish the correlation between corners and edges guided by optical flow. Furthermore, by minimizing the distances between corners and edges, the 6DoF object pose is iteratively optimized to achieve continuous pose tracking. Experimental results of both simulated and real events demonstrate that our methods outperform event-based state-of-the-art methods in terms of both accuracy and robustness.

Camera calibration is a crucial step in photogrammetry and 3D vision applications. In practical scenarios with a long working distance to cover a wide area, target-based calibration methods become complicated and inflexible due to site limitations. This paper introduces a novel camera calibration method using a collimator system, which can provide a reliable and controllable calibration environment for cameras with varying working distances. Based on the optical geometry of the collimator system, we prove that the relative motion between the target and camera conforms to the spherical motion model, reducing the original 6DOF relative motion to 3DOF pure rotation motion. Furthermore, a closed-form solver for multiple views and a minimal solver for two views are proposed for camera calibration. The performance of our method is evaluated in both synthetic and real-world experiments, which verify the feasibility of calibration using the collimator system and demonstrate that our method is superior to the state-of-the-art methods. Demo code is available at https://github.com/LiangSK98/CollimatorCalibration.

Mobile devices equipped with a multi-camera system and an inertial measurement unit (IMU) are widely used nowadays, such as self-driving cars. The task of relative pose estimation using visual and inertial information has important applications in various fields. To improve the accuracy of relative pose estimation of multi-camera systems, we propose a globally optimal solver using affine correspondences to estimate the generalized relative pose with a known vertical direction. First, a cost function about the relative rotation angle is established after decoupling the rotation matrix and translation vector, which minimizes the algebraic error of geometric constraints from affine correspondences. Then, the global optimization problem is converted into two polynomials with two unknowns based on the characteristic equation and its first derivative is zero. Finally, the relative rotation angle can be solved using the polynomial eigenvalue solver, and the translation vector can be obtained from the eigenvector. Besides, a new linear solution is proposed when the relative rotation is small. The proposed solver is evaluated on synthetic data and real-world datasets. The experiment results demonstrate that our method outperforms comparable state-of-the-art methods in accuracy.

For self-driving cars, smartphones, and unmanned aerial vehicles, cameras are typically used in combination with inertial measurement units (IMUs). The vertical direction of camera views can be determined based on partial IMU measurements—that is, the relative orientation of two views is reduced to a single degree of freedom. In this paper, we propose a globally optimal solver using affine correspondences to estimate the relative pose of two views with a known vertical direction. First, the relative pose estimation problem is formalized as a cost function optimization, which minimizes algebraic error in the least-squares sense. Then, the cost function optimization is converted into two polynomials with only two unknowns, which are represented by the parameter of the rotation angle. Finally, we use the eigenvalue polynomial solver to calculate the parameter of the rotation angle. The proposed solver is evaluated using synthetic data and real-world scenes from the KITTI and EuRoC benchmarks. The experimental results show that our method performs better than existing state-of-the-art solvers.

High-resolution high-frame-rate videos can record motion scenes detailedly and smoothly, but usually only professional cameras have enough transmission bandwidth to meet the video capture requirement. The conventional solutions use video processing methods such as video super-resolution (VSR) and video frame interpolation (VFI), but their results suffer from unreal spatial-temporal details in complex dynamic cases. To address this problem, we reconstruct a more real high-resolution high-frame-rate video using a hybrid video input, including a low-resolution high-frame-rate video (main video) and a high-resolution low-frame-rate video (auxiliary video). We propose a deep learning model named HIS-VSR, which consists of three parts: super-resolution of the main video, detail feature extraction of the auxiliary video and hybrid video information aggregation. Among them, the first part processes the main video to generate preliminary high-resolution frames; the second part warps the auxiliary frames for alignment and extracts their high-resolution detail features; the last part uses a weighted aggregation method to fuse the results of the first and second part. We train our model on synthetic datasets and demonstrate its excellent performance of reconstructing dynamic scenes by comparing it with Deep-SloMo on synthetic and real videos.