Seongmin Lee

한국멀티미디어학회(The Journal of Curriculum Evaluation)

Comparing different 3D shapes is challenging due to their irregularities. Motivated by the human visual system mechanism, where the entire 3D geometry is clearly perceived as a series of multiple projections, we propose a novel facial shape similarity measurement using multiview deep perceptual representations. We introduce a multiview disentangling scheme that accurately represents a facial mesh in multiple coordinates and the training strategy with view specificity and regional consistency to reliably train the network with multiple projections. View specificity pertains to the human visual perception to better recognize facial similarity. Regional consistency mitigates regional redundancy among views. Hence, robust perceptual features with respect to views are embedded and accurate similarity can be measured. Consequently, the view-specific integration scheme incorporates the similarities of all views, allowing for highly consistent measurement. The experiments demonstrate that the proposed similarity outperforms state-of-the-arts and significantly improves the details in terms of geometry and human perception.

Recent learning-based multi-view stereo (MVS) still exhibits insufficient accuracy in large occlusion cases, such as environments with significant inter-camera distance or when capturing objects with complex shapes. This is because incorrect image features extracted from occluded areas serve as significant noise in the cost volume construction. To address this, we propose a visibility-aware MVS using surface normal weighting (SnowMVSNet) based on explicit 3D geometry. It selectively suppresses mismatched features in the cost volume construction by computing inter-view visibility. Additionally, we present a geometry-guided cost volume regularization that enhances true depth among depth hypotheses using a surface normal prior. We also propose intra-view visibility that distinguishes geometrically more visible pixels within a reference view. Using intra-view visibility, we introduce the visibility-weighted training and depth estimation methods. These methods enable the network to achieve accurate 3D point cloud reconstruction by focusing on visible regions. Based on simple inter-view and intra-view visibility computations, SnowMVSNet accomplishes substantial performance improvements relative to computational complexity, particularly in terms of occlusion robustness. To evaluate occlusion robustness, we constructed a multi-view human (MVHuman) dataset containing general human body shapes prone to self-occlusion. Extensive experiments demonstrated that SnowMVSNet significantly outperformed state-of-the-art methods in both low- and high-occlusion scenarios.

Denoising diffusion models have shown a powerful capacity for generating high-quality image samples by progressively removing noise. Inspired by this, we present a diffusion-based mesh denoiser that progressively removes noise from mesh. In general, the iterative algorithm of diffusion models attempts to manipulate the overall structure and fine details of target meshes simultaneously. For this reason, it is difficult to apply the diffusion process to a mesh denoising task that removes artifacts while maintaining a structure. To address this, we formulate a structure-preserving diffusion process. Instead of diffusing the mesh vertices to be distributed as zero-centered isotopic Gaussian distribution, we diffuse each vertex into a specific noise distribution, in which the entire structure can be preserved. In addition, we propose a topology-agnostic mesh diffusion model by projecting the vertex into multiple 2-D viewpoints to efficiently learn the diffusion using a deep network. This enables the proposed method to learn the diffusion of arbitrary meshes that have an irregular topology. Finally, the denoised mesh can be obtained via refinement based on 2-D projections obtained from reverse diffusion. Through extensive experiments, we demonstrate that our method outperforms the state-of-the-art mesh denoising methods in both quantitative and qualitative evaluations.

One of the most crucial aspects of 3D facial models is facial reconstruction. However, it is unclear if face shape distortion is caused by identity or expression when the 3D morphable model (3DMM) is fitted into largely expressive faces. In order to overcome the problem, we introduce neural networks to reconstruct stable and precise faces in time. The reconstruction network extracts the 3DMM parameters from video sequences to represent 3D faces in time. Meanwhile, our displacement networks learn the changes in facial landmarks. In particular, the networks learn changes caused by facial identity, facial expression, and temporal cues, respectively. The proposed facial alignment network exhibits reliable and precise performance in reconstructing static and dynamic faces by leveraging these displacement networks. The 300 Videos in the Wild (300VW) dataset is utilized for qualitative and quantitative evaluations to confirm the effectiveness of our method. The results demonstrate the considerable advantages of our method in reconstructing 3D faces from video sequences.

한국멀티미디어학회(The Journal of Curriculum Evaluation)

It is possible to construct cost-efficient three-dimensional (3D) or four-dimensional (4D) scanning systems using multiple affordable off-the-shelf RGB-D sensors to produce high-quality reconstructions of 3D objects. However, the quality of these systems’ reconstructions is sensitive to a number of factors in reconstruction pipelines, such as multi-view calibration, depth estimation, 3D reconstruction, and color mapping accuracy, because the successive pipelines to reconstruct 3D meshes from multiple active stereo sensors are strongly correlated with each other. This paper categorizes the pipelines into sub-procedures and analyze various factors that can significantly affect reconstruction quality. Thus, this paper provides analytical and practical guidelines for high-quality 3D reconstructions with off-the-shelf sensors. For each sub-procedure, this paper shows comparisons and evaluations of several methods using data captured by 18 RGB-D sensors and provide analyses and discussions towards robust 3D reconstruction. Through various experiments, it has been demonstrated that significantly more accurate 3D scans can be obtained with the considerations along the pipelines. We believe our analyses, benchmarks, and guidelines will help anyone build their own studio and their further research for 3D reconstruction.

The relationship between the disparity and depth information of corresponding pixels is inversely proportional. Thus, in order to accurately estimate depth from stereo vision, it is important to obtain accurate disparity maps, which encode the difference between horizontal coordinates of corresponding image points. Stereo vision can be classified as either passive or active. Active stereo vision generates pattern texture, which passive stereo vision does not have, on the image to fill the textureless regions. In passive stereo vision, many surveys have discovered that disparity accuracy is heavily reliant on attributes, such as radiometric variation and color variation, and have found the best-performing conditions. However, in active stereo matching, the accuracy of the disparity map is influenced not only by those affecting the passive stereo technique, but also by the attributes of the generated pattern textures. Therefore, in this paper, we analyze and evaluate the relationship between the performance of the active stereo technique and the attributes of pattern texture. When evaluating, experiments are conducted under various settings, such as changing the pattern intensity, pattern contrast, number of pattern dots, and global gain, that may affect the overall performance of the active stereo matching technique. Through this evaluation, our discovery can act as a noteworthy reference for constructing an active stereo system.