Chengli Tan

In data mining, subspace clustering is a crucial technique which determines the union of the underlying subspace to cluster data points in an unsupervised manner. Although deep-learning-based subspace clustering, typically referred to as deep subspace clustering (DSC), has significantly improved clustering accuracy, existing DSC models still struggle to capture a comprehensive and compact latent representation as they generally explore the spatial domain to extract useful information and face difficulty in balancing the high mutual and low redundant information between the original input space and latent subspace. This leads to the performance of the model being dependent on initialization, resulting in a lack of stability. In this study, a novel network is proposed to extract features in both the frequency domain and spatial domain. We introduce three types of ResBlocks in the discrete Fourier transform (DFT), discrete cosine transform (DCT), or discrete wavelet transform (DWT) frequency domains separately to learn both the low-frequency and high-frequency information in the proposed networks. Additionally, to extract concise and rich latent representations, IB loss is employed by deriving a variational lower bound on the IB objective. Extensive experiments on several benchmark datasets verify the effectiveness of our networks compared to state-of-the-art models. In addition, detailed ablation studies are performed to demonstrate the advantages of the two introduced components.

Deep neural networks have been increasingly used in safety-critical applications such as medical diagnosis and autonomous driving. However, many studies suggest that they are prone to being poorly calibrated and have a propensity for overconfidence, which may have disastrous consequences. In this paper, unlike standard training such as stochastic gradient descent, we show that the recently proposed sharpness-aware minimization (SAM) counteracts this tendency towards overconfidence. The theoretical analysis suggests that SAM allows us to learn models that are already well-calibrated by implicitly maximizing the entropy of the predictive distribution. Inspired by this finding, we further propose a variant of SAM, coined as CSAM, to ameliorate model calibration. Extensive experiments on various datasets, including ImageNet-1K, demonstrate the benefits of SAM in reducing calibration error. Meanwhile, CSAM performs even better than SAM and consistently achieves lower calibration error than other approaches

Accurate interpolation of seismic data is crucial for improving the quality of imaging and interpretation. In recent years, deep learning models such as U-Net and generative adversarial networks have been widely applied to seismic data interpolation. However, they often underperform when the training and test missing patterns do not match. To alleviate this issue, here we propose a novel framework that is built upon the multi-modal adaptable diffusion models. In the training phase, following the common wisdom, we use the denoising diffusion probabilistic model with a cosine noise schedule. This cosine global noise configuration improves the use of seismic data by reducing the involvement of excessive noise stages. In the inference phase, we introduce the denoising diffusion implicit model to reduce the number of sampling steps. Different from the conventional unconditional generation, we incorporate the known trace information into each reverse sampling step for achieving conditional interpolation. To enhance the coherence and continuity between the revealed traces and the missing traces, we further propose two strategies, including successive coherence correction and resampling. Coherence correction penalizes the mismatches in the revealed traces, while resampling conducts cyclic interpolation between adjacent reverse steps. Extensive experiments on synthetic and field seismic data validate our model’s superiority and demonstrate its generalization capability to various missing patterns and different noise levels with just one training session. In addition, uncertainty quantification and ablation studies are also investigated.

Lookahead is a popular stochastic optimizer that can accelerate the training process of deep neural networks. However, the solutions found by Lookahead often generalize worse than those found by its base optimizers, such as SGD and Adam. To address this issue, we propose Sharpness-Aware Lookahead (SALA), a novel optimizer that aims to identify flat minima that generalize well. SALA divides the training process into two stages. In the first stage, the direction towards flat regions is determined by leveraging a quadratic approximation of the optimization trajectory, without incurring any extra computational overhead. In the second stage, however, it is determined by Sharpness-Aware Minimization (SAM), which is particularly effective in improving generalization at the terminal phase of training. In contrast to Lookahead, SALA retains the benefits of accelerated convergence while also enjoying superior generalization performance compared to the base optimizer. Theoretical analysis of the expected excess risk, as well as empirical results on canonical neural network architectures and datasets, demonstrate the advantages of SALA over Lookahead. It is noteworthy that with approximately 25% more computational overhead than the base optimizer, SALA can achieve the same generalization performance as SAM which requires twice the training budget of the base optimizer.

Deep neural networks complete a feature extraction task by propagating the inputs through multiple modules. However, how the representations evolve with the gradient-based optimization remains unknown. Here we leveraged the intrinsic dimension of the representations to study the learning dynamics and found that the training process underwent a phase transition from expansion to compression under disparate training regimes---a phenomenon that is ubiquitous across a wide variety of model architectures, optimizers, and data sets. We showed that the variation in the intrinsic dimension is consistent with the complexity of the learned hypothesis, which can be quantitatively assessed by the critical sample ratio that was rooted in adversarial robustness. Meanwhile, we mathematically demonstrated that this phenomenon can be analyzed in terms of the mutable correlation between neurons. Although the evoked activities obey a power-law decaying rule in biological circuits, we identified that the power-law exponent of the representations in deep neural networks predicted adversarial robustness well only at the end of the training but not during the training process. These results together suggest that deep neural networks are prone to producing robust representations by adaptively eliminating or retaining redundancies. The code is publicly available at \url{https://github.com/cltan023/learning2022}.

Stochastic gradient descent (SGD) is of fundamental importance in deep learning. Despite its simplicity, elucidating its efficacy remains challenging. Conventionally, the success of SGD is ascribed to the stochastic gradient noise (SGN) incurred in the training process. Based on this consensus, SGD is frequently treated and analyzed as the Euler–Maruyama discretization of stochastic differential equations (SDEs) driven by either Brownian or Lévy stable motion. In this study, we argue that SGN is neither Gaussian nor Lévy stable. Instead, inspired by the short-range correlation emerging in the SGN series, we propose that SGD can be viewed as a discretization of an SDE driven by fractional Brownian motion (FBM). Accordingly, the different convergence behavior of SGD dynamics is well-grounded. Moreover, the first passage time of an SDE driven by FBM is approximately derived. The result suggests a lower escaping rate for a larger Hurst parameter, and thus, SGD stays longer in flat minima. This happens to coincide with the well-known phenomenon that SGD favors flat minima that generalize well. Extensive experiments are conducted to validate our conjecture, and it is demonstrated that short-range memory effects persist across various model architectures, datasets, and training strategies. Our study opens up a new perspective and may contribute to a better understanding of SGD.