KNKenta Niwa

Kenta Niwa

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article

Previous Organizations / Institutes

The organizations / institutes from the scholar's other articles in the past ten years


Kenta Niwa has a total of 79 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Natural Gradient Primal-Dual Method for Decentralized Learning

    by Kenta Niwa, Hiro Ishii, Hiroshi Sawada, Akinori Fujino, Noboru Harada, Rio Yokota

    We propose the Natural Gradient Primal-Dual (NGPD) method for decentralized learning of parameters in Deep Neural Networks (DNNs). Conventional approaches, such as the primal-dual method, constrain the local parameters to be similar between connected nodes. However, since most of them follow a first-order optimization method and the loss functions of DNNs may have ill-conditioned curvatures, many local parameter updates and communication among local nodes are needed. For fast convergence, we integrate the second-order natural gradient method into the primal-dual method (NGPD). Since additional constraint minimizes the amount of output change before and after the parameter updates, robustness towards ill-conditioned curvatures is expected. We theoretically demonstrate the convergence rate for the averaged parameter (the average of the local parameters) under certain assumptions. As a practical implementation of NGPD without a significant increase in computational overheads, we introduce Kronecker Factored Approximate Curvature (K-FAC). Our experimental results confirmed that NGPD achieved the highest test accuracy through image classification tasks using DNNs.

  • Open Access

    DP-Norm: Differential Privacy Primal-Dual Algorithm for Decentralized Federated Learning

    by Takumi Fukami, Tomoya Murata, Kenta Niwa, Iifan Tyou

    A novel algorithm is proposed for highly privacy-preserving decentralized federated learning (FL). Several studies have reported security risks in decentralized FL by reconstructing data even from model update differences. A common approach to overcome this issue is to use the diffusion process following differential privacy (DP), i.e., message passing between nodes is hidden by noise. However, this often makes the learning process unstable, leading to degraded results compared to without using DP diffusion process. In this paper, we propose a primal-dual DP algorithm with denoising normalization (DP-Norm) for less sensitivity to noise/interference, such as DP diffusion and heterogeneous data allocation. For DP-Norm, privacy analysis to determine minimal noise level and convergence analysis are conducted. Through image classification benchmark tests, we confirmed that DP-Norm performed close to the single-node reference score, even when statistically heterogeneous data was allocated on six nodes.


  • Open Access

    A Localized Primal-Dual Method for Centralized/Decentralized Federated Learning Robust to Data Heterogeneity

    by Iifan Tyou, Tomoya Murata, Takumi Fukami, Yuki Takezawa, Kenta Niwa

    Generalized Edge-Consensus Learning (G-ECL) is a primal-dual method to solve loss-sum minimization problems. We propose Local Generalized Edge-Consensus Learning (Local G-ECL) as an extension of previous G-ECL, aiming to be a decentralized/centralized FL algorithm robust to heterogeneous data sets with a large number of local updates. Our contributions are as follows: (C1) success in theoretical gradient norm convergence analysis nearly independently of data heterogeneity, and (C2) equivalency proof between our primal-dual Local G-ECL and a pure primal Stochastic Controlled Averaging (SCAFFOLD) algorithm in centralized settings, where the difference in the initial local model for each round is ignored. Numerical experiments using image classification tests validated that Local G-ECL is robust to heterogeneous data with a large number of local updates.


  • Open Access

    Embarrassingly Simple Text Watermarks

    by Ryoma Sato, Yuki Takezawa, Han Bao, Kenta Niwa, Makoto Yamada

    We propose Easymark, a family of embarrassingly simple yet effective watermarks. Text watermarking is becoming increasingly important with the advent of Large Language Models (LLM). LLMs can generate texts that cannot be distinguished from human-written texts. This is a serious problem for the credibility of the text. Easymark is a simple yet effective solution to this problem. Easymark can inject a watermark without changing the meaning of the text at all while a validator can detect if a text was generated from a system that adopted Easymark or not with high credibility. Easymark is extremely easy to implement so that it only requires a few lines of code. Easymark does not require access to LLMs, so it can be implemented on the user-side when the LLM providers do not offer watermarked LLMs. In spite of its simplicity, it achieves higher detection accuracy and BLEU scores than the state-of-the-art text watermarking methods. We also prove the impossibility theorem of perfect watermarking, which is valuable in its own right. This theorem shows that no matter how sophisticated a watermark is, a malicious user could remove it from the text, which motivate us to use a simple watermark such as Easymark. We carry out experiments with LLM-generated texts and confirm that Easymark can be detected reliably without any degradation of BLEU and perplexity, and outperform state-of-the-art watermarks in terms of both quality and reliability.


  • Open Access

    On Accelerating Diffusion-Based Sampling Process via Improved Integration Approximation

    by Guoqiang Zhang, Niwa Kenta, W. Bastiaan Kleijn

    A popular approach to sample a diffusion-based generative model is to solve an ordinary differential equation (ODE). In existing samplers, the coefficients of the ODE solvers are pre-determined by the ODE formulation, the reverse discrete timesteps, and the employed ODE methods. In this paper, we consider accelerating several popular ODE-based sampling processes (including EDM, DDIM, and DPM-Solver) by optimizing certain coefficients via improved integration approximation (IIA). We propose to minimize, for each time step, a mean squared error (MSE) function with respect to the selected coefficients. The MSE is constructed by applying the original ODE solver for a set of fine-grained timesteps, which in principle provides a more accurate integration approximation in predicting the next diffusion state. The proposed IIA technique does not require any change of a pre-trained model, and only introduces a very small computational overhead for solving a number of quadratic optimization problems. Extensive experiments show that considerably better FID scores can be achieved by using IIA-EDM, IIA-DDIM, and IIA-DPM-Solver than the original counterparts when the neural function evaluation (NFE) is small (i.e., less than 25).

  • Open Access

    Momentum Tracking: Momentum Acceleration for Decentralized Deep Learning on Heterogeneous Data

    by Yuki Takezawa, Han Bao, Kenta Niwa, Ryoma Sato, Makoto Yamada

    SGD with momentum is one of the key components for improving the performance of neural networks. For decentralized learning, a straightforward approach using momentum is Distributed SGD (DSGD) with momentum (DSGDm). However, DSGDm performs worse than DSGD when the data distributions are statistically heterogeneous. Recently, several studies have addressed this issue and proposed methods with momentum that are more robust to data heterogeneity than DSGDm, although their convergence rates remain dependent on data heterogeneity and deteriorate when the data distributions are heterogeneous. In this study, we propose Momentum Tracking, which is a method with momentum whose convergence rate is proven to be independent of data heterogeneity. More specifically, we analyze the convergence rate of Momentum Tracking in the setting where the objective function is non-convex and the stochastic gradient is used. Then, we identify that it is independent of data heterogeneity for any momentum coefficient $\beta \in [0, 1)$. Through experiments, we demonstrate that Momentum Tracking is more robust to data heterogeneity than the existing decentralized learning methods with momentum and can consistently outperform these existing methods when the data distributions are heterogeneous.

  • Open Access

    Communication Compression for Decentralized Learning With Operator Splitting Methods

    by Yuki Takezawa, Kenta Niwa, Makoto Yamada

    In decentralized learning, operator splitting methods using a primal-dual formulation (e.g., Edge-Consensus Learning (ECL)) have been shown to be robust to heterogeneous data and have attracted significant attention in recent years. However, in the ECL, a node needs to exchange dual variables with its neighbors. These exchanges incur significant communication costs. For the Gossip-based algorithms, many compression methods have been proposed, but these Gossip-based algorithms do not perform well when the data distribution held by each node is statistically heterogeneous. In this work, we propose a novel framework of the compression methods for the ECL, called the Communication Compressed ECL (C-ECL). Specifically, we reformulate the update formulas of the ECL and propose to compress the update values of the dual variables. We demonstrate experimentally that the C-ECL can achieve a nearly equivalent performance with fewer parameter exchanges than the ECL. Moreover, we demonstrate that the C-ECL is more robust to heterogeneous data than the Gossip-based algorithms.


  • Open Access

    Lookahead Diffusion Probabilistic Models for Refining Mean Estimation

    by Guoqiang Zhang, Niwa Kenta, W. Bastiaan Kleijn

    We propose lookahead diffusion probabilistic models (LA-DPMs) to exploit the correlation in the outputs of the deep neural networks (DNNs) over subsequent timesteps in diffusion probabilistic models (DPMs) to refine the mean estimation of the conditional Gaussian distributions in the backward process. A typical DPM first obtains an estimate of the original data sample $\boldsymbol{x}$ by feeding the most recent state $\boldsymbol{z}_i$ and index $i$ into the DNN model and then computes the mean vector of the conditional Gaussian distribution for $\boldsymbol{z}_{i-1}$. We propose to calculate a more accurate estimate for $\boldsymbol{x}$ by performing extrapolation on the two estimates of $\boldsymbol{x}$ that are obtained by feeding $(\boldsymbol{z}_{i+1},i+1)$ and $(\boldsymbol{z}_{i},i)$ into the DNN model. The extrapolation can be easily integrated into the backward process of existing DPMs by introducing an additional connection over two consecutive timesteps, and fine-tuning is not required. Extensive experiments showed that plugging in the additional connection into DDPM, DDIM, DEIS, S-PNDM, and high-order DPM-Solvers leads to a significant performance gain in terms of FID score.

  • CoordiNet: Constrained Dynamics Learning for State Coordination Over Graph

    by Kenta Niwa, Naonori Ueda, Hiroshi Sawada, Akinori Fujino, Shoichiro Takeda, Guoqiang Zhang, W. Bastiaan Kleijn

    A neural architecture to efficiently coordinate the transitions of state variables (states) over a graph is proposed. We consider the coordination of the time evolution of the state variables associated with the nodes on a graph. The states are associated with physical attributes of agents, e.g., the speed and/or location of vehicles. Efficient coordination then corresponds to the optimization of dynamics (how fast do vehicles travel) that may be subject to constraints (travel must be collision-free). The aim of this paper is (i) the formulation of learnable constrained dynamics which governs the transition of states under constraints over a graph and (ii) to show its industrial application. Firstly, we formulate continuous ordinary differential equations (ODEs), namely forward propagation of state transitions over a graph and backward propagation to optimize model parameters for the learnable constrained dynamics. Discretization of these continuous ODEs results in the neural architecture CoordiNet . Secondly, as an application of CoordiNet, we address a traffic coordination problem by learning constrained dynamics such that vehicles can travel as fast as possible without collisions. Simulation experiments of traffic coordination confirm that our method maximizes the vehicles' speed states while avoiding collisions.


  • Open Access

    A DNN Optimizer that Improves over AdaBelief by Suppression of the Adaptive Stepsize Range

    by Guoqiang Zhang, Kenta Niwa, W. Bastiaan Kleijn

    We make contributions towards improving adaptive-optimizer performance. Our improvements are based on suppression of the range of adaptive stepsizes in the AdaBelief optimizer. Firstly, we show that the particular placement of the parameter epsilon within the update expressions of AdaBelief reduces the range of the adaptive stepsizes, making AdaBelief closer to SGD with momentum. Secondly, we extend AdaBelief by further suppressing the range of the adaptive stepsizes. To achieve the above goal, we perform mutual layerwise vector projections between the gradient g_t and its first momentum m_t before using them to estimate the second momentum. The new optimization method is referred to as Aida. Thirdly, extensive experimental results show that Aida outperforms nine optimizers when training transformers and LSTMs for NLP, and VGG and ResNet for image classification over CIAF10 and CIFAR100 while matching the best performance of the nine methods when training WGAN-GP models for image generation tasks. Furthermore, Aida produces higher validation accuracies than AdaBelief for training ResNet18 over ImageNet. Code is available at this URL