Ion Necoara

In this paper, we propose an adaptive forward-backward-forward splitting algorithm for finding a zero of a pseudo-monotone operator that is split as a sum of three operators: the first is continuous single-valued, the second is Lipschitzian, and the third is maximally monotone. This setting covers, in particular, constrained minimization scenarios, such as problems having smooth and convex functional constraints (e.g., quadratically constrained quadratic programs) or problems with a pseudo-convex objective function minimized over a simple closed convex set (e.g., quadratic over linear fractional programs). For the general problem, we design a forward-backward-forward splitting type method based on novel adaptive step-size strategies. Under an additional generalized Lipschitz property of the first operator, sublinear convergence rate is derived for the sequence generated by our adaptive algorithm. Moreover, if the sum is uniformly pseudo-monotone, linear/sublinear rates are derived depending on the parameter of uniform pseudo-monotonicity. Preliminary numerical experiments demonstrate the good performance of our method when compared with some existing optimization methods and software. Funding: The research leading to these results has received funding from project TraDE-OPT funded by the European Union’s Horizon 2020 Research and Innovation Programme under the Marie Skolodowska-Curie grant agreement [Grant 861137]; Unitatea Executiva pentru Finantarea Invatamantului Superior, a Cercetarii, Dezvoltarii si Inovarii, Romania [Grant PN-III-P4-PCE-2021-0720] under project L2O-MOC, nr. 70/2022; and GAR2023 funded by the Patrimony Foundation of Romanian Academy, nr. 260/2023.

Our proof of linear convergence for Dykstra’s algorithm was erroneous, and in fact, there even exists a counterexample showing that the result is false.

We study the worst-case behavior of Block Coordinate Descent (BCD) type algorithms for unconstrained minimization of coordinate-wise smooth convex functions. This behavior is indeed not completely understood, and the practical success of these algorithms is not fully explained by current convergence analyses. We extend the recently proposed Performance Estimation Problem (PEP) approach to convex coordinate-wise smooth functions by proposing necessary interpolation conditions. We then exploit this to obtain improved numerical upper bounds on the worst-case convergence rate of three different BCD algorithms, namely Cyclic Coordinate Descent (CCD), Alternating Minimization (AM), and a Cyclic version of the Random Accelerated Coordinate Descent introduced in Fercoq and Richtárik (2015) (CACD), substantially outperforming the best current bounds in some situations. In addition, we show the convergence of the CCD algorithm with more natural assumptions in the context of convex optimization than those typically made in the literature. Our methodology uncovers a number of phenomena, some of which can be formally established. These include a scale-invariance property of the worst case of CCD with respect to the coordinate-wise smoothness constants and a lower bound on the worst-case performance of CCD which is equal to the number of blocks times the worst-case of full gradient descent over the class of smooth convex functions. We also adapt our framework to the analysis of random BCD algorithms, and present numerical results showing that the standard acceleration scheme in Fercoq and Richtárik (2015) appears to be inefficient for deterministic algorithms.

This paper deals with composite optimization problems having the objective function formed as the sum of two terms; one has a Lipschitz continuous gradient along random subspaces and may be nonconvex, and the second term is simple and differentiable but possibly nonconvex and nonseparable. Under these settings, we design a stochastic coordinate proximal gradient method that takes into account the nonseparable composite form of the objective function. This algorithm achieves scalability by constructing at each iteration a local approximation model of the whole nonseparable objective function along a random subspace with user-determined dimension. We outline efficient techniques for selecting the random subspace, yielding an implementation that has low cost per iteration, also achieving fast convergence rates. We present a probabilistic worst case complexity analysis for our stochastic coordinate proximal gradient method in convex and nonconvex settings; in particular, we prove high-probability bounds on the number of iterations before a given optimality is achieved. Extensive numerical results also confirm the efficiency of our algorithm. Funding: This work was supported by Norway Grants 2014-2021 [Grant ELO-Hyp 24/2020]; Unitatea Executiva pentru Finantarea Invatamantului Superior, a Cercetarii, Dezvoltarii si Inovarii [Grants PN-III-P4-PCE-2021-0720, L2O-MOC, nr 70/2022]; and the ITN-ETN project TraDE-OPT funded by the European Union’s Horizon 2020 Research and Innovation Programme under the Marie Skłodowska-Curie grant agreement [Grant 861137].

Dimensionality reduction can be applied to hyperspectral images so that the most useful data can be extracted and processed more quickly. This is critical in any situation in which data volume exceeds the capacity of the computational resources, particularly in the case of remote sensing platforms (e.g., drones, satellites), but also in the case of multi-year datasets. Moreover, the computational strategies of unsupervised dimensionality reduction often provide the basis for more complicated supervised techniques. In this work, eight unsupervised dimensionality reduction algorithms are tested on hyperspectral data from the HYPSO-1 earth observation satellite. Each particular algorithm is chosen to be representative of a broader collection of methods. Our extensive experiments probe the computational complexity, reconstruction accuracy, signal clarity, sensitivity to artifacts, and effects on target detection and classification of the different algorithms. No algorithm consistently outperformed the others across all tests, but some general trends regarding the characteristics of the algorithms did emerge. With half a million pixels, computational time requirements of the methods varied by 5 orders of magnitude, and the reconstruction error varied by about 3 orders of magnitude. A relationship between mutual information and artifact susceptibility was suggested by the tests. The relative performance of the algorithms differed significantly between the target detection and classification tests. Overall, these experiments both show the power of dimensionality reduction and give guidance regarding how to evaluate a technique prior to incorporating it into a processing pipeline.

In this paper, we propose an adaptive forward-backward-forward splitting algorithm for finding a zero of a pseudo-monotone operator which is split as a sum of three operators: the first is continuous single-valued, the second is Lipschitzian, and the third is maximally monotone. This setting covers, in particular, constrained minimization scenarios, such as problems having smooth and convex functional constraints (e.g., quadratically constrained quadratic programs) or problems with a pseudo-convex objective function minimized over a simple closed convex set (e.g., quadratic over linear fractional programs). For the general problem, we design a forward-backward-forward splitting type method based on novel adaptive stepsize strategies. Under an additional generalized Lipschitz property of the first operator, sublinear convergence rate is derived for the sequence generated by our adaptive algorithm. Moreover, if the sum is uniformly pseudo-monotone, linear/sublinear rates are derived depending on the parameter of uniform pseudo-monotonicity. Preliminary numerical experiments demonstrate the good performance of our method when compared to some existing optimization methods and software.