On the Robustness of Kernel Ridge Regression Using the Cauchy Loss Function
Preprint
Abstract
Robust regression aims to develop methods for estimating an unknown regression function in the presence of outliers, heavy-tailed distributions, or contaminated data, which can severely impact performance. Most existing theoretical results in robust regression assume that the noise has a finite absolute mean, an assumption violated by certain distributions, such as Cauchy and some Pareto noise. In this paper, we introduce a generalized Cauchy noise framework that accommodates all noise distributions with finite moments of any order, even when the absolute mean is infinite. Within this framework, we study the \textit{kernel Cauchy ridge regressor} (\textit{KCRR}), which minimizes a regularized empirical Cauchy risk to achieve robustness. To derive the $L_2$-risk bound for KCRR, we establish a connection between the excess Cauchy risk and $L_2$-risk for sufficiently large scale parameters of the Cauchy loss, which reveals that these two risks are equivalent. Furthermore, under the assumption that the regression function satisfies Hölder smoothness, we derive excess Cauchy risk bounds for KCRR, showing improved performance as the scale parameter decreases. By considering the twofold effect of the scale parameter on the excess Cauchy risk and its equivalence with the $L_2$-risk, we establish the almost minimax-optimal convergence rate for KCRR in terms of $L_2$-risk, highlighting the robustness of the Cauchy loss in handling various types of noise. Finally, we validate the effectiveness of KCRR through experiments on both synthetic and real-world datasets under diverse noise corruption scenarios.Keywords
All Related Versions
This publication has 15 references indexed in Scilit:
- Minimax rates of convergence for nonparametric location-Scale modelsJournal of Econometrics, 2026
- Massive MIMO With Cauchy Noise: Channel Estimation, Achievable Rate and Data DecodingIEEE Transactions on Wireless Communications, 2023
- Robust Regression Analysis in Analyzing Financial Performance of Public Sector Banks: A Case Study of IndiaAnnals of Data Science, 2022
- Measurements and statistical analyses of electromagnetic noise for industrial wireless communicationsInternational Journal of Intelligent Systems, 2020
- Cauchy noise removal by nonlinear diffusion equationsComputers & Mathematics with Applications, 2020
- Nonlocal Myriad Filters for Cauchy Noise RemovalJournal of Mathematical Imaging and Vision, 2018
- Convergence of the Huber Regression M-Estimate in the Presence of Dense OutliersIEEE Signal Processing Letters, 2014
- Least Absolute Deviation Support Vector RegressionMathematical Problems in Engineering, 2014
- Cauchy Estimation for Linear Scalar SystemsIEEE Transactions on Automatic Control, 2010
- A Distribution-Free Theory of Nonparametric Regression2002