Deep operator network approximation rates for Lipschitz operators

Authors:
Top Cited Papers
Publisher website
Google Scholar
Abstract
We establish a framework for universality and expression rate bounds for a class of neural Deep Operator Networks (DONs) emulating Lipschitz (or Hölder) continuous maps [ ] between (subsets of) separable Hilbert spaces [ ], [ ]. The DON architecture considered uses linear encoders [ ] and decoders [ ] via (biorthogonal) Riesz bases of [ ], [ ], and an approximator network of an infinite-dimensional, parametric coordinate map that is Lipschitz continuous on the sequence space [ ]. Unlike previous works [L. Herrmann, C. Schwab and J. Zech, Neural and spectral operator surrogates: Construction and expression rate bounds, Adv. Comput. Math. 50(4) (2024) 72; C. Marcati and C. Schwab, Exponential convergence of deep operator networks for elliptic partial differential equations, SIAM J. Numer. Anal. 61(3) (2023) 1513–1545] which required for example [ ] to be holomorphic, the present expression rate results require mere Lipschitz (or Hölder) continuity of [ ]. Key in the proof of the present expression rate bounds is the use of either superexpressive activations (e.g., [Z. Shen, H. Yang and S. Zhang, Neural network approximation: Three hidden layers are enough, Neural Netw. 141 (2021) 160–173; Z. Shen, H. Yang and S. Zhang, Deep network approximation: Achieving arbitrary accuracy with fixed number of neurons, J. Mach. Learn. Res. 23(276) (2022) 1–60; D. Yarotsky, Elementary superexpressive activations, in Proc. 38th Int. Conf. Machine Learning (PMLR, 2021), pp. 11932–11940] and the references there) which are inspired by the Kolmogorov superposition theorem ([A. N. Kolmogorov, On the representation of continuous functions of many variables by superposition of continuous functions of one variable and addition, Dokl. Akad. Nauk SSSR 114 (1957) 953–956] or [G. G. Lorentz, Approximation of Functions (Holt, Rinehart and Winston, New York–Chicago, IL–Toronto, ON, 1966), Chap. 11] for a comprehensive exposition), or of nonstandard NN architectures with standard (ReLU) activations as recently proposed in [Z. Shen, H. Yang and S. Zhang, Deep network approximation: Achieving arbitrary accuracy with fixed number of neurons, J. Mach. Learn. Res. 23(276) (2022) 1–60; S. Zhang, Z. Shen and H. Yang, Neural network architecture beyond width and depth, in Advances in Neural Information Processing Systems, Vol. 35 (Curran Associates, 2022), pp. 5669–5681]. We illustrate the abstract results by approximation rate bounds for emulation of (a) solution operators for parametric elliptic variational inequalities and (b) Lipschitz maps of Hilbert–Schmidt operators.
Funding Information
  • EPSRC (EP/R014604/1)

This publication has 51 references indexed in Scilit: