Joost A. A. Opschoor

We prove deep neural network (DNN for short) expressivity rate bounds for solution sets of a model class of singularly perturbed, elliptic two-point boundary value problems, in Sobolev norms, on the bounded interval $$(-1,1)$$ ( - 1 , 1 ) . We assume that the given source term and reaction coefficient are analytic in $$[-1,1]$$ [ - 1 , 1 ] . The expression rate bounds in Sobolev norms in terms of the NN size are robust, i.e. uniform with respect to the singular perturbation parameter $$\varepsilon \in (0,1]$$ ε ∈ ( 0 , 1 ] for several classes of DNN architectures. In particular, ReLU NNs, spiking NNs, and $$\tanh $$ tanh - and sigmoid-activated NNs. The latter activations can represent “exponential boundary layer solution features” explicitly, in the last hidden layer of the DNN, i.e. in a shallow subnetwork, and afford improved robust expression rate bounds in terms of the NN size. All DNN architectures allow robust exponential solution expression in so-called ‘energy’ as well as in ‘balanced’ Sobolev norms, for analytic input data.

We prove deep neural network (DNN for short) expressivity rate bounds for solution sets of a model class of singularly perturbed, elliptic two-point boundary value problems, in Sobolev norms, on the bounded interval $(-1,1)$. We assume that the given source term and reaction coefficient are analytic in $[-1,1]$. We establish expression rate bounds in Sobolev norms in terms of the NN size which are uniform with respect to the singular perturbation parameter for several classes of DNN architectures. In particular, ReLU NNs, spiking NNs, and $\tanh$- and sigmoid-activated NNs. The latter activations can represent ``exponential boundary layer solution features'' explicitly, in the last hidden layer of the DNN, i.e. in a shallow subnetwork, and afford improved robust expression rate bounds in terms of the NN size. We prove that all DNN architectures allow robust exponential solution expression in so-called `energy' as well as in `balanced' Sobolev norms, for analytic input data.

On general regular simplicial partitions $\mathcal{T}$ of bounded polytopal domains $\Omega \subset \mathbb{R}^d$, $d\in\{2,3\}$, we construct \emph{exact neural network (NN) emulations} of all lowest order finite element spaces in the discrete de Rham complex. These include the spaces of piecewise constant functions, continuous piecewise linear (CPwL) functions, the classical ``Raviart-Thomas element'', and the ``N\'{e}d\'{e}lec edge element''. For all but the CPwL case, our network architectures employ both ReLU (rectified linear unit) and BiSU (binary step unit) activations to capture discontinuities. In the important case of CPwL functions, we prove that it suffices to work with pure ReLU nets. Our construction and DNN architecture generalizes previous results in that no geometric restrictions on the regular simplicial partitions $\mathcal{T}$ of $\Omega$ are required for DNN emulation. In addition, for CPwL functions our DNN construction is valid in any dimension $d\geq 2$. Our ``FE-Nets'' are required in the variationally correct, structure-preserving approximation of boundary value problems of electromagnetism in nonconvex polyhedra $\Omega \subset \mathbb{R}^3$. They are thus an essential ingredient in the application of e.g., the methodology of ``physics-informed NNs'' or ``deep Ritz methods'' to electromagnetic field simulation via deep learning techniques. We indicate generalizations of our constructions to higher-order compatible spaces and other, non-compatible classes of discretizations, in particular the ``Crouzeix-Raviart'' elements and Hybridized, Higher Order (HHO) methods.

In certain polytopal domains $$\varOmega $$Ω , in space dimension $$d=2,3$$d = 2 , 3 , we prove exponential expressivity with stable ReLU Neural Networks (ReLU NNs) in $$H^1(\varOmega )$$H 1 ( Ω ) for weighted analytic function classes. These classes comprise in particular solution sets of source and eigenvalue problems for elliptic PDEs with analytic data. Functions in these classes are locally analytic on open subdomains $$D\subset \varOmega $$D ⊂ Ω , but may exhibit isolated point singularities in the interior of $$\varOmega $$Ω or corner and edge singularities at the boundary $$\partial \varOmega $$∂ Ω . The exponential approximation rates are shown to hold in space dimension $$d = 2$$d = 2 on Lipschitz polygons with straight sides, and in space dimension $$d=3$$d = 3 on Fichera-type polyhedral domains with plane faces. The constructive proofs indicate that NN depth and size increase poly-logarithmically with respect to the target NN approximation accuracy $$\varepsilon >0$$ε > 0 in $$H^1(\varOmega )$$H 1 ( Ω ) . The results cover solution sets of linear, second-order elliptic PDEs with analytic data and certain nonlinear elliptic eigenvalue problems with analytic nonlinearities and singular, weighted analytic potentials as arise in electron structure models. Here, the functions correspond to electron densities that exhibit isolated point singularities at the nuclei.

For a parameter dimension $$d\in {\mathbb {N}}$$d ∈ N , we consider the approximation of many-parametric maps $$u: [-\,1,1]^d\rightarrow {\mathbb R}$$u : [ - 1 , 1 ] d → R by deep ReLU neural networks. The input dimension d may possibly be large, and we assume quantitative control of the domain of holomorphy of u: i.e., u admits a holomorphic extension to a Bernstein polyellipse $${{\mathcal {E}}}_{\rho _1}\times \cdots \times {{\mathcal {E}}}_{\rho _d} \subset {\mathbb {C}}^d$$E ρ 1 × ⋯ × E ρ d ⊂ C d of semiaxis sums $$\rho _i>1$$ρ i > 1 containing $$[-\,1,1]^{d}$$[ - 1 , 1 ] d . We establish the exponential rate $$O(\exp (-\,bN^{1/(d+1)}))$$O ( exp ( - b N 1 / ( d + 1 ) ) ) of expressive power in terms of the total NN size N and of the input dimension d of the ReLU NN in $$W^{1,\infty }([-\,1,1]^d)$$W 1 , ∞ ( [ - 1 , 1 ] d ) . The constant $$b>0$$b > 0 depends on $$(\rho _j)_{j=1}^d$$( ρ j ) j = 1 d which characterizes the coordinate-wise sizes of the Bernstein-ellipses for u. We also prove exponential convergence in stronger norms for the approximation by DNNs with more regular, so-called “rectified power unit” activations. Finally, we extend DNN expression rate bounds also to two classes of non-holomorphic functions, in particular to d-variate, Gevrey-regular functions, and, by composition, to certain multivariate probability distribution functions with Lipschitz marginals.