Tyler J. Jarvis

A central problem in data science is to use potentially noisy samples of an unknown function to predict function values for unseen inputs. In classical statistics, the predictive error is understood as a trade-off between the bias and the variance that balances model simplicity with its ability to fit complex functions. However, overparametrized models exhibit counterintuitive behaviors, such as “double descent” in which models of increasing complexity exhibit generalization error. Other models may exhibit more complicated patterns of predictive error with multiple peaks and valleys. Neither double descent nor multiple descent phenomena are well explained by the bias-variance decomposition. We introduce a decomposition that we call the generalized aliasing decomposition (GAD) to explain the relationship between predictive performance and model complexity. The GAD decomposes the predictive error into three parts: (1) , which dominates when the number of parameters is much smaller than the number of data points, (2) , which dominates when the number of parameters is much greater than the number of data points, and (3) , which dominates between these two extremes. We demonstrate the applicability of the GAD to diverse applications, including random feature models from machine learning, Fourier transforms from signal processing, solution methods for differential equations, and predictive formation enthalpy in materials discovery. Because key components of the generalized aliasing decomposition can be explicitly calculated from the relationship between model class and samples without seeing any data labels, it can answer questions related to experimental design and model selection collecting data or performing experiments. We further demonstrate this approach on several examples and discuss implications for predictive modeling and data science.

Calculations of properties of materials require performing numerical integrals over the Brillouin zone (BZ). Integration points in density functional theory codes are uniformly spread over the BZ (despite integration error being concentrated in small regions of the BZ) and preserve symmetry to improve computational efficiency. Integration points over an irreducible Brillouin zone (IBZ), a rotationally distinct region of the BZ, do not have to preserve crystal symmetry for greater efficiency. This freedom allows the use of adaptive meshes with higher concentrations of points at locations of large error, resulting in improved algorithmic efficiency. We have created an algorithm for constructing an IBZ of any crystal structure in 2D and 3D. The algorithm uses convex hull and half-space representations for the BZ and IBZ to make many aspects of construction and symmetry reduction of the BZ trivial. The algorithm is simple, general, and available as open-source software. Brillouin zone, irreducible Brillouin zone. 68U05, 20H15, 52B55, 52C07, 68W40, 57Z05, 57Z15 COPYRIGHT: © Global Science Press

Due to the success of residual networks (resnets) and related architectures, shortcut connections have quickly become standard tools for building convolutional neural networks. The explanations in the literature for the apparent effectiveness of shortcuts are varied and often contradictory. We hypothesize that shortcuts work primarily because they act as linear counterparts to nonlinear layers. We test this hypothesis by using several variations on the standard residual block, with different types of linear connections, to build small image classification networks. Our experiments show that other kinds of linear connections can be even more effective than the identity shortcuts. Our results also suggest that the best type of linear connection for a given application may depend on both network width and depth.

We present a general framework for training deep neural networks without backpropagation. This substantially decreases training time and also allows for construction of deep networks with many sorts of learners, including networks whose layers are defined by functions that are not easily differentiated, like decision trees. The main idea is that layers can be trained one at a time, and once they are trained, the input data are mapped forward through the layer to create a new learning problem. The process is repeated, transforming the data through multiple layers, one at a time, rendering a new data set, which is expected to be better behaved, and on which a final output layer can achieve good performance. We call this forward thinking and demonstrate a proof of concept by achieving state-of-the-art accuracy on the MNIST dataset for convolutional neural networks. We also provide a general mathematical formulation of forward thinking that allows for other types of deep learning problems to be considered.

The success of deep neural networks has inspired many to wonder whether other learners could benefit from deep, layered architectures. We present a general framework called forward thinking for deep learning that generalizes the architectural flexibility and sophistication of deep neural networks while also allowing for (i) different types of learning functions in the network, other than neurons, and (ii) the ability to adaptively deepen the network as needed to improve results. This is done by training one layer at a time, and once a layer is trained, the input data are mapped forward through the layer to create a new learning problem. The process is then repeated, transforming the data through multiple layers, one at a time, rendering a new dataset, which is expected to be better behaved, and on which a final output layer can achieve good performance. In the case where the neurons of deep neural nets are replaced with decision trees, we call the result a Forward Thinking Deep Random Forest (FTDRF). We demonstrate a proof of concept by applying FTDRF on the MNIST dataset. We also provide a general mathematical formulation that allows for other types of deep learning problems to be considered.