Michael Biehl

Using techniques borrowed from statistical physics, we study neural network systems at thermal equilibrium in a student-teacher model setting. The architecture studied here is the so-called Soft Committee Machine, a simple two-layered feedforward neural network with square integrable activation function, with respect to the standard Gaussian measure. Our focus is on understanding the effects that a mismatch in the number of internal units between the student and the teacher networks has in the system's behavior. We introduce two new ans\"atze for the saddle points of the under- and over-parameterized systems (supported by Monte Carlo simulations) and obtain the corresponding free energy and state equations for systems with any degree of over-parameterization and any square integrable activation in the high temperature limit. We show that the critical point grows with the degree of over-parameterization, while the asymptotic error decreases up until the number of internal units is the same as the teacher network's. For larger degrees of over-parameterizations the system always converges to a perfect generalization state. Moreover, we also show that, in an over-parameterized setting, out of all the activation functions analyzed here, the ReLU is the one with the highest decay rate for the order parameter that controls the baseline error created by the extra units.

This paper introduces Ordinal Generalized Matrix Learning Vector Quantization (ORGMLVQ), an enhanced version of the GMLVQ algorithm designed for classifying data with an inherent order among classes. ORGMLVQ incorporates ordinal constraints directly into the metric learning process, allowing the model to better capture the progression between categories-an important aspect in applications such as medical diagnostics or risk grading. Through experiments on multiple ordinal regression datasets, as well as standard UCI benchmarks and real-world problems, the proposed method demonstrates significant improvement of MAUC while maintaining the interpretability and prototype-based nature of the original GMLVQ. These results suggest that our method is a strong, interpretable alternative for learning from structured, ordered data.