Generalization Bounds for Nonlinear Least Squares via Learned Feature Geometry

A preprint posted to arXiv introduces generalization bounds for ridge-regularized nonlinear least-squares models derived through on-average algorithmic stability, a framework that measures how sensitive a learning algorithm's output is to small changes in training data. Central to the analysis is a data-dependent effective dimension computed from the empirical Jacobian Gram matrix at the trained parameters, augmented by a residual-curvature term that captures nonlinearity. In the linear special case, the curvature term vanishes and the result recovers classical effective dimension, but crucially evaluated post-training rather than at initialization as in standard neural tangent kernel analyses. The effective dimension is further bounded using covering complexity of gradient features, yielding guarantees that scale with intrinsic data dimensionality for manifold-supported data and piecewise Lipschitz Jacobians. For one-hidden-layer ReLU networks specifically, the mechanism is made explicit through counts of activation-stable regions. Experiments on synthetic manifolds, clustered distributions, and benchmark datasets support the theoretical predictions, showing trained-Jacobian compression and agreement between the stability bound and observed generalization gaps. The derivation relies on the Brascamp-Lieb inequality under strongly log-concave noise, which the authors highlight as a notably simple and principled proof technique.