Activation-Matched Gibbs Initialization for Neural Networks
preprint, 2026
Abstract
Initialization plays an important role in neural network training. A classical way to address this task relies on %Classical initialization methods often rely on Gaussian weights and preactivations, motivated in part by their analytical tractability and their emergence in large-width approximations. We propose an activation-induced initialization based on a statistical-mechanical viewpoint: for activations that arise as the gradient of a Legendre potential, the latter defines a Gibbs family of preactivation distributions. This perspective allows the preactivation distribution, rather than only its scale, to be adapted to the activation geometry.