Title:Separation of Scales and a Thermodynamic Description of Feature Learning in Some CNNs

Abstract:Deep neural networks (DNNs) are powerful tools for compressing and distilling information. Their scale and complexity, often involving billions of inter-dependent internal degrees of freedom, renders direct microscopic analysis difficult. Under such circumstances, a common strategy is to identify slow degrees of freedom that average out the erratic behavior of the underlying fast microscopic variables. Here, we identify such a separation of scales occurring in fully trained over-parameterized deep convolutional neural networks (CNNs). Specifically, we show that DNN layers couple only through the second moment (kernels) of their activations and pre-activations. Moreover, in various settings, the latter fluctuate in a nearly Gaussian manner. For CNNs with infinitely many channels, these kernels are inert, while for finite CNNs they adapt to the data. In several deep non-linear CNN models trained on real data, the resulting thermodynamic theory of deep learning yields accurate predictions. In addition, it provides new ways of analyzing and understanding CNNs, and DNNs in general.