Title:The limitations of gradient descent as a principle of brain function

Abstract:The idea that the brain functions in a way that minimizes certain costs pervades theoretical neuro- science. Since a cost function by itself does not predict how the brain finds its minima, additional assumptions about the optimization method are usually made in order to obtain predictions about physiological quantities. In this context, steepest ascent or descent is often suggested as a biolog- ically implemented optimization algorithm. However, the notion of a steepest direction depends on the choice of a Riemannian metric. Since this choice involves infinite degrees of freedom, the predictive power of models that are based on this principle can be called into question. Despite its great success in machine learning, especially in training deep neural networks, arguments about brain function that rely on gradient-based optimization have to be closely examined.