Title:A Bridge Between Past and Present: Exchange and Conditional Gradient Methods are Equivalent

Abstract:We study an optimization program over nonnegative Borel measures that encourages sparsity in its solution. Efficient solvers for this program are increasingly needed, as it arises when learning from data generated by a "continuum-of-subspaces" model, a recent trend with applications in signal processing, machine learning, and statistics. We prove that the conditional gradient method (CGM) applied to this infinite-dimensional program, as proposed recently in the literature, is equivalent to the exchange method (EM) applied to its Lagrangian dual, which is a semi-infinite program. In doing so, we formally connect such infinite-dimensional programs to the well-established field of semi-infinite programming. On the one hand, the equivalence established in this paper allows us to provide a rate of convergence for EM which is more general than those existing in the literature. On the other hand, this connection and the resulting geometric insights might in the future lead to the design of improved variants of CGM for infinite-dimensional programs, which has been an active research topic. CGM is also known as the Frank-Wolfe algorithm.