Title:Extended Regularized Dual Averaging

Abstract:We present a new class of algorithms for solving regularized optimization and saddle point problems. We derive these new methods as an extension of regularized dual averaging and analyze their convergence for convex problems. The new theoretical idea is a more general bound on the objective error and duality gap, which leads to a new class of primal dual subgradient methods. In addition, we argue that this new class of methods should be effective for non-convex problems with regularization, in particular regularized neural network training, by drawing comparisons with existing methods for training quantized neural networks and citing recent numerical calculations. To conclude, we exhibit a variety of special cases and compare them with existing methods.