An Accelerated Gradient Method For Trace Norm Minimization

Abstract

We consider the minimization of a smooth loss function regularized by the trace norm of the matrix variable. Such formulation finds applications in many machine learning tasks including multi-task learning, matrix classification, and matrix completion. The standard semidefinite programming formulation for this problem is computationally expensive. In addition, due to the non-smooth nature of the trace norm, the optimal first-order black-box method for solving such class of problems converges as <i>O</i>(1/√<i>k</i>), where <i>k</i> is the iteration counter. In this paper, we exploit the special structure of the trace norm, based on which we propose an extended gradient algorithm that converges as <i>O</i>(1/<i>k</i>). We further propose an accelerated gradient algorithm, which achieves the optimal convergence rate of <i>O</i>(1/<i>k</i>²) for smooth problems. Experiments on multi-task learning problems demonstrate the efficiency of the proposed algorithms.