Faster Adaptive Optimization via Expected Gradient Outer Product Reparameterization
Published in SIAM Journal on Mathematics of Data Science (under review), 2025
Keywords: adaptive optimization · Adam · Adagrad · low-rank preconditioning · EGOP · randomized SVD
My contribution. Led the design of the Auxiliary-Variable and Reduced EGOP algorithms. Used randomized SVD to extract the top-r eigenvectors of the EGOP matrix and introduced an orthogonal-complement auxiliary variable enabling lossless full-space optimization while storing only a small basis. Implemented the module end-to-end and benchmarked on Fashion-MNIST, EMNIST, CIFAR-10; on a 78,400-dim layer, retaining less than 1.3% of components (r=1000), and even r=50, captured the dominant geometry of the loss landscape and accelerated convergence.
Recommended citation: DePavia, A., Cruzado, J., Liang, J., Charisopoulos, V., & Willett, R. (2025). "Faster Adaptive Optimization via Expected Gradient Outer Product Reparameterization." Submitted to SIAM Journal on Mathematics of Data Science (SIMODS).
Download Paper