Stiefel-AdamW: Geometry-Aware AdamW for Linear Factorization Blocks

Emanuele Zangrando, Marco Sutti, Francesco Tudisco

A pervasive structural pattern in modern deep learning is the linear factorization block: a submodule of the form $W = BA$ in which two parameter matrices are multiplied directly, with no intervening nonlinearity. Such blocks appear in LoRA adapters, low-rank compressed layers, query-key products of self-attention, and …

Arxiv·Cite