Stiefel-AdamW: Geometry-Aware AdamW for Linear Factorization Blocks
A pervasive structural pattern in modern deep learning is the linear factorization block: a submodule of the form $W = BA$ in which two parameter matrices are multiplied directly, with no intervening nonlinearity. Such blocks appear in LoRA adapters, low-rank compressed layers, query-key products of self-attention, and …
dEBORA: Efficient Bilevel Optimization-based low-Rank Adaptation
International Conference on Learning Representations (ICLR), 2025
Low-rank adaptation methods are a popular approach for parameter-efficient fine-tuning of large-scale neural networks. However, selecting the optimal rank for each layer remains a challenging problem that significantly affects both performance and efficiency. In this paper, we introduce a novel bilevel optimization …
GeoLoRA: Geometric integration for parameter efficient fine-tuning
International Conference on Learning Representations (ICLR), 2025
Low-Rank Adaptation (LoRA) has become a widely used method for parameter-efficient fine-tuning of large-scale, pre-trained neural networks. However, LoRA and its extensions face several challenges, including the need for rank adaptivity, robustness, and computational efficiency during the fine-tuning process. We …
Low-rank lottery tickets: finding efficient low-rank neural networks via matrix differential equations
Advances in Neural Information Processing Systems (NeurIPS), 2022
Neural networks have achieved tremendous success in a large variety of applications. However, their memory footprint and computational demand can render them impractical in application settings with limited hardware or energy resources. In this work, we propose a novel algorithm to find efficient low-rank subnetworks. …