Low-Rank Compression of Language Models via Differentiable Rank Selection
Language Resources and Evaluation Conference (LREC), 2026
Approaches for compressing large-language models using low-rank decomposition have made strides, particularly with the introduction of activation and loss-aware SVD, which improves the trade-off between decomposition rank and downstream task performance. Despite these advancements, a persistent challenge …
dEBORA: Efficient Bilevel Optimization-based low-Rank Adaptation
International Conference on Learning Representations (ICLR), 2025
Low-rank adaptation methods are a popular approach for parameter-efficient fine-tuning of large-scale neural networks. However, selecting the optimal rank for each layer remains a challenging problem that significantly affects both performance and efficiency. In this paper, we introduce a novel bilevel optimization …
GeoLoRA: Geometric integration for parameter efficient fine-tuning
International Conference on Learning Representations (ICLR), 2025
Low-Rank Adaptation (LoRA) has become a widely used method for parameter-efficient fine-tuning of large-scale, pre-trained neural networks. However, LoRA and its extensions face several challenges, including the need for rank adaptivity, robustness, and computational efficiency during the fine-tuning process. We …
Geometry-aware training of factorized layers in tensor Tucker format
Advances in Neural Information Processing Systems (NeurIPS), 2024
Reducing parameter redundancies in neural network architectures is crucial for achieving feasible computational and memory requirements during training and inference phases. Given its easy implementation and flexibility, one promising approach is layer factorization, which reshapes weight tensors into a matrix format …
Provable Emergence of Deep Neural Collapse and Low-Rank Bias in $L^2$-Regularized Nonlinear Networks
Transactions on Machine Learning Research (TMLR), 2026
Recent work in deep learning has shown strong empirical and theoretical evidence of an implicit low-rank bias: weight matrices in deep networks tend to be approximately low-rank. Moreover, removing relatively small singular values during training, or from available trained models, may significantly reduce model size …
Robust low-rank training via approximate orthonormal constraints
Advances in Neural Information Processing Systems (NeurIPS), 2023
With the growth of model and data sizes, a broad effort has been made to design pruning techniques that reduce the resource demand of deep learning pipelines, while retaining model performance. In order to reduce both inference and training costs, a prominent line of work uses low-rank matrix factorizations to …
Low-rank lottery tickets: finding efficient low-rank neural networks via matrix differential equations
Advances in Neural Information Processing Systems (NeurIPS), 2022
Neural networks have achieved tremendous success in a large variety of applications. However, their memory footprint and computational demand can render them impractical in application settings with limited hardware or energy resources. In this work, we propose a novel algorithm to find efficient low-rank subnetworks. …