2026
Low-Rank Compression of Language Models via Differentiable Rank Selection
Language Resources and Evaluation Conference (LREC), 2026
Approaches for compressing large-language models using low-rank decomposition have made strides, particularly with the introduction of activation and loss-aware SVD, which improves the trade-off between decomposition rank and downstream task performance. Despite these advancements, a persistent challenge …
2024
Geometry-aware training of factorized layers in tensor Tucker format
Advances in Neural Information Processing Systems (NeurIPS), 2024
Reducing parameter redundancies in neural network architectures is crucial for achieving feasible computational and memory requirements during training and inference phases. Given its easy implementation and flexibility, one promising approach is layer factorization, which reshapes weight tensors into a matrix format …
2023
Robust low-rank training via approximate orthonormal constraints
Advances in Neural Information Processing Systems (NeurIPS), 2023
With the growth of model and data sizes, a broad effort has been made to design pruning techniques that reduce the resource demand of deep learning pipelines, while retaining model performance. In order to reduce both inference and training costs, a prominent line of work uses low-rank matrix factorizations to …
2022
Low-rank lottery tickets: finding efficient low-rank neural networks via matrix differential equations
Advances in Neural Information Processing Systems (NeurIPS), 2022
Neural networks have achieved tremendous success in a large variety of applications. However, their memory footprint and computational demand can render them impractical in application settings with limited hardware or energy resources. In this work, we propose a novel algorithm to find efficient low-rank subnetworks. …