Low Rank Affine Adversarial Attacks on Image Classifiers
Springer INdAM Series, 2026
Neural networks for image classification are vulnerable to adversarial attacks; an imperceptible perturbation to an image can cause a change in classification. Standard attack algorithms use explicit or approximate partial derivative information with respect to the input data. Here, we explore the idea of using a less …
Low-Rank Compression of Language Models via Differentiable Rank Selection
Language Resources and Evaluation Conference (LREC), 2026
Approaches for compressing large-language models using low-rank decomposition have made strides, particularly with the introduction of activation and loss-aware SVD, which improves the trade-off between decomposition rank and downstream task performance. Despite these advancements, a persistent challenge …
Are we measuring oversmoothing in graph neural networks correctly?
International Conference on Learning Representations (ICLR), 2026
Oversmoothing is a fundamental challenge in graph neural networks (GNNs): as the number of layers increases, node embeddings become increasingly similar, and model performance drops sharply. Traditionally, oversmoothing has been quantified using metrics that measure the similarity of neighbouring node features, such as …
A Low-Rank Perspective on Oversmoothing in Graph Neural Networks
AAAI Conference on Artificial Intelligence (AAAI), Workshop on Connecting Low-Rank Representations in AI, 2025
Oversmoothing is a fundamental challenge in graph neural networks (GNNs): as the number of layers increases, node embeddings become increasingly similar, and model performance drops sharply. Traditionally, oversmoothing has been quantified using metrics that measure the similarity of neighbouring node features, such as …
dEBORA: Efficient Bilevel Optimization-based low-Rank Adaptation
International Conference on Learning Representations (ICLR), 2025
Low-rank adaptation methods are a popular approach for parameter-efficient fine-tuning of large-scale neural networks. However, selecting the optimal rank for each layer remains a challenging problem that significantly affects both performance and efficiency. In this paper, we introduce a novel bilevel optimization …
GeoLoRA: Geometric integration for parameter efficient fine-tuning
International Conference on Learning Representations (ICLR), 2025
Low-Rank Adaptation (LoRA) has become a widely used method for parameter-efficient fine-tuning of large-scale, pre-trained neural networks. However, LoRA and its extensions face several challenges, including the need for rank adaptivity, robustness, and computational efficiency during the fine-tuning process. We …
Low-Rank Adversarial PGD Attack
Adversarial attacks on deep neural network models have seen rapid development and are extensively used to study the stability of these networks. Among various adversarial strategies, Projected Gradient Descent (PGD) is a widely adopted method in computer vision due to its effectiveness and quick implementation, making …
Geometry-aware training of factorized layers in tensor Tucker format
Advances in Neural Information Processing Systems (NeurIPS), 2024
Reducing parameter redundancies in neural network architectures is crucial for achieving feasible computational and memory requirements during training and inference phases. Given its easy implementation and flexibility, one promising approach is layer factorization, which reshapes weight tensors into a matrix format …
Provable Emergence of Deep Neural Collapse and Low-Rank Bias in $L^2$-Regularized Nonlinear Networks
Transactions on Machine Learning Research (TMLR), 2026
Recent work in deep learning has shown strong empirical and theoretical evidence of an implicit low-rank bias: weight matrices in deep networks tend to be approximately low-rank. Moreover, removing relatively small singular values during training, or from available trained models, may significantly reduce model size …
Robust low-rank training via approximate orthonormal constraints
Advances in Neural Information Processing Systems (NeurIPS), 2023
With the growth of model and data sizes, a broad effort has been made to design pruning techniques that reduce the resource demand of deep learning pipelines, while retaining model performance. In order to reduce both inference and training costs, a prominent line of work uses low-rank matrix factorizations to …
Low-rank lottery tickets: finding efficient low-rank neural networks via matrix differential equations
Advances in Neural Information Processing Systems (NeurIPS), 2022
Neural networks have achieved tremendous success in a large variety of applications. However, their memory footprint and computational demand can render them impractical in application settings with limited hardware or energy resources. In this work, we propose a novel algorithm to find efficient low-rank subnetworks. …