Stiefel-AdamW: Geometry-Aware AdamW for Linear Factorization Blocks
A pervasive structural pattern in modern deep learning is the linear factorization block: a submodule of the form $W = BA$ in which two parameter matrices are multiplied directly, with no intervening nonlinearity. Such blocks appear in LoRA adapters, low-rank compressed layers, query-key products of self-attention, and …
Simplifying Transformer-Based U-Net Neural Physics Simulators
Advances in Neural Information Processing Systems (NeurIPS), 2026
U-Net-style architectures are widely adopted for modeling physical systems, as their multiscale structure enables efficient processing of high-resolution data while reflecting the hierarchical structure of many physical phenomena. The most successful U-Net-based neural physics simulators often combine convolutions, …
Low Rank Affine Adversarial Attacks on Image Classifiers
Springer INdAM Series, 2026
Neural networks for image classification are vulnerable to adversarial attacks; an imperceptible perturbation to an image can cause a change in classification. Standard attack algorithms use explicit or approximate partial derivative information with respect to the input data. Here, we explore the idea of using a less …
Neural-HSS: Hierarchical Semi-Separable Neural PDE Solver
International Conference on Machine Learning (ICML), 2026
Deep learning-based methods have shown remarkable effectiveness in solving PDEs, largely due to their ability to enable fast simulations once trained. However, despite the availability of high-performance computing infrastructure, many critical applications remain constrained by the substantial computational costs …
Ambient Physics: Training Neural PDE Solvers with Partial Observations
In many scientific contexts, obtaining complete observations of PDE coefficients and solutions proves expensive, hazardous, or unfeasible. Recent diffusion-based approaches can reconstruct fields from incomplete data, yet require fully observed examples during training. We present Ambient Physics, a framework enabling …
Low-Rank Compression of Language Models via Differentiable Rank Selection
Language Resources and Evaluation Conference (LREC), 2026
Approaches for compressing large-language models using low-rank decomposition have made strides, particularly with the introduction of activation and loss-aware SVD, which improves the trade-off between decomposition rank and downstream task performance. Despite these advancements, a persistent challenge …
Are we measuring oversmoothing in graph neural networks correctly?
International Conference on Learning Representations (ICLR), 2026
Oversmoothing is a fundamental challenge in graph neural networks (GNNs): as the number of layers increases, node embeddings become increasingly similar, and model performance drops sharply. Traditionally, oversmoothing has been quantified using metrics that measure the similarity of neighbouring node features, such as …
Algebraformer: A Neural Approach to Linear Systems
NeurIPS Workshop on Machine Learning and the Physical Sciences, 2025
Recent work in deep learning has opened new possibilities for solving classical algorithmic tasks using end-to-end learned models. In this work, we investigate the fundamental task of solving linear systems, particularly those that are ill-conditioned. Existing numerical methods for ill-conditioned systems often …
Stuart-Landau Oscillatory Graph Neural Network
The Web Conference (WWW), 2026
Oscillatory Graph Neural Networks (OGNNs) are an emerging class of physics-inspired architectures designed to mitigate oversmoothing and vanishing gradient problems in deep GNNs. In this work, we introduce the Complex-Valued Stuart-Landau Graph Neural Network (SLGNN), a novel architecture grounded in Stuart-Landau …
Test-Time Accuracy-Cost Control in Neural Simulators via Recurrent-Depth
International Conference on Learning Representations (ICLR), 2026
Accuracy-cost trade-offs are a fundamental aspect of scientific computing. Classical numerical methods inherently offer such a trade-off: increasing resolution, order, or precision typically yields more accurate solutions at higher computational cost. We introduce \textbf{Recurrent-Depth Simulator} (\textbf{RecurrSim}) …
Nonlinear Joint Spectral Radius
We introduce a nonlinear extension of the joint spectral radius (JSR) for switched discrete-time dynamical systems. The classical JSR characterizes the exponential growth rate of linear switched systems, but its extension to nonlinear systems has remained an open problem. Our approach builds on the framework of …
Graph-Convolutional-Beta-VAE for Synthetic Abdominal Aortic Aneurysm Generation
Medical & Biological Engineering & Computing, vol. 64, pp. 827–845, 2026
Synthetic data generation plays a crucial role in medical research by mitigating privacy concerns and enabling large-scale patient data analysis. This study presents a beta-Variational Autoencoder Graph Convolutional Neural Network framework for generating synthetic Abdominal Aorta Aneurysms (AAA). Using a small …
Approximation properties of neural ODEs
We study the approximation properties of neural ordinary differential equations (neural ODEs) in the space of continuous functions. Since a neural ODE requires input and output dimensions to be the same, while input and output dimensions of a continuous function are generally different, we need to embed an input into …
Improving the robustness of neural ODEs with minimal weight perturbation
We propose a method to enhance the stability of a neural ordinary differential equation (neural ODE) by reducing the maximum error growth subsequent to a perturbation of the initial value. Since the stability depends on the logarithmic norm of the Jacobian matrix associated with the neural ODE, we control the …
A Low-Rank Perspective on Oversmoothing in Graph Neural Networks
AAAI Conference on Artificial Intelligence (AAAI), Workshop on Connecting Low-Rank Representations in AI, 2025
Oversmoothing is a fundamental challenge in graph neural networks (GNNs): as the number of layers increases, node embeddings become increasingly similar, and model performance drops sharply. Traditionally, oversmoothing has been quantified using metrics that measure the similarity of neighbouring node features, such as …
dEBORA: Efficient Bilevel Optimization-based low-Rank Adaptation
International Conference on Learning Representations (ICLR), 2025
Low-rank adaptation methods are a popular approach for parameter-efficient fine-tuning of large-scale neural networks. However, selecting the optimal rank for each layer remains a challenging problem that significantly affects both performance and efficiency. In this paper, we introduce a novel bilevel optimization …
GeoLoRA: Geometric integration for parameter efficient fine-tuning
International Conference on Learning Representations (ICLR), 2025
Low-Rank Adaptation (LoRA) has become a widely used method for parameter-efficient fine-tuning of large-scale, pre-trained neural networks. However, LoRA and its extensions face several challenges, including the need for rank adaptivity, robustness, and computational efficiency during the fine-tuning process. We …
Low-Rank Adversarial PGD Attack
Adversarial attacks on deep neural network models have seen rapid development and are extensively used to study the stability of these networks. Among various adversarial strategies, Projected Gradient Descent (PGD) is a widely adopted method in computer vision due to its effectiveness and quick implementation, making …
Geometry-aware training of factorized layers in tensor Tucker format
Advances in Neural Information Processing Systems (NeurIPS), 2024
Reducing parameter redundancies in neural network architectures is crucial for achieving feasible computational and memory requirements during training and inference phases. Given its easy implementation and flexibility, one promising approach is layer factorization, which reshapes weight tensors into a matrix format …
Subhomogeneous Deep Equilibrium Models
International Conference on Machine Learning (ICML), 2024
Implicit-depth neural networks have grown as powerful alternatives to traditional networks in various applications in recent years. However, these models often lack guarantees of existence and uniqueness, raising stability, performance, and reproducibility issues. In this paper, we present a new analysis of the …
Solaris: A Foundation Model for the Sun
NeurIPS Workshop on Foundation Models for Science, 2024
Foundation models have demonstrated remarkable success across various scientific domains, motivating our exploration of their potential in solar physics. In this paper, we present Solaris, the first foundation model for forecasting the Sun’s atmosphere. We leverage 13 years of full-disk, multi-wavelength solar imagery …
Mixture of Neural Operators: Incorporating Historical Information for Longer Rollouts
ICLR Workshop on AI4DifferentialEquations in Science, 2024
Traditional numerical solvers for time-dependent partial differential equations (PDEs) notoriously require high computational resources and necessitate recomputation when faced with new problem parameters. In recent years, neural surrogates have shown great potential to overcome these limitations. However, it has been …
Contractivity of neural ODEs: an eigenvalue optimization problem
Mathematics of Computation, vol. 95, pp. 293–319, 2026
We propose a novel methodology to solve a key eigenvalue optimization problem which arises in the contractivity analysis of neural ODEs. When looking at contractivity properties of a one layer weight-tied neural ODE $\dot{u}(t)=σ(Au(t)+b)$ (with $u,b \in {\mathbb R}^n$, $A$ is a given $n \times n$ matrix, $σ: {\mathbb …
Provable Emergence of Deep Neural Collapse and Low-Rank Bias in $L^2$-Regularized Nonlinear Networks
Transactions on Machine Learning Research (TMLR), 2026
Recent work in deep learning has shown strong empirical and theoretical evidence of an implicit low-rank bias: weight matrices in deep networks tend to be approximately low-rank. Moreover, removing relatively small singular values during training, or from available trained models, may significantly reduce model size …
Learning the effective order of a hypergraph dynamical system
Science Advances, 2024
Dynamical systems on hypergraphs can display a rich set of behaviours not observable for systems with pairwise interactions. Given a distributed dynamical system with a putative hypergraph structure, an interesting question is thus how much of this hypergraph structure is actually necessary to faithfully replicate the …
Robust low-rank training via approximate orthonormal constraints
Advances in Neural Information Processing Systems (NeurIPS), 2023
With the growth of model and data sizes, a broad effort has been made to design pruning techniques that reduce the resource demand of deep learning pipelines, while retaining model performance. In order to reduce both inference and training costs, a prominent line of work uses low-rank matrix factorizations to …
Low-rank lottery tickets: finding efficient low-rank neural networks via matrix differential equations
Advances in Neural Information Processing Systems (NeurIPS), 2022
Neural networks have achieved tremendous success in a large variety of applications. However, their memory footprint and computational demand can render them impractical in application settings with limited hardware or energy resources. In this work, we propose a novel algorithm to find efficient low-rank subnetworks. …