Understanding and Mitigating Gradient Flow Pathologies in Physics-Informed Neural Networks
Понимание и устранение патологий потока градиентов в физически информированных нейронных сетях
2021-01-01
SCID: 54.1/xvs5vh38
Discuss with AI
gradient flow pathologieslearning rate annealingnumerical stiffnesspartial differential equationsphysics-informed neural networks
Figures from the paper
Abstract (AI)
The widespread use of neural networks across different scientific domains often involves constraining them to satisfy certain symmetries, conservation laws, or other domain knowledge. Such constraints are often imposed as soft penalties during model training and effectively act as domain-specific regularizers of the empirical risk loss. Physics-informed neural networks is an example of this philosophy in which the outputs of deep neural networks are constrained to approximately satisfy a given set of partial differential equations. In this work we review recent advances in scientific machine learning with a specific focus on the effectiveness of physics-informed neural networks in predicting outcomes of physical systems and discovering hidden physics from noisy data. We also identify and analyze a fundamental mode of failure of such approaches that is related to numerical stiffness leading to unbalanced back-propagated gradients during model training. To address this limitation we present a learning rate annealing algorithm that utilizes gradient statistics during model training to balance the interplay between different terms in composite loss functions. We also propose a novel neural network architecture that is more resilient to such gradient pathologies. Taken together, our developments provide new insights into the training of constrained neural networks and consistently improve the predictive accuracy of physics-informed neural networks by a factor of 50--100$\times$ across a range of problems in computational physics. All code and data accompanying this manuscript are publicly available at https://github.com/PredictiveIntelligenceLab/GradientPathologiesPINNs.
Key Findings
1
A gradient-statistics-based learning-rate annealing algorithm is introduced to balance competing terms in composite PINN loss functions.
2
A novel neural-network architecture is proposed that is more resilient to gradient-flow pathologies.
3
Physics-informed neural networks can suffer a fundamental failure mode caused by numerical stiffness and unbalanced back-propagated gradients during training.
4
The combined developments consistently improve PINN predictive accuracy by 50–100× across a range of computational-physics problems.
5
The work reviews PINN effectiveness for physical-system prediction and hidden-physics discovery from noisy data, with accompanying code and data publicly released.
Research Object
physics-informed neural networks for predicting physical systems and discovering hidden physics from noisy data
Research Subject
gradient-flow pathologies caused by numerical stiffness, including unbalanced back-propagated gradients during training, and their mitigation to improve predictive accuracy
Publication Details
Publication Date
2021-01-01
Journal
Publisher
ISSN
Cited by
3421
Open access PDF
Access Type
Author Information
Download PDF
Subscribe to digest
References available in scid.ai3
Cited by8
Scientific Machine Learning Through Physics–Informed Neural Networks: Where we are and What’s Next2022
Physics-Informed Neural Networks for Heat Transfer Problems2021
Interpretable machine learning: Fundamental principles and 10 grand challenges2022
A-PINN: Auxiliary physics informed neural networks for forward and inverse problems of nonlinear integro-differential equations2022
Physics‐Informed Neural Networks of the Saint‐Venant Equations for Downscaling a Large‐Scale River Model2023
Physics-informed machine learning in geotechnical engineering: a direction paper2025
Physics-Informed Neural Network for Reconstructing Free-Surface Transient Flow Fields in Long-Distance Water-Conveyance Tunnels from Sparse Observations2026
APPLICATION OF PHYSICS-INFORMED NEURAL NETWORKS (PINN) COUPLED WITH ONE-DIMENSIONAL UNSTEADY FLOW SIMULATION TO OPEN CHANNEL FLOW2025