Understanding and Mitigating Gradient Flow Pathologies in Physics-Informed Neural Networks

Понимание и устранение патологий потока градиентов в физически информированных нейронных сетях
Paris G. Perdikaris, Sifan Wang, Yujun Teng
2021-01-01

gradient flow pathologieslearning rate annealingnumerical stiffnesspartial differential equationsphysics-informed neural networks
The widespread use of neural networks across different scientific domains often involves constraining them to satisfy certain symmetries, conservation laws, or other domain knowledge. Such constraints are often imposed as soft penalties during model training and effectively act as domain-specific regularizers of the empirical risk loss. Physics-informed neural networks is an example of this philosophy in which the outputs of deep neural networks are constrained to approximately satisfy a given set of partial differential equations. In this work we review recent advances in scientific machine learning with a specific focus on the effectiveness of physics-informed neural networks in predicting outcomes of physical systems and discovering hidden physics from noisy data. We also identify and analyze a fundamental mode of failure of such approaches that is related to numerical stiffness leading to unbalanced back-propagated gradients during model training. To address this limitation we present a learning rate annealing algorithm that utilizes gradient statistics during model training to balance the interplay between different terms in composite loss functions. We also propose a novel neural network architecture that is more resilient to such gradient pathologies. Taken together, our developments provide new insights into the training of constrained neural networks and consistently improve the predictive accuracy of physics-informed neural networks by a factor of 50--100$\times$ across a range of problems in computational physics. All code and data accompanying this manuscript are publicly available at https://github.com/PredictiveIntelligenceLab/GradientPathologiesPINNs.
1
A gradient-statistics-based learning-rate annealing algorithm is introduced to balance competing terms in composite PINN loss functions.
2
A novel neural-network architecture is proposed that is more resilient to gradient-flow pathologies.
3
Physics-informed neural networks can suffer a fundamental failure mode caused by numerical stiffness and unbalanced back-propagated gradients during training.
4
The combined developments consistently improve PINN predictive accuracy by 50–100× across a range of computational-physics problems.
5
The work reviews PINN effectiveness for physical-system prediction and hidden-physics discovery from noisy data, with accompanying code and data publicly released.

physics-informed neural networks for predicting physical systems and discovering hidden physics from noisy data

gradient-flow pathologies caused by numerical stiffness, including unbalanced back-propagated gradients during training, and their mitigation to improve predictive accuracy

Publication Details
Publication Date
2021-01-01
Journal
Publisher
ISSN
Cited by
3421
Access Type
Author Information
Authors
Paris G. Perdikaris
Sifan Wang
Yujun Teng
Explore further
Open the scid.ai AI chat with a ready-made request: it will find papers on a similar topic and help build a literature review.
Find similar papers in the chat →
Make a presentation
100%