Evaluation of Seismic Artificial Intelligence with Uncertainty
Оценка искусственного интеллекта в сейсмологии с учетом неопределенности
2025-07-29
SCID: 54.1/jgu8jd6m
Discuss with AI
PhaseNetlearning efficiencyperformance uncertaintyseismic artificial intelligenceseismic phase picking
Figures from the paper
Abstract (AI)
Abstract Artificial intelligence has transformed the seismic community with deep learning models (DLMs) that are trained to complete specific tasks within workflows. However, there is still a lack of robust evaluation frameworks for evaluating and comparing DLMs. We address this gap by designing an evaluation framework that jointly incorporates two crucial aspects: performance uncertainty and learning efficiency. To target these aspects, we meticulously construct the training, validation, and test splits using a clustering method tailored to seismic data and enact an expansive training design to segregate performance uncertainty arising from stochastic training processes and random data sampling. The framework’s ability to guard against misleading declarations of model superiority is demonstrated through the evaluation of PhaseNet (Zhu and Beroza, 2018), a popular seismic phase picking DLM, under three training approaches. Our framework helps practitioners choose the best model for their problem and set performance expectations by explicitly analyzing model performance with uncertainty at varying budgets of training data.
Key Findings
1
An expansive training design separates performance uncertainty caused by stochastic optimization from uncertainty caused by random data sampling.
2
Clustering-based construction of training, validation, and test splits is used to better account for seismic-data characteristics.
3
Evaluation of PhaseNet under three training approaches demonstrates that the framework can prevent misleading claims of model superiority.
4
The framework supports model selection and performance expectation setting across different training-data budgets by explicitly quantifying uncertainty.
5
The paper introduces a seismic-specific evaluation framework jointly addressing model performance uncertainty and learning efficiency.
Research Object
seismic deep learning models (DLMs), exemplified by PhaseNet
Research Subject
performance uncertainty and learning efficiency in training and evaluating seismic DLMs
Publication Details
Publication Date
2025-07-29
Journal
Publisher
ISSN
Cited by
8
Open access PDF
Access Type
Author Information
Download PDF
Subscribe to digest
References available in scid.ai8
Adam: A Method for Stochastic Optimization2014
Scikit-learn: Machine Learning in Python2012
PyTorch: An Imperative Style, High-Performance Deep Learning Library2019
Scikit-learn: Machine Learning in Python2011
On the Opportunities and Risks of Foundation Models2021
A survey of uncertainty in deep neural networks2023
Earthquake transformer—an attentive deep-learning model for simultaneous earthquake detection and phase picking2020
INSTANCE – the Italian seismic dataset for machine learning2021