Double-check: validation of diagnostic statistics for PLS-DA models in metabolomics studies

Повторная проверка: валидация диагностических статистик для моделей PLS-DA в метаболомических исследованиях
Ewa Szymańska, Edoardo Saccenti, Age K. Smilde, Johan A. Westerhuis
2011-07-07

AUROCPLS-DAdiagnostic statisticsmetabolomicspermutation testing
Partial Least Squares-Discriminant Analysis (PLS-DA) is a PLS regression method with a special binary 'dummy' y-variable and it is commonly used for classification purposes and biomarker selection in metabolomics studies. Several statistical approaches are currently in use to validate outcomes of PLS-DA analyses e.g. double cross validation procedures or permutation testing. However, there is a great inconsistency in the optimization and the assessment of performance of PLS-DA models due to many different diagnostic statistics currently employed in metabolomics data analyses. In this paper, properties of four diagnostic statistics of PLS-DA, namely the number of misclassifications (NMC), the Area Under the Receiver Operating Characteristic (AUROC), Q(2) and Discriminant Q(2) (DQ(2)) are discussed. All four diagnostic statistics are used in the optimization and the performance assessment of PLS-DA models of three different-size metabolomics data sets obtained with two different types of analytical platforms and with different levels of known differences between two groups: control and case groups. Statistical significance of obtained PLS-DA models was evaluated with permutation testing. PLS-DA models obtained with NMC and AUROC are more powerful in detecting very small differences between groups than models obtained with Q(2) and Discriminant Q(2) (DQ(2)). Reproducibility of obtained PLS-DA models outcomes, models complexity and permutation test distributions are also investigated to explain this phenomenon. DQ(2) and Q(2) (in contrary to NMC and AUROC) prefer PLS-DA models with lower complexity and require higher number of permutation tests and submodels to accurately estimate statistical significance of the model performance. NMC and AUROC seem more efficient and more reliable diagnostic statistics and should be recommended in two group discrimination metabolomic studies. ELECTRONIC SUPPLEMENTARY MATERIAL: The online version of this article (doi:10.1007/s11306-011-0330-3) contains supplementary material, which is available to authorized users.
1
DQ(2) and Q(2) favor less complex PLS-DA models and require more permutation tests and submodels to estimate statistical significance accurately.
2
Differences among diagnostic statistics are associated with model reproducibility, complexity, and permutation-test distributions.
3
NMC and AUROC are identified as more efficient and reliable statistics for two-group discrimination in metabolomics studies.
4
NMC and AUROC detect very small differences between control and case groups more powerfully than Q(2) and DQ(2).
5
The study evaluates four PLS-DA diagnostic statistics—NMC, AUROC, Q(2), and DQ(2)—across three metabolomics datasets with varying sizes and group differences.

PLS-DA models applied to two-group metabolomics data sets

Validation and comparative performance of diagnostic statistics (NMC, AUROC, Q(2), and DQ(2)) for model optimization, significance assessment, reproducibility, and complexity

Publication Details
Publication Date
2011-07-07
Journal
Publisher
ISSN
Access Type
Author Information
Authors
Ewa Szymańska
Edoardo Saccenti
Age K. Smilde
Johan A. Westerhuis
Explore further
Open the scid.ai AI chat with a ready-made request: it will find papers on a similar topic and help build a literature review.
Find similar papers in the chat →
Make a presentation
100%