Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation
Оценка: от precision, recall и F-меры к ROC, informedness, markedness и корреляции
2020-10-11
SCID: 54.1/hm3npwer
Discuss with AI
CorrelationF-MeasureInformednessMarkednessPrecisionROCRand AccuracyRecallSignificancechance-corrected evaluationdichotomous and multi-class extension
Figures from the paper
Abstract (AI)
Commonly used evaluation measures including Recall, Precision, F-Measure and Rand Accuracy are biased and should not be used without clear understanding of the biases, and corresponding identification of chance or base case levels of the statistic. Using these measures a system that performs worse in the objective sense of Informedness, can appear to perform better under any of these commonly used measures. We discuss several concepts and measures that reflect the probability that prediction is informed versus chance. Informedness and introduce Markedness as a dual measure for the probability that prediction is marked versus chance. Finally we demonstrate elegant connections between the concepts of Informedness, Markedness, Correlation and Significance as well as their intuitive relationships with Recall and Precision, and outline the extension from the dichotomous case to the general multi-class case.
Key Findings
1
A system can score better on Precision/Recall/F-measure yet perform worse by the objective measure of Informedness.
2
Common evaluation measures like Recall, Precision, F-measure and Rand Accuracy are biased and can be misleading without accounting for chance or base rates.
3
Markedness is introduced as a dual measure assessing the probability that prediction is marked versus chance.
4
The authors demonstrate formal connections between Informedness, Markedness, Correlation, Significance and their relationships to Recall and Precision, and provide an extension to multi-class cases.
5
The paper advocates measures that reflect the probability that prediction is informed versus chance, notably Informedness.
Research Object
Evaluation measures for binary and multiclass predictive systems (Recall, Precision, F-Measure, Rand Accuracy, ROC-related measures, Informedness, Markedness, Correlation)
Research Subject
Biases, chance/base-rate effects and relationships among evaluation metrics, and the properties of Informedness, Markedness, Correlation (and their connections to Recall, Precision, ROC and significance) including extension from dichotomous to multiclass cases
Publication Details
Publication Date
2020-10-11
Journal
arXiv (Cornell University)
Publisher
Cornell University
ISSN
2331-8422
Cited by
7710
Open access PDF
Access Type
Author Information
Download PDF
Subscribe to digest
Cited by6
The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation2020
U-Net and Its Variants for Medical Image Segmentation: A Review of Theory and Applications2021
Automated Breast Ultrasound Lesions Detection Using Convolutional Neural Networks2017
Predicting Employee Attrition using Machine Learning2018
Mapping the Spatiotemporal Dynamics of Cropland Abandonment and Recultivation across the Yangtze River Basin2024
Monitoring Abandoned Cropland in Fragmented Mountainous Landscapes Based on the ML-LandTrendr Framework2026