The file drawer problem and tolerance for null results.
Проблема «файлового ящика» и устойчивость к нулевым результатам
1979-05-01
SCID: 54.1/938hjx45
Discuss with AI
Type I errorfile drawer problemp > .05publication biastolerance for null results
Figures from the paper
Abstract (AI)
For any given research area, one cannot tell how many studies have been con-ducted but never reported. The extreme view of the "file drawer problem " is that journals are filled with the 5 % of the studies that show Type I errors, while the file drawers are filled with the 95 % of the studies that show non-significant results. Quantitative procedures for computing the tolerance for filed and future null results are reported and illustrated, and the implications are discussed. Both behavioral researchers and statisti-cians have long suspected that the studies published in the behavioral sciences are a biased sample of the studies that are actually carried out (Bakan, 1967; McNemar, 1960; Smart, 1964; Sterling, 1959). The extreme view of this problem, the "file drawer prob-lem, " is that the journals are filled with the 5 % of the studies that show Type I errors, while the file drawers back at the lab are filled with the 95 % of the studies that show nonsignificant (e.g., p>.05) results. In the past there was very little one could do to assess the net effect of studies, tucked away in file drawers, that did not make the magic.05 level (Rosenthal & Gaito, 1963, 1964). Now, however, although no definitive solution to the problem is available, one can establish reasonable boundaries on the prob-lem and estimate the degree of damage to any research conclusion that could be done by the file drawer problem. This advance in our ability to cope with the file drawer is an outgrowth of the in-creasing interest of behavioral scientists in summarizing bodies of research literature sys-Preparation of this article was supported in part by the National Science Foundation. I would like to thank Judith A. Hall and Donald B. Rubin for their valuable improvements of an earlier version of this article. Requests for reprints should be sent to Robert
Key Findings
1
Although no definitive solution exists, reasonable boundaries and estimates can be established to assess how unpublished null results could damage research conclusions.
2
The authors present quantitative procedures to compute tolerance for filed and future null results, allowing estimation of the file drawer's impact on research conclusions.
3
The file drawer problem posits that published journals overrepresent Type I error studies (≈5%) while unpublished file drawers contain the majority (≈95%) of nonsignificant results.
4
The proposed approach arises from increased interest in summarizing research literature and enables better coping with publication bias in behavioral sciences.
Research Object
The file drawer problem (unpublished/null-result studies omitted from the published literature)
Research Subject
Quantitative tolerance for and impact of unpublished null results on the conclusions of research syntheses (estimating bounds on how many null studies could change meta-analytic or inferential conclusions)
Publication Details
Publication Date
1979-05-01
Journal
Publisher
ISSN
Cited by
8730
Access Type
Author Information
Download PDF
Subscribe to digest
References available in scid.ai1
Cited by20
Meta-analysis of Observational Studies in Epidemiology A Proposal for Reporting2000
Estimating the reproducibility of psychological science2015
Social Relationships and Mortality Risk: A Meta-analytic Review2010
Enhancing well‐being and alleviating depressive symptoms with positive psychology interventions: a practice‐friendly meta‐analysis2009
Predictors of posttraumatic stress disorder and symptoms in adults: A meta-analysis.2003
Toward an integrative social identity model of collective action: A quantitative research synthesis of three socio-psychological perspectives.2008
… and the Cross-Section of Expected Returns2015
Predictors of Postpartum Depression2001
Physical Activity and Incident Depression: A Meta-Analysis of Prospective Cohort Studies2018
The effects of integrating mobile devices with teaching and learning on students' learning performance: A meta-analysis and research synthesis2015
The Relationship Between Parenting and Delinquency: A Meta-analysis2009
Research Methods in Psychology2016
Scientific Utopia2012
Meta-Analysis of Inquiry-Based Learning: Effects of Guidance2016
Predictors of posttraumatic stress disorder and symptoms in adults: A meta-analysis.2008
The Relationship Between Burnout, Depression, and Anxiety: A Systematic Review and Meta-Analysis2019
A Meta-Analysis of School-Based Bullying Prevention Programs' Effects on Bystander Intervention Behavior2012
Technology readiness: a meta-analysis of conceptualizations of the construct and its impact on technology usage2019
Machine learning for medical imaging: methodological failures and recommendations for the future2022
Replicability, Robustness, and Reproducibility in Psychological Science2021