Which academic search systems are suitable for systematic reviews or meta‐analyses? Evaluating retrieval qualities of Google Scholar, PubMed, and 26 other resources

Какие академические поисковые системы подходят для систематических обзоров или метаанализов? Оценка качества поиска в Google Scholar, PubMed и 26 других ресурсах
Neal Haddaway, Michael Gusenbauer
2019-10-15

academic search systemsevidence synthesismeta-analysesprecision, recall, and reproducibilitysystematic reviews
Rigorous evidence identification is essential for systematic reviews and meta-analyses (evidence syntheses) because the sample selection of relevant studies determines a review's outcome, validity, and explanatory power. Yet, the search systems allowing access to this evidence provide varying levels of precision, recall, and reproducibility and also demand different levels of effort. To date, it remains unclear which search systems are most appropriate for evidence synthesis and why. Advice on which search engines and bibliographic databases to choose for systematic searches is limited and lacking systematic, empirical performance assessments. This study investigates and compares the systematic search qualities of 28 widely used academic search systems, including Google Scholar, PubMed, and Web of Science. A novel, query-based method tests how well users are able to interact and retrieve records with each system. The study is the first to show the extent to which search systems can effectively and efficiently perform (Boolean) searches with regards to precision, recall, and reproducibility. We found substantial differences in the performance of search systems, meaning that their usability in systematic searches varies. Indeed, only half of the search systems analyzed and only a few Open Access databases can be recommended for evidence syntheses without adding substantial caveats. Particularly, our findings demonstrate why Google Scholar is inappropriate as principal search system. We call for database owners to recognize the requirements of evidence synthesis and for academic journals to reassess quality requirements for systematic reviews. Our findings aim to support researchers in conducting better searches for better evidence synthesis.
1
A novel query-based evaluation measures how effectively and efficiently each system supports Boolean searching in terms of precision, recall, and reproducibility.
2
Google Scholar is inappropriate as the principal search system for systematic reviews or meta-analyses.
3
Only about half of the evaluated systems, including few open-access databases, are recommendable for evidence syntheses without substantial caveats.
4
Search-system performance varies substantially, resulting in major differences in suitability for systematic searches.
5
The study systematically compares retrieval quality across 28 academic search systems, including Google Scholar, PubMed, and Web of Science.

28 widely used academic search systems, including Google Scholar, PubMed, and Web of Science

Their systematic-search retrieval quality and performance, including precision, recall, reproducibility, usability, and search effort

Publication Details
Publication Date
2019-10-15
Journal
Publisher
ISSN
Cited by
2936
Access Type
Author Information
Authors
Neal Haddaway
Michael Gusenbauer
Explore further
Open the scid.ai AI chat with a ready-made request: it will find papers on a similar topic and help build a literature review.
Find similar papers in the chat →
Make a presentation
100%