A gold standard of scientific analysis is fast becoming tarnished, according to a report by a leading meta-researcher.
Systematic reviews and meta-analyses distil scientific articles on similar questions into what is meant to be an authoritative take on a particular topic — often how well a particular treatment works across medical settings — and they are key tools in evidence-based medicine.
But valuable reports are getting diluted by “a massive production of unnecessary, misleading and conflicted systematic reviews and meta-analyses”, according to John Ioannidis at Stanford University in California, who has published a report in The Milbank Quarterly1 looking at trends in the publication of these articles.
He decided to try to quantify the problem after noticing “an epidemic” of poor and obviously flawed articles, he says. “When the most influential literature is wrong, the harm that is done is worse than when unimportant studies are wrong.”
Ioannidis counted the number of articles that had been tagged as ‘systematic reviews’ and ‘meta-analyses’ in PubMed, a database of biomedical and life sciences publications. From 1991 to 2014, the numbers of these articles published annually increased by more than 2,600%, to 28,959 for systematic reviews and 9,135 for meta-analyses. Over the same time period, the total number of articles appearing in PubMed each year increased by 153%.
The most prolific source of meta-analyses — 63% of the total in 2014 — were genetic association studies from authors in China, many of which did not account for the high likelihood of finding false positives, Ioannidis concluded.
One reason that systematic reviews are increasing is that more people around the world are doing research and are eager to get publications, says Christopher Schmid, a biostatistician at Brown University School of Public Health in Providence, Rhode Island. Schmid, who handles meta-analyses at the American Journal of Kidney Diseases, says that …