World Bank report muddies the systematic review waters

NEWS | 2015-03-12

A recent World Bank report reviewed six overlapping ‘systematic reviews’ and found that their conclusions differed. Systematic reviews should therefore be ‘taken with a grain of salt’, the authors claim. A collaboration involving EviEM project managers examined the report and found fundamental misunderstandings about what makes a review ‘systematic’.

World Bank report says systematic reviews
may not be reliable

In a World Bank report, David Evans and Anna Popova claim to have ‘systematically reviewed’ a set of six similar systematic reviews on the same subject in international development. Each of the reviews they examine investigated attempts to improve children’s learning outcomes in low and middle income countries.

The report claims that the reviews differed significantly in their findings, a surprising result considering their substantial subject overlap. According to Evans and Popova, such disagreement demonstrates that systematic reviews ‘should be taken with a grain of salt’, since their findings may not be reliable.

The conclusions of the report have recently been highlighted in a blog post at the World Bank website.

‘Systematic’ reviews were not really systematic

EviEM project managers Neal Haddaway and Magnus Land were concerned by the statements in the World Bank report. They worked with Laurenz Langerfrom the Africa Evidence Network (AEN) to find out more about what was causing the disagreements.

Through a critical appraisal of both the Evans and Popova report and the supposed ‘systematic reviews’ that the report was based on, Haddaway, Land and Langer found that very little in any of the reviews was systematic. They have written about their findings in a blog on the AEN website, showing that only two of the included reviews in Evans and Popova’s report could come close to being called a ‘systematic review’, and that several of the included reviews could never be classed as systematic.

Haddaway, Land and Langer argue that a review is not systematic simply because it has a systematic search: systematic methods are necessary throughout the process. They conclude that systematic reviews that are undertaken to a high standard, such as those demanded by systematic review coordinating bodies like CEE and Campbell, remain reliable products that provide vital summaries of the best available evidence on a particular topic.