Systematic reviews are more applicable than people may believe

NEWS | 2015-04-14

Misconceptions regarding the applicability of systematic reviews are common. One recent example occurred in a review of the impacts of marine infrastructure published in the journal Frontiers in Ecology and the Environment. EviEM Project Manager Neal Haddaway responded in a letter published in the journal in April.

Some authors believe systematic reviews
aren’t appropriate

In a recent article in the journal Frontiers in Ecology and the Environment, Australian researcher Katherine Dafforn and five co-authors reviewed the evidence on marine infrastructure and developments. The authors highlighted the ecological consequences of in-water artificial structures around our cities, but they stated that “a systematic review was not possible given that much of the relevant literature crosses scholarly disciplines and is located in books, conference proceedings, and gray literature that would not have appeared in searches”.

EviEM Project Manager Neal Haddaway, in collaboration with Postdoctoral Research Officer Helen Bayliss from the Centre for Evidence-Based Conservation in Wales, decided to point out that these statements reflect misunderstandings of the applicability of systematic review methods. In their response, Haddaway and Bayliss identified two misconceptions in particular: i) that systematic reviews are unsuitable for multidisciplinary projects, and ii) that systematic reviews cannot account for grey literature.

Systematic reviews cope well
with multidisciplinary contexts

Systematic review methods were originally developed some 25 years ago in the field of medicine. They have since been transferred to the social sciences, psychology, public health, international development and the environmental sciences.

Systematic reviews have been published in a much wider range of fields, however. Not only are systematic review methods applicable to a huge range of disciplines; they are also very useful where different fields combine, for example environmental science and public health, or education and international development. This fact is easily demonstrated by a glance over recent systematic reviews published by the Collaboration for Environmental Evidence and by 3ie (International Initiative for Impact Evaluation).

Far from imposing restrictions, systematic reviews offer a reliable, transparent, repeatable method for reviewing evidence where different disciplines may use divergent methods to measure the impact of a specific factor.

Systematic reviews should always
attempt to identify grey literature

Typically, conventional literature reviews consider only the academic literature when synthesising the evidence on a specific topic. In doing so, these reviews may be missing significant and very useful pieces of evidence that have been published as grey literature, including organisational reports, government papers, or academic theses. Focusing only on traditional academic literature may therefore cause the findings of a review to be wrong, for example giving an overestimate of the real effect. This publication bias can be a considerable risk for those needing a reliable answer to a review question.

Properly conducted systematic reviews, on the other hand, always include an attempt to identify relevant grey literature. This is typically undertaken by searching in dedicated databases for theses, reports and white papers. It might also include contacting relevant practitioners for information directly. Again, far from being incompatible with systematic reviews, searches for grey literature should form a vital part of any such review.

Systematic review may not always be a suitable way of summarising evidence on a topic, particularly where available resources are limited. However, multidisciplinary topics with potentially significant bodies of grey literature are often ideal for systematic review.

By raising awareness of the misconceptions in the review by Dafforn and her co-authors, Haddaway and Bayliss hope they have demonstrated that systematic reviews are often more applicable than people think.