LibGuides: Systematic Reviews &amp; Evidence Synthesis Methods: Critical Appraisal

Critical Appraisal

Some reviews require a critical appraisal for each study that makes it through the screening process. This involves a risk of bias assessment and/or a quality assessment. The goal of these reviews is not just to find all of the studies, but to determine their methodological rigor, and therefore, their credibility.

"Critical appraisal is the balanced assessment of a piece of research, looking for its strengths and weaknesses and them coming to a balanced judgement about its trustworthiness and its suitability for use in a particular context." 1

It's important to consider the impact that poorly designed studies could have on your findings and to rule out inaccurate or biased work.

Selection of a valid critical appraisal tool, testing the tool with several of the selected studies, and involving two or more reviewers in the appraisal are good practices to follow.

1. Purssell E, McCrae N. How to Perform a Systematic Literature Review: A Guide for Healthcare Researchers, Practitioners and Students. 1st ed. Springer; 2020.

Evaluation Tools

The Appraisal of Guidelines for Research & Evaluation Instrument (AGREE II)
The Appraisal of Guidelines for Research & Evaluation Instrument (AGREE II) was developed to address the issue of variability in the quality of practice guidelines.
Centre for Evidence-Based Medicine (CEBM). Critical Appraisal Tools
"contains useful tools and downloads for the critical appraisal of different types of medical evidence. Example appraisal sheets are provided together with several helpful examples."
Critical Appraisal Skills Programme (CASP) Checklists
Critical Appraisal checklists for many different study types
Critical Review Form for Qualitative Studies
Version 2, developed out of McMaster University
Development of a critical appraisal tool to assess the quality of cross-sectional studies (AXIS)
Downes MJ, Brennan ML, Williams HC, et al. Development of a critical appraisal tool to assess the quality of cross-sectional studies (AXIS). BMJ Open 2016;6:e011458. doi:10.1136/bmjopen-2016-011458
Downs & Black Checklist for Assessing Studies
Downs, S. H., & Black, N. (1998). The Feasibility of Creating a Checklist for the Assessment of the Methodological Quality Both of Randomised and Non-Randomised Studies of Health Care Interventions. Journal of Epidemiology and Community Health (1979-), 52(6), 377–384.
GRADE
The Grading of Recommendations Assessment, Development and Evaluation (GRADE) working group "has developed a common, sensible and transparent approach to grading quality (or certainty) of evidence and strength of recommendations."
Grade Handbook
Full handbook on the GRADE method for grading quality of evidence.
MAGIC (Making GRADE the Irresistible choice)
Clear succinct guidance in how to use GRADE
Joanna Briggs Institute. Critical Appraisal Tools
"JBI’s critical appraisal tools assist in assessing the trustworthiness, relevance and results of published papers." Includes checklists for 13 types of articles.
Latitudes Network
This is a searchable library of validity assessment tools for use in evidence syntheses. This website also provides access to training on the process of validity assessment.
Mixed Methods Appraisal Tool
A tool that can be used to appraise a mix of studies that are included in a systematic review - qualitative research, RCTs, non-randomized studies, quantitative studies, mixed methods studies.
RoB 2 Tool
Higgins JPT, Sterne JAC, Savović J, Page MJ, Hróbjartsson A, Boutron I, Reeves B, Eldridge S. A revised tool for assessing risk of bias in randomized trials In: Chandler J, McKenzie J, Boutron I, Welch V (editors). Cochrane Methods. Cochrane Database of Systematic Reviews 2016, Issue 10 (Suppl 1). dx.doi.org/10.1002/14651858.CD201601.
ROBINS-I Risk of Bias for non-randomized (observational) studies or cohorts of interventions
Sterne J A, HernÃ¡n M A, Reeves B C, SavoviÄ‡ J, Berkman N D, Viswanathan M et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions BMJ 2016; 355 :i4919 doi:10.1136/bmj.i4919
Scottish Intercollegiate Guidelines Network. Critical Appraisal Notes and Checklists
"Methodological assessment of studies selected as potential sources of evidence is based on a number of criteria that focus on those aspects of the study design that research has shown to have a significant effect on the risk of bias in the results reported and conclusions drawn. These criteria differ between study types, and a range of checklists is used to bring a degree of consistency to the assessment process."
The TREND Statement (CDC)
Des Jarlais DC, Lyles C, Crepaz N, and the TREND Group. Improving the reporting quality of nonrandomized evaluations of behavioral and public health interventions: The TREND statement. Am J Public Health. 2004;94:361-366.

Learn more

Assembling the Pieces of a Systematic Reviews, Chapter 8: Evaluating: Study Selection and Critical Appraisal.
Cochrane Handbook, Chapter 14: Completing "Summary of Findings' Tables and Grading the Certainty of the Evidence Cochrane has adopted the GRADE approach (Grading of Recommendations Assessment, Development and Evaluation) for assessing certainty (or quality) of a body of evidence. The GRADE approach specifies four levels of the certainty for a body of evidence for a given outcome: high, moderate, low and very low. GRADE assessments of certainty are determined through consideration of five domains: risk of bias, inconsistency, indirectness, imprecision and publication bias. For evidence from non-randomized studies and rarely randomized studies, assessments can then be upgraded through consideration of three further domains.
How to Perform a Systematic Literature Review, Chapter: Critical Appraisal: Assessing the Quality of Studies.

Systematic Reviews & Evidence Synthesis Methods

Critical Appraisal

Critical Appraisal

Evaluation Tools

Learn more

Other library guides