A term used to indicate the deviation of results or inferences from the actual truth. Bias can occur at any stage when conducting a study, for example, during data collection, data analysis, data interpretation, publication or review of the data. Bias generally falls into three categories:
See also Risk of Bias Assessment.
Citation Managers
See Reference/Citation Managers.
Cochrane
An international organization that synthesizes medical research findings to facilitate evidence-based choices about health interventions. The group conducts systematic reviews of healthcare interventions and diagnostic tests and publishes them in the Cochrane Library. Some reviews require paid subscription or registration before reading.
Conflict of Interest
A conflict of interest occurs when someone derives personal benefit from actions or decisions made in their official capacity. In a systematic review, conflicts of interest can introduce bias to the selection of studies, the presentation and analysis of data, and the phrasing of results and conclusions of the review. Reviewers should disclose conflicts of interest to increase transparency and reduce the appearance of bias.
Critical Appraisal
A step in the systematic review process. Critical appraisal involves assessing the quality and validity of the studies to be considered in the review. There are several tools that can assist the reviewer in conducting a critical appraisal.
Data collection is the process of gathering and measuring new, specific data that does not exist elsewhere, from a source. Data collection methods may include surveys, interviews, focus groups, observation, and measurements. Data collection differs from data extraction which involves retrieving information from existing sources.
Data extraction is the process of retrieving information from existing sources like eligible studies, databases, or websites, and capturing that information in a structured and standardized form. Data extraction differs from data collection which involves gathering and measuring new, specific data.
In a systematic review, data extraction occurs between identifying eligible studies for inclusion in the review and analyzing the data for risk of bias.
Deduplication
Deduplication is the process of identifying and removing duplicate records from a systematic review. It is a risk-mitigating step to prevent introducing bias in the conclusions of a review.
There are two forms of deduplication:
The process can be done manually or with the aid of software (e.g., reference/citation managers, screening software).
Eligibility (Inclusion/Exclusion) Criteria
Eligibility criteria, also known as inclusion and exclusion criteria, are a set of characteristics that define which studies are included or excluded from a systematic review. The framing of the research question, and what the researchers deem relevant, constitute the criteria that works must meet to be eligible for comparison. Common criteria include: language, publication date, geography, characteristics of the population studied, exposure (or intervention), and study design.
Empirical Research
Empirical research is any study whose conclusions are derived from concrete, verifiable evidence, rather than theory or belief. Empirical research can use either qualitative or quantitative research methods to collect data. Data is gathered through experience, observation, or measurements taken with the help of calibrated scientific tools.
Evidence-Based Practice
Evidence-based practice (EBP) is a process that involves reviewing, analyzing, and translating the latest scientific evidence. The goal of EBP is to inform decision-making and guide the delivery of services and interventions to improve outcomes for a patient, a client, or an organization. EBP can be applied in a variety of fields including healthcare, education, and social work.
Evidence Synthesis
Evidence synthesis is a research method that combines all relevant information from primary studies and grey literature to answer a research question. Different types of evidence synthesis include systematic reviews, scoping reviews, rapid reviews, umbrella reviews, and meta-analyses.
As a step in the systematic review process, evidence synthesis involves combining appraised data from eligible studies and analyzing it to determine if there is enough evidence to state conclusively that together the studies answer a research question.
Exclusion criteria are characteristics that prevent a study from being included in the review. Some areas to consider when defining exclusion criteria include study design, methodology, interventions, publication type/date, and language.
See also Eligibility Criteria.
Forest Plot
A graphical representation that displays the results of a meta-analysis, i.e., the results of multiple studies that address a common research question.
Funnel Plot
A graphical tool used to identify publication bias in a meta-analysis or systematic review.
Grey Literature
Grey literature is unpublished work or work published in a noncommercial manner. It is often produced by government agencies, academic institutions, and the for-profit sector. Grey literature does not typically go through a peer-review process. Examples include reports, policy literature, proceedings, government documents, white papers, evaluations, conference abstracts, dissertations, and theses.
Inclusion criteria are characteristics that a study must have to be included in the review. Some areas to consider when defining inclusion criteria include study design, participants, interventions, outcome measures, publication, and language.
See also Eligibility Criteria.
MeSH Terms
MeSH (Medical Subject Headings) is a controlled vocabulary thesaurus used by the National Library of Medicine (NLM) for indexing, cataloging, and searching for articles in the PubMed database.
Meta-Analysis
Meta-analysis is a quantitative statistical method that combines data from multiple studies on a similar topic to draw conclusions about that body of research. Meta-analyses can be misleading if specific study designs, biases, and reporting biases are not carefully considered.
Mixed Methods Research
Mixed methods research involves collecting and analyzing both qualitative (descriptive, non-numerical) and quantitative (numerical) data. A mixed methods approach can be useful to explore a research question more deeply while also providing statistical context, or to better understand the "why" behind a quantitative datapoint by gathering qualitative insights.
Observational Trial
Unlike randomized controlled trials, in observational trials, participants are not assigned to a treatment or other intervention. Instead, researchers observe participants to assess health outcomes. Examples of observational trials include case-control studies, cross-sectional studies, and longitudinal studies (e.g., cohort study, panel study).
PICO Model
A structured approach for developing research questions within a systematic review. The PICO model uses four components to express the key elements needed for a focused research question:
Population being addressed
Intervention (or indicator, exposure, diagnostic tool)
Comparator (or control)
Outcome
Sometimes additional elements are added to the model to capture time (T) or study type (S), changing the acronym to PICOT or PICOS, respectively.
PRISMA
The PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses) statement and its extensions provide guidelines for the reporting of different types of systematic reviews and other types of evidence synthesis (e.g., scoping reviews). It consists of a checklist of items that are required to ensure the quality of reports. It also consists of a diagram that depicts the flow of information through the different phases of a systematic review.
PROSPERO
PROSPERO is an international database of prospectively registered systematic reviews in health and social care, welfare, public health, education, crime, justice, and international development, where there is a health-related outcome. (PROSPERO) See this page for other places to register your protocols.
Protocol
A protocol for a systematic review is a detailed plan that outlines the methods and objectives of the review. It promotes consistent conduct and accountability across the review team, research integrity, and transparency. It should be created before the review begins and published through a public registry (e.g., Cochrane, PROSPERO).
Qualitative research methods involve collecting and analyzing non-numerical data (e.g., text, audio, video) to understand experiences, perspectives, and behaviors. Qualitative data are often gathered through interviews, focus groups, observation, document analysis, case studies, or surveys with open-ended questions. The goal is to explore answers to the questions "why" and "how."
Quality Assessment
A quality assessment is an evaluation of the methodologies used by studies included in a systematic review. The aim of the process is to minimize the risk of bias and increase confidence in the review findings. Quality assessment is also known as quality appraisal, critical appraisal, and risk of bias assessment.
Quantitative research methods involve collecting and analyzing numerical data and statistics to explain, measure, and/or prove a link between two different concepts (e.g., diet and obesity). Quantitative data are often gathered through surveys with closed-ended questions, experiments, or observations with measurable outcomes. The goal is to explore answers to the questions "how much" and "how many."
A scientific experiment (usually medical or health-related) that compares the effectiveness of two or more interventions/treatments. In this type of experiment, participants are randomly assigned to groups, either experimental or control. The control is typically a standard treatment (gold standard) or no treatment (placebo).
A reference/citation manager helps organize and track articles and books as you find them, tag and annotate them, and easily create citations and bibliographies in Microsoft Word, Google Docs, or other writing software. It allows you to share sources and collaborate with classmates and colleagues. It is a useful tool when conducting a systematic review because you can export to, manage, and deduplicate (i.e., remove copies) search results within the tool. Some popular tools include EndNote, EndNote Online, Mendeley, and Zotero. See here to choose a reference/citation manager that is right for you.
Reliability is an important concept to consider when evaluating the quality of a systematic review. Reliability refers to the consistency of a measure, or the reproducibility of study results if the same instruments were used in different situations.
See also Validity.
When conducting a systematic review, studies to be included in the review need to be evaluated for risk of bias. This elevates the methodological rigor and transparency of reported results. Bias can be found in the individual studies but also in the actions of the review authors who are writing the systematic review. Domains of bias include the randomization process, deviations from intended interventions, missing outcome data, measurement of the outcome, and selection of reported results.
Common tools that assess risk of bias include:
Scoping Review
A scoping review is one approach to evidence synthesis. It is similar to a systematic review, but with some key differences. A scoping review is used to identify and map the available evidence for a broad topic. It provides an overview of a potentially large and diverse body of literature without assessing the quality of individual studies, whereas a systematic review aims to find an answer to a specific research question by combining empirical evidence from a relatively small number of studies and assessing the quality of those studies.
A scoping review might ask: What non-pharmaceutical interventions have been investigated for the treatment of ADHD?
A systematic review might ask: Is cognitive behavioral therapy an effective treatment for adolescents with ADHD?
Screening
Screening is the process of determining which articles meet the inclusion criteria (as established in the protocol) and should be included in the review. A reference/citation manager and/or screening software can help with this process.
Search Hedges and Filters
Search hedges and filters are types of search strategies used to retrieve journal articles from databases. They both use pre-defined combinations of keywords, phrases, and subject terms to limit results, but they differ in how they refine concepts.
Hedges are designed to extract articles based on subject matter. They combine terms with their synonyms and variant spellings to produce results. Search filters are designed to extract articles that meet specific static characteristics (e.g., study design).
Hedges and filters are beneficial for systematic reviews because they are typically designed to retrieve a comprehensive list of search results. Since hedges and filters can be reused, they save the researcher time from having to build searches from scratch.
Search Strategy
A search strategy is a plan for finding relevant information in a database. Search strategies are created by combining key concepts from a topic or research question using Boolean operators and adding keywords, phrases, subject headings, and limiters to retrieve results.
Supplementary/Hand Searching
Supplementary or hand searching refers to identifying relevant literature by other means outside of one's search strategy. Methods might include examining specific relevant journals and websites, or "snowballing" by either mining the bibliographies of exemplary articles for other relevant references or looking for other articles that have cited the exemplary articles.
Systematic Review
A systematic review is one approach to evidence synthesis. It aims to minimize bias by using explicit, systematic methods documented in advance with a protocol. It is similar to a scoping review, but with some key differences. A systematic review is used to find an answer to a specific research question by combining empirical evidence from a relatively small number of studies and assessing the quality of those studies, whereas a scoping review provides an overview of a potentially large and diverse body of literature without assessing the quality of individual studies.
A scoping review might ask: What non-pharmaceutical interventions have been investigated for the treatment of ADHD?
A systematic review might ask: Is cognitive behavioral therapy an effective treatment for adolescents with ADHD?
Systematic Search Strategy
A systematic search strategy requires careful planning to identify all relevant records that meet the review’s eligibility criteria in a reproducible and transparent manner. Building a search strategy involves identifying concepts, related terms, search fields, Boolean operators, and any special features used for each database or resource that is searched.
Validity is an important concept to consider when evaluating the quality of a systematic review. Validity refers to the accuracy of a measure, or whether the study instruments measure what they are intended to measure. Systematic reviews report on the internal validity of each included study (risk of bias, methodological errors) and potentially some aspects of external validity (i.e., generalizability, applicability, and feasibility).
See also Reliability.
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 Generic License.