Elsevier

Public Health

Volume 128, Issue 5, May 2014, Pages 444-474
Public Health

Original Research
Appraisal of guidelines developed by the World Health Organization

https://doi.org/10.1016/j.puhe.2014.01.002Get rights and content

Abstract

Objective

To appraise the quality of guidelines developed by the World Health Organization (WHO) that were approved by its Guidelines Review Committee (GRC) and identify strengths and weaknesses in the guideline development process.

Study design

Cross-sectional.

Methods

Three individuals independently assessed GRC-approved WHO guidelines using the Appraisal of Guidelines for Research and Evaluation II instrument (AGREE II). Scores were standardized across domains and overall quality was determined through consensus.

Results

124 guidelines met inclusion criteria and were assessed. 58 guidelines were recommended for use, 58 were recommended with modifications and eight were not recommended. The highest scoring domains across guidelines were scope and purpose, and clarity of presentation. The recommended guidelines had higher rigor of development and applicability domain scores in comparison to other guidelines. 77% of the guidelines referenced an underlying evidence review and 49% used GRADE to assess the body of evidence or the strength of the recommendation. The domains in need of improvement included stakeholder engagement, editorial independence, and applicability. Guidelines not recommended for use were generally insufficient in their rigor of development.

Conclusions

WHO guidelines need further improvement, most importantly in the rigor of their development (i.e., use of evidence reviews). Other areas for improvement include increased stakeholder engagement, a more explicit process for recommendation formulation and disclosure of interests, discussion of the facilitators, barriers, resource implications, and criteria for monitoring the outcomes of guideline implementation. WHO guidelines can improve through increased transparency, adherence to the WHO Handbook for Guideline Development, and better oversight by the GRC.

Introduction

The World Health Organization (WHO) is the leading authority for global public health guidance and policy. One of its core functions is the development of evidence-based guidelines to help policymakers, healthcare practitioners and consumers make informed decisions regarding health interventions.1 WHO publishes nearly 200 guidelines each year on a broad range of topics including reproductive health, chronic and communicable diseases, nutrition and many others.2 WHO guidelines are used by most of the 194 United Nations member countries, thus, WHO recommendations can impact the health of millions of individuals. It is imperative that these guidelines are developed using rigorous, transparent processes, and that recommendations are based on an unbiased synthesis of the best available evidence.

In 2007, Oxman and colleagues criticized WHO guidelines for their infrequent use of systematic evidence reviews, inconsistent use of systematic guideline development processes, heavy reliance on expert opinion to develop the recommendations, and lack of transparency.3 In addition, WHO guidelines lacked implementation strategies and attention to resource implications, especially in low- and middle-income countries. This study also revealed that WHO had little to no internal support or resources devoted to guideline development.

In response to this criticism in 2007, the WHO Director General established the Guidelines Review Committee (GRC) to ensure that guidelines are developed using transparent, evidence-based processes and meet the highest international standards.4 The GRC revised the WHO Handbook for Guideline Development, instituting internationally recognized standards and methods for guideline development.5 These methods focused particularly on the use of high quality systematic evidence reviews upon which to base recommendations, disclosure and management of secondary interests, assembly of a diverse guideline development group, and the use of an explicit framework for assessing the quality of the evidence and for translating evidence into recommendations (Grading of Recommendations, Assessment, Development and Evaluation [GRADE]).6, 7

Several recent publications examined the quality of a small number of WHO guidelines and suggest that, although there may have been improvements since the inauguration of the GRC, concerns remain and further improvements are needed.8, 9, 10 Our study builds on these previous, small or topic-specific studies by appraising the quality of all WHO guidelines approved by the GRC. The results will help inform WHO of areas in the guideline development process and methods that need to be improved upon, and may help users determine how to prioritize resources across multiple guidelines and potential interventions.

Section snippets

Methods

The authors obtained a list of all published, GRC-approved guidelines from the GRC Secretariat on May 23, 2013. Guidelines are defined at WHO as ‘any document that contains a recommendation for the use of a health intervention, whether these are clinical, public health, or policy recommendations.’5 Because the GRC was implemented in 2007, only WHO guidelines published from that year forward are included in our cohort. Supplementary materials specifically referenced in each guideline were

Results

The list provided by the GRC Secretariat encompassed 381 guidelines in various stages of review and approval; 124 guidelines met inclusion criteria (Fig. 1). The WHO guidelines covered a wide variety of topics; HIV/AIDS and/or tuberculosis were the most frequent health topic (34% of all guidelines addressed one or both of these topics). The type of document also varied, from clinical practice guidelines to policy statements to model chapters for textbooks (Appendix Table 1).

The overall quality

Discussion

There are marked variations in the quality of guidelines developed by the WHO since the inauguration of the GRC, as well as variation within individual guidelines across the domains assessed by AGREE II. Delineation of guidelines' scope and purpose, and clarity of presentation were generally better done, however, stakeholder engagement, editorial independence, and applicability were often poorly reported and/or executed.

Our findings are similar to those reported in other studies examining

Acknowledgements

No funding was provided for this project. The authors would like to thank Susan L. Norris, M.D., M.Sc., M.P.H. for providing a list of GRC-approved guidelines and for providing her expertise on the WHO guideline development process; Michael Leo, Ph.D. at the Kaiser Permanente Center for Health Research for his advice on the data analysis; Sherril Gelmon, Dr.P.H. at Portland State University for identifying a graduate student to participate on the project; and Travis B. Carter for technical

References (150)

  • S. Polus et al.

    Appraisal of WHO guidelines in maternal health using the AGREE II assessment tool

    PLoS One

    (2012)
  • D. Sinclair et al.

    Guideline development at the World Health Organization: an evaluation

    PLoS One

    (2013)
  • M.C. Brouwers et al.

    AGREE II: advancing guideline development, reporting and evaluation in health care

    CMAJ

    (2009 Dec 14)
  • P.E. Shrout et al.

    Intraclass correlations: uses in assessing rater reliability

    Psychol Bull

    (1979 Mar)
  • Institute of Medicine

    Finding what works in health care: standards for systematic reviews

    (2011)
  • U.S. Preventive Services Task Force procedures manual. Washington, D.C.

    (2008)
  • H.K. Holmer et al.

    Quality of clinical practice guideline for glycemic control in type 2 diabetes mellitus

    PLoS One

    (2013)
  • A. Lo Vecchio et al.

    Evaluation of the quality of guidelines for acute gastroenteritis in children with the AGREE instrument

    J Pediatr Gastroenterol Nutr

    (2011 Feb)
  • L.A. Al-Ansary et al.

    A systematic review of recent clinical practice guidelines on the diagnosis, assessment and management of hypertension

    PLoS One

    (2013)
  • A. Acuna-Izcaray et al.

    Quality assessment of asthma clinical practice guidelines: a systematic appraisal

    Chest

    (2013 Feb 28)
  • A.C. Don-Wauchope et al.

    Applicability of the AGREE II instrument in evaluating the development process and quality of current National Academy of Clinical Biochemistry guidelines

    Clin Chem

    (2012 Oct)
  • T. Kredo et al.

    Clinical practice guidelines within the Southern African Development Community: a descriptive study of the quality of guideline development and concordance with best evidence for five priority diseases

    Health Res Policy Syst

    (2012)
  • WHO recommendations for the prevention and treatment of pre-eclampsia and eclampsia

    (2011)
  • WHO recommendations for induction of labour

    (2011)
  • WHO guidelines for the management of postpartum hemorrhage and retained placenta

    (2009)
  • WHO recommendations for the prevention of postpartum hemorrhage

    (2007)
  • Cochrane handbook for systematic reviews of interventions. Version 5.1.0 ed.

    (2011)
  • Centre for Reviews and Dissemination

    Systematic reviews: CRDs guidance for undertaking reviews in health care United Kingdom

    (2008)
  • Acceptable medical reasons for breast milk substitutes

    (2009)
  • Age-friendly primary health care centres toolkit

    (2008)
  • Anthrax in humans and animals

    (2008)
  • Antiretroviral therapy for HIV infection in adults and adolescents: recommendations for a public health approach

    (2010)
  • Antiretroviral therapy for HIV infection in infants and children: towards universal access. Recommendations for a public health approach, 2010 revision

    (2010)
  • Antiretroviral drugs for treating pregnant women and prevention HIV infection in infants

    (2010)
  • Baby-friendly hospital initiative: revised, updated and expanded for integrated care

    (2009)
  • WHO best practices for injections and related procedures toolkit

    (2010)
  • Cancer control: knowledge into action: WHO guide for effective programs. Diagnosis and treatment

    (2008)
  • Clinical guidelines for withdrawal management and treatment of drug dependence in closed settings

    (2009)
  • Guidelines on basic newborn resuscitation

    (2012)
  • Clinical management of acute pesticide intoxication: prevention of suicidal behaviors

    (2008)
  • Communicable diseases and severe food shortage: WHO technical note

    (2010)
  • Community involvement in tuberculosis care and prevention: towards partnerships for health

    (2008)
  • Guideline: daily iron and folic acid supplementation in pregnant women

    (2012)
  • Dengue: guidelines for diagnosis, treatment, prevention and control

    (2009)
  • Fluorescent light-emitting diode (LED) microscopy for diagnosis of tuberculosis: policy statement

    (2011)
  • Global recommendations on physical activity for health

    (2010)
  • Part I: guidance document of characterizing and communicating uncertainty in exposure assessment

    (2008)
  • Guidance on couples HIV testing and counselling including antiretroviral therapy for treatment and prevention of serodiscordant couples: recommendations for a public health approach

    (2012)
  • Guidance on pre-exposure oral prophylaxis (PrEP) for serodiscordant couples, men and transgender women who have sex with men at high risk of HIV: recommendations for use in the context of demonstration projects

    (2012)
  • Guidance on prevention of viral hepatitis B and C among people who inject drugs

    (2012)
  • Cited by (30)

    • Assessing the World Health Organization: What does the academic debate reveal and is it democratic?

      2022, Social Science and Medicine
      Citation Excerpt :

      For instance, one of these studies observed participants at the World Health Assembly and reviewed documents to explain how the format of this body changed over time and which ramifications this had for WHO's governance (Eckl, 2017). There were also two observational studies that analyzed its guidelines (Burda et al., 2014) and its mental health policy (Shen, 2014). One of the two reviews surveyed WHO reform proposals broadly (Cassels et al., 2014a), the other WHO reform proposals for better pandemic preparedness and control following the 2014 Ebola outbreak (Moon et al., 2017).

    • Identifying research questions for HIV, tuberculosis, tuberculosis-HIV, malaria, and neglected tropical diseases through the World Health Organization guideline development process: a retrospective analysis, 2008–2018

      2020, Public Health
      Citation Excerpt :

      In addition, guideline development groups should formulate research questions needed to address identified gaps in knowledge.6,7 There has been significant improvement within the WHO in developing public health guidelines.8,9 However, there has been little emphasis on the opportunity provided by the guideline development process to identify, formulate, and compile relevant research questions that address knowledge gaps.

    • CNGOF Guidelines for Clinical Practice: Revision of the Methodology

      2020, Gynecologie Obstetrique Fertilite et Senologie
    • Improvement evident but still necessary in clinical practice guideline quality: a systematic review

      2017, Journal of Clinical Epidemiology
      Citation Excerpt :

      The included reviews contained AGREE II domain scores from a total of 415 CPGs with publication dates ranging from 1992 to 2014. Each review of CPG quality included a mean of 18 AGREE II quality assessments [14–39], with 26% generating scores from two reviewers, 47% from three reviewers, and 57% from four or more reviewers. Only 44% (10 of 25) of reviews reported the interobserver agreement achieved while scoring CPGs; however, the mean reported interclass correlation coefficients (ICCs) were quite good (mean ICC, 0.857; range, 0.671–0.965).

    • Strength of recommendations in WHO guidelines using GRADE was associated with uptake in national policy

      2015, Journal of Clinical Epidemiology
      Citation Excerpt :

      WHO adopted the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) [2] system in January 2009 to support the formulation of evidence-based recommendations. Since that time, there has been a steady increase in the number of recommendations issued by WHO that are based on the GRADE approach [3–5]. According to GRADE, the strength of the recommendation is designated as either “strong” or “weak” (commonly termed “conditional” in WHO guidelines), with strong recommendations generally based on high-quality evidence from randomized controlled trials, except in certain circumstances [6].

    View all citing articles on Scopus
    View full text