National Guideline Clearinghouse | Expert Commentaries: Introducing the National Guideline Clearinghouse Revised Inclusion Criteria

full-text ►
National Guideline Clearinghouse | Expert Commentaries: Introducing the National Guideline Clearinghouse Revised Inclusion Criteria

July 22, 2013

July 22, 2013

Introducing the National Guideline Clearinghouse Revised Inclusion Criteria

By: The National Guideline Clearinghouse and National Quality Measures Clearinghouse Project Teams
In 1997 at the inception of the National Guideline Clearinghouse (NGC), the initial inclusion criteria were created to represent the state of the art in guideline development and the Institute of Medicine's definition of clinical practice guidelines (CPGs) that were current at that time. These inclusion criteria have been in place since NGC first went live in December 1998. The NGC inclusion criteria have now been revised to reflect advances in CPG development. In March 2011, the Institute of Medicine (IOM) published Clinical Practice Guidelines We Can Trust , (1) which updated the definition of CPGs to "statements that include recommendations intended to optimize patient care that are informed by a systematic review of evidence and an assessment of the benefits and harms of alternative care options." This report represents the first update of the IOM's definition of clinical practice guidelines since 1990 and also includes standards for guideline development. The revised inclusion criteria, scheduled for implementation starting in June 2014, bring NGC into alignment with this new definition of guidelines and are the foundation for the more rigorous standards set forth by the IOM.
The IOM Report—A New Definition of CPG and Eight Standards
The new definition for a CPG puts particular emphasis on the important roles of a systematic review (SR) of the evidence and an explicit assessment of benefits and harms. The NGC Project team* supports this focus, considering these as minimum requirements for a CPG to be included in NGC. The systematic review is a critical foundation for a guideline, providing an up-to-date evidence base undergirding the guideline's recommendations while also helping to minimize bias. The initial NGC inclusion criteria did include a requirement for a systematic literature search and review; however, the revised inclusion criteria take this criterion a step further by requiring a full systematic review to support the guideline. The NGC revised inclusion criteria also identify the specific components of the SR that must be reported. This includes not only a systematic literature search but also synthesis of the evidence and a summary presentation of the evidence synthesis as it relates to the guideline's recommendations. The second major addition to the inclusion criteria is the requirement for an explicit assessment of benefits and harms of recommended care and alternative care options. According to the IOM report, "...consideration of the benefits and harms associated with particular recommendations, and unambiguous translation of this appraisal..." (1) is critical for clinicians and patients to weigh different treatment alternatives.

*The NGC Project team included the Agency for Healthcare Research and Quality (AHRQ), ECRI Institute (AHRQ's technical contractor for NGC and the National Quality Measures Clearinghouse [NQMC], and a subset of the NGC/NQMC Editorial Board (EB).

In addition to the new CPG definition, the IOM report called for more rigorous and transparent development of guidelines and set forth eight principal standards to which clinical practice guidelines should adhere. These covered areas of transparency, conflicts of interest, guideline development group composition, systematic review, evidence foundations for recommendations, rating strength of and articulation of recommendations, external review, and updating. These eight principles laid out by the IOM standards will be incorporated in the future by documenting the extent of adherence to the standards of each guideline represented on NGC that meets the revised inclusion criteria.
Methods to Develop Revised Inclusion Criteria
The extensive process by which NGC revised the inclusion criteria occurred over the past two years, from 2011 to 2013. It began initially with evaluation of the IOM report, which included a pilot study examining guideline developers' awareness and perceptions of the IOM standards and their intentions to meet them. The study also gauged guideline developers' perceptions of how well they were currently meeting the IOM standards. Taking the outcomes of the pilot study and AHRQ's priorities into consideration, the NGC project team and the NGC/NQMC Expert Panel carefully determined how NGC should best incorporate the IOM's new definition and the standards. The selected approach phases in the new expectations, beginning with the revised inclusion criteria.
The process to revise NGC inclusion criteria began in 2010 with a notice on NGC's Inclusion Criteria page advising users of the intent to revise the criteria and inviting them to provide comments. Then in 2011, NGC sent an e-mail to active guideline developers advising them that the inclusion criteria might be revised and requesting their assistance in that effort. Public comment was again requested during AHRQ's Webinar titled "AHRQ and NGC Approach to Addressing the IOM Standards: Implications for the North American Guideline Development Community," presented in March 2012 to the Guidelines International Network (G-I-N) North America. All comments received were considered by the NGC project team.
Following publication of the IOM Report in 2011, three NGC project team members (J. Jue, P. Shekelle, K. Lohr) began parsing the IOM definition and creating appropriate sub-criteria that were judged to accurately reflect its meaning and could be implemented. The NGC project team then drafted these criteria and presented them to a spectrum of guideline developers, topic experts, the EB, and members of the Expert Panel asking them to rate each criterion for 1) importance, 2) the urgency of implementation, and 3) the perceived level of difficulty associated with implementation. The responses were analyzed and discussed with a subset of the EB to fine-tune the revisions to the inclusion criteria.
Together, NGC's project team and the EB undertook numerous iterations of the revised inclusion criteria over several months of discussion. Our task was to balance the overall goal of revising NGC's inclusion criteria with the challenges that might pose to guideline developers. With each iteration, we also field-tested the criteria with a randomly selected** set of recent guidelines represented in NGC. In total, we reviewed 100 guidelines against the new inclusion criteria. All these steps resulted in concisely worded criteria that can be implemented, that incorporate the new IOM definition of a clinical practice guideline, and that retain and clarify the initial NGC inclusion criteria.

**The proposed inclusion criteria were field-tested against 100 guidelines currently represented in NGC. The testing was performed in two batches of 30 guidelines each and a final batch of 40 guidelines, totaling 100. The batches of 30 were selected through a random sample of NGC guideline unique numeric identifiers represented in the Clearinghouse at the time of the audit. The numbers were determined using an on-line random number generator (random.org ). The process to select the final 40 summaries started with a list of developers represented on NGC as of April 18, 2013, and then removed all organizations that were included only as co-developers (in order to eliminate the chance of selecting two organizations that worked on the same guideline). This left 200 organizations, which were numbered from 1 to 200. Using the same random number generator, we created a list of 40 integers between 1 and 200. The results were matched up with the developer and we then selected the most recent summary from that developer available on NGC. The revised inclusion criteria were then reviewed against the full-text original guideline document and companion documents.

Systematic Review (SR)
Along with the guidance for trustworthy guidelines, in 2011, the IOM also published Finding what works in health care: standards for systematic reviews ." (2) This report established optimal practice for systematic reviews and clearly distinguishes an SR of the evidence from other types of literature reviews. According to the IOM, SRs are rigorous protocol-driven literature reviews that summarize evidence by identifying, selecting, assessing, and synthesizing the findings of similar but separate studies. They can help clarify what is known and not known about the potential benefits and harms of drugs, devices, and other health care services. (2) In the past, many guidelines were supported only by narrative literature reviews. Going forward, to be included in NGC, they must be supported by an SR of the evidence. For NGC to determine whether an SR underpinned the guideline, the SR must be provided to NGC upon submission.
The feedback received from public comment, guideline developers, topic experts, the EB, and members of the Expert Panel enabled the NGC project team to use the IOM criteria and create a subset of criteria required for an SR (2) that underpins a guideline. These criteria were deemed the most critical to implement initially in the NGC revised inclusion criteria. Starting in June 2014, to meet NGC's revised inclusion criteria the following items related to the SR, listed under inclusion criterion 3 as items a-e (shown below), should be present in the guideline or its supporting documents (e.g., SR, evidence report, or technology assessment).

1. An explicit statement that the clinical practice guideline was based on a systematic review.
2. A description of the search strategy that includes a listing of database(s) searched, a summary of search terms used, the specific time period covered by the literature search including the beginning date (month/year) and end date (month/year), and the date(s) when the literature search was done.
3. A description of study selection that includes the number of studies identified, the number of studies included, and a summary of inclusion and exclusion criteria.
4. A synthesis of evidence from the selected studies, e.g., a detailed description or evidence tables.
5. A summary of the evidence synthesis (see 3d above) included in the guideline that relates the evidence to the recommendations, e.g., a descriptive summary or summary tables.

It might be helpful to note the difference between sub-criteria d and e.

• Sub-criterion d is a synthesis of the evidence which might be an extensive discussion of the evidence or evidence tables presented in the SR (see Example 1 below).
• Sub-criterion e is a brief summary of the information presented in the evidence synthesis or evidence tables (see highlighted text in Example 2 below).

In the event that the published evidence is insufficient to support a guideline's recommendations, the related SR should identify specific gaps in the evidence base for some of the guideline's recommendations. In that case, NGC will not exclude the guideline (on the grounds of not meeting sub-criteria d and e. An example of this situation is when evidence is insufficient to address all the clinical questions specified in the SR that are also relevant to the guideline.
Assessing Benefits and Harms
In addition to an SR, the IOM determined that a clinical practice guideline should include an assessment of the benefits and harms of alternative care options. (1) In revising NGC's inclusion criteria, the NGC project team determined that this criterion would be fulfilled by documentation demonstrating that the recommendations in the guideline were developed taking into account an assessment of benefits and harms of recommended care and alternative care options. NGC prefers that the guideline or supporting documents list and describe the potential benefits and harms of recommended care and alternative care options. However, having a statement in the guideline describing briefly the process used to assess benefits and harms will be deemed sufficient. It is expected that both benefits and harms will have been assessed in the systematic review underpinning the guideline recommendations. An acceptable example would be use of a recommendation rating scheme that takes into account the balance of benefits and harms, such as the one designed by the U.S. Preventive Services Task Force. (3)
Implementation of Revised Inclusion Criteria
The revised NGC inclusion criteria, posted on the NGC Web site on June 3, 2013, are scheduled to go into effect starting in June 2014. Guideline developers who are currently able to meet these criteria now are strongly encouraged to do so. Based on the NGC's initial pilot study, many guideline developers included in NGC have already begun changes to conform to the IOM standards; the June 2014 deadline provides an additional timeframe for developers who intend to conform to the revised inclusion criteria.
Through May 2014, guidelines submitted for consideration to NGC will be evaluated against the original inclusion criteria. If accepted, these guidelines and those already represented in NGC will remain posted until they are updated by the developer, withdrawn by the developer, or withdrawn because they no longer meet the requirement for currency (≥5 years). Therefore, by June 2019, all guidelines in the NGC will meet the revised inclusion criteria.
Next Steps
As mentioned previously, the IOM report also detailed a set of eight standards for clinical practice guidelines. In addition, the report identified NGC as a vehicle to implement these standards by specifically recommending to AHRQ that NGC "provide a clear indication of the extent to which CPGs submitted to it adhere to standards for trustworthiness." Over the next few years NGC plans to incorporate an assessment of the extent to which guidelines meeting the revised inclusion criteria adhere to the IOM standards and will engage stakeholders in that effort.
NGC is one of the world's few sources of freely accessible guideline information and plans to remain a critical publicly available resource for health care providers and other health care decision makers. Despite the challenges that the guideline community will undoubtedly face during the transition, this initiative will result in a greater proportion of higher quality and trustworthy guidelines available at NGC. It is our shared belief that these changes are timely and will help improve patient care quality and safety.

---

Authors
The following members of the NGC Project team authored or provided feedback for this commentary:
Primary authors: J. Jane S. Jue MD MSc (ECRI Institute), Paul G. Shekelle MD PhD MPH (RAND), and Kathleen N. Lohr PhD (RTI)
Reviewers: Jean Slutsky, PA MSPH (AHRQ), Mary P. Nix, MS(ASCP)SBB (AHRQ), Vivian Coates, MBA (ECRI Institute) and Lisa T. Haskell, MS, OTR/L (ECRI Institute)
Potential Conflicts of Interest
Dr. Jue is employed by ECRI Institute and is a member of the NGC/NQMC Editorial Board (EB). Dr. Shekelle and Dr. Lohr are both members of the NGC/NQMC EB. All three also perform work for AHRQ's Evidence-based Practice Centers.
Ms. Coates and Ms. Haskell also work for ECRI Institute.

References

1. Institute of Medicine. Graham R, Mancher M, Wolman DM, Greenfield S, Steinberg E, editor(s). Clinical practice guidelines we can trust. Washington (DC): National Academies Press; 2011. 2p. Also available: http://www.iom.edu/Reports/2011/Clinical-Practice-Guidelines-We-Can-Trust/Standards.aspx .
2. Institute of Medicine. Committee on Standards for Systematic Reviews. Eden J, Levit L, Berg A, Morton S, editor(s). Finding what works in health care: standards for systematic reviews. Washington (DC): National Academies Press; 2011 Mar 23. Also available: http://www.iom.edu/Reports/2011/Finding-What-Works-in-Health-Care-Standards-for-Systematic-Reviews.aspx .
3. Grade definitions [internet]. Rockville (MD): U.S. Preventive Services Task Force (USPSTF). [accessed 7/9/13]. Also available at: http://www.uspreventiveservicestaskforce.org/uspstf/grades.htm#brec .

---

Example 1: Evidence Synthesis

CPAP At the time of the previous report, there were few prospective studies on continuous positive airway pressure (CPAP) use in children, although several retrospective studies indicated that CPAP was efficacious in the treatment of pediatric obstructive sleep apnea syndrome (OSAS). Since that time, there have been at least 7 recent studies evaluating the use of positive airway pressure (PAP) in children and adolescents who have OSAS. One of these was a randomized trial with low power (level II),142 and others were case series without controls (level IV). A descriptive study examined the use of behavioral intervention in improving CPAP adherence.143 In addition, a level III study described use of a high-flow nasal cannula as an alternative to CPAP.144 In contrast to the previous guidelines, several of the current studies obtained objective evaluation of CPAP adherence by downloading usage data from the CPAP device. In most studies, CPAP therapy was instituted for persistent OSAS after adenotonsillectomy (AT); in many cases, the patients had additional risk factors for OSAS, such as obesity or craniofacial anomalies. A multicenter study (level II) evaluated PAP in 29 children who were randomly assigned either CPAP or bilevel positive airway pressure (BPAP).142 Patients demonstrated significant improvement in sleepiness, snoring, apnea hypopnea index (AHI), and oxyhemoglobin saturation while using PAP during the 6-month follow-up period. However, approximately one-third of patients dropped out, and of those who used PAP, objective adherence was 5.3 ± 2.5 hours/night. Parents overestimated the hours of PAP use compared with the devices' actual objective recordings of use. There was no significant difference in adherence between the CPAP and BPAP groups. A retrospective chart review of 46 children started on PAP for OSAS that persisted after AT also showed significant improvement in symptoms of OSAS as well as in polysomnographic parameters (level IV).145 Seventy percent of patients were considered adherent. Parental report of adherence was most divergent from the machines' recording in the least adherent patients. More than one-half of the children had complicating factors, such as Down syndrome and Prader-Willi syndrome.145 Another study of a heterogeneous group of patients displayed varying CPAP adherence, with 31 of 79 children showing continued CPAP use (level IV).146 A small, nonblinded retrospective study (level IV) suggested that adherence to CPAP could be improved with behavioral techniques if the family accepted the interventions.143 A retrospective review described 9 children who successfully used BPAP in the intensive care setting because of respiratory compromise after AT.147 Another retrospective review described the successful use of CPAP in 9 patients of a heterogeneous group of 18 children aged < 2 years.148 A nonrandomized, prospective level III study of 12 children who had OSAS treated in the sleep laboratory with a high-flow open nasal cannula system as an alternative to formal CPAP demonstrated an improvement in oxyhemoglobin saturation and arousals, but not AHI, compared with baseline.144 There was a decrease in sleep efficiency with the cannula compared with baseline. Long-term use and use in the home situation were not assessed.

A nonrandomized, prospective level III study of 12 children who had OSAS treated in the sleep laboratory with a high-flow open nasal cannula system as an alternative to formal CPAP demonstrated an improvement in oxyhemoglobin saturation and arousals, but not AHI, compared with baseline.¹⁴⁴ There was a decrease in sleep efficiency with the cannula compared with baseline. Long-term use and use in the home situation were not assessed. In summary, several studies (levels II–IV) have confirmed earlier data demonstrating that nasal CPAP is effective in the treatment of both symptoms and polysomnographic evidence of OSAS, even in young children. However, adherence can be a major barrier to effective CPAP use. For this reason, CPAP is not recommended as first-line therapy for OSAS when AT is an option. However, it is useful in children who do not respond adequately to surgery or in whom surgery is contraindicated. Patient and family preference may also be a consideration (e.g., in families with religious beliefs against surgery or blood transfusions). Objective assessment of CPAP adherence is important because parental estimates of use are often inaccurate. If the patient is nonadherent, then attempts should be made to improve adherence (e.g., by addressing adverse effects, by using behavior modification techniques), or the patient should be treated with alternative methods. A study described in the previous report noted that CPAP pressures change over time in children, presumably because of growth and development.¹⁴⁹ Therefore, it is recommended that CPAP pressures be periodically reassessed in children. At this time, data are insufficient to make a recommendation on the use of high-flow, open nasal cannula systems.
Areas for Future Research

• Efficacy of CPAP use as a first-line treatment of obese children.
• Determinants of CPAP adherence and ways to improve adherence.
• Long-term effects of CPAP, particularly on the development of the face, jaw, and teeth.
• Changes in CPAP pressure over time, and the frequency with which this needs to be monitored.
• Development of pediatric-specific devices and interfaces.

Source: Technical Report: diagnosis and management of childhood obstructive sleep apnea syndrome. Pediatrics 2012 Sep;130(3):e714-e755. Electronic copies: Available from the American Academy of Pediatrics (AAP) Policy Web site .

Example 2: Summary of Evidence Synthesis (see highlighted text below)

Key Action Statement (KAS) 6: CPAP Clinicians should refer patients for CPAP management if symptoms/signs or objective evidence of OSAS persists after adenotonsillectomy or if adenotonsillectomy is not performed. (Evidence Quality: Grade B, Recommendation Strength: Recommendation.) Evidence Profile KAS 6 Aggregate evidence quality: B Benefit: Improve OSAS and accompanying symptoms and sequelae. Harm: Expense, time, anxiety; parental sleep disruption; nasal and skin adverse effects; possible midface remodeling; extremely rare serious pressure-related complications, such as pneumothorax; poor adherence. Benefits-harms assessment: Preponderance of benefit over harm. Value judgments: Panelists believe that CPAP is the most effective treatment of OSAS that persists postoperatively and that the benefits of treatment outweigh the adverse effects. Other treatments (e.g., rapid maxillary expansion) may be effective in specially selected patients. Role of patient preferences: Other treatments may be effective in specially selected patients. Exclusions: Rare patients at increased risk of severe pressure complications. Intentional vagueness: None. Policy level: Recommendation. CPAP therapy is delivered by using an electronic device that delivers air at positive pressure via a nasal mask, leading to mechanical stenting of the airway and improved functional residual capacity in the lungs. There is no clear advantage of using bilevel pressure over CPAP.15 CPAP should be managed by an experienced and skilled clinician with expertise in its use in children. CPAP pressure requirements vary among individuals and change over time; thus, CPAP must be titrated in the sleep laboratory before prescribing the device and periodically readjusted thereafter. Behavioral modification therapy may be required, especially for young children or those with developmental delays. Objective monitoring of adherence, by using the equipment software, is important. If adherence is suboptimal, the clinician should institute measures to improve adherence (such as behavioral modification, or treating side effects of CPAP) and institute alternative treatments if these measures are ineffective. Source: Marcus CL, Brooks LJ, Draper KA, Gozal D, Halbower AC, Jones J, Schechter MS, Sheldon SH, Spruyt K, Ward SD, Lehmann C, Shiffman RN. Diagnosis and management of childhood obstructive sleep apnea syndrome. Pediatrics. 2012 Sep;130(3):576-84. [17 references] PubMed

Back to top