National Guideline Clearinghouse | Expert Commentaries: Consensus-based Selection Criteria for Accountability Measures: Viewpoint from the National Guideline Clearinghouse/National Quality Measures Clearinghouse Editorial Board

full-text:
National Guideline Clearinghouse Expert Commentaries: Consensus-based Selection Criteria for Accountability Measures: Viewpoint from the National Guideline Clearinghouse/National Quality Measures Clearinghouse Editorial Board

Consensus-based Selection Criteria for Accountability Measures: Viewpoint from the National Guideline Clearinghouse/National Quality Measures Clearinghouse Editorial Board

By: NGC/NQMC Editorial Board: Richard C. Hermann, MD, MS (co-chair), Paul G. Shekelle, MD, MPH, PhD (co-chair), Ethan Balk, MD, MPH, JiWon Jane S. Jue, MD, MSc, Jerod M. Loeb, PhD, Kathleen N. Lohr, PhD, and Eric C. Schneider, MD, MSc

Note: This is the second consensus viewpoint by the Editorial Board of the National Guideline Clearinghouse/National Quality Measures Clearinghouse (NGC/NQMC). (1) Serving in an advisory capacity to NGC/NQMC, the Editorial Board brings diverse expertise to the development and implementation of practice guidelines and quality measures in health care.

Use of quality measures to assess health care delivered by hospitals, health systems, and other providers in the United States (U.S.) has grown in recent years and will likely become increasingly common in the years to come. The purpose of this viewpoint by NGC/NQMC's Editorial Board is to endorse a proposal for consensus-based selection criteria for quality measures used for the purpose of accountability.

In 2010, Chassin et al. called for stakeholders in the quality measurement enterprise to adopt rigorous selection criteria for process measures used to achieve accountability—i.e., for public reporting, payment, or accreditation. (2) (Those authors noted that outcome measures were important but raise further issues to be discussed separately.) Accountability measures, they proposed, should focus explicitly on maximizing net health benefits to patients. To advance this goal, the authors recommended that measures employed for accountability meet the following criteria: (2)

1. The measure has a strong evidence base showing that the care process improves outcomes.
2. The measure accurately captures whether the evidence-based care process has, in fact, been provided.
3. The measure addresses a care process that has few intervening processes that must occur before the improved outcome is realized.
4. Implementing the measure has little or no chance of inducing unintended adverse consequences.

Data on more than 2,000 quality measures inventoried in the NQMC provide insight into why explicit, well-defined selection criteria are needed to choose accountability measures. Stakeholder organizations developing or selecting measures for broad use generally agree conceptually about the characteristics that are desirable in a quality measure. Although their terminology may differ, most organizations group these characteristics under three domains reflecting a measure's clinical importance, scientific soundness, and feasibility. Organizations seek to optimize these properties in developing and selecting measures.

For each quality measure, NQMC provides its users with information on many of these attributes. Examples of these characteristics (and the domains to which they belong) in NQMC include:

• The measure's basis in research evidence (Clinical Importance)
• Study of the measure's reliability and validity (Scientific Soundness)
• Sources of data needed to construct the measure, reflecting its data-collection burden (Feasibility).

Although organizations agree conceptually about desirable attributes, the measures they develop or select vary widely with regard to these properties. In addition to the inherent difficulty of developing good measures, other reasons contribute to this variation. First, simultaneously optimizing the clinical importance, scientific soundness, and feasibility of a measure may not be possible. Trade-offs frequently need to be made in one domain to achieve the goals of another. For example, measures that address important, evidence-based processes often require clinically richer, but more burdensome, data sources. As a result, measures of greater clinical importance can prove less feasible to implement. Second, selecting measures requires that organizations set priorities among attributes; those representing different stakeholders may not share the same priorities. Third, even when an attribute is widely considered a priority, organizations may have different thresholds for considering it fulfilled. An example is the attribute "evidence-based". Developers provide NQMC with information on research evidence supporting their measures. Across measures submitted, we see wide variation in the types of evidence provided, the aspects of a measure that the evidence supports, and the quality and quantity of studies that make up the relevant body of evidence.

This variability illustrates the need for explicit selection criteria for the evidence base of any measure considered for accountability purposes. The first of the four proposed criteria defines the type of evidence needed: evidence linking the process measured with improved outcomes. Further details will be needed that define the type and number of studies required for this criterion to be met. In an important step, the National Quality Forum has begun to develop criteria for evaluating and testing the scientific acceptability of quality measures. (3)

We endorse the development of criteria along the lines proposed by Chassin et al., even as we recognize that they may present challenges to the development of accountability measures of certain types of clinical processes and, possibly, for some specialty areas in health care.

The clinical processes most commonly evaluated in clinical trials are treatment interventions—predominantly pharmacological. Thus, measures assessing the performance of these interventions are those most likely to have an evidence base qualifying them as an accountability measure. Research studies are less likely to be available for other processes that may be essential for delivering an effective intervention, such as detection of the condition requiring treatment, access to the treatment, and continuity of care. The gap between the importance of these processes and the absence (or indirectness) of research evidence to support them may be a consideration when organizations try to operationalize criteria for accountability measures.

According to Chassin et al., a substantial proportion of current Joint Commission core measures presently used by most U.S. hospitals for accreditation meet all four of the criteria they propose. This welcome assessment, however, does not apply to all major areas of health care. For instance, only one of the seven measures in the Joint Commission's core measure set for inpatient psychiatry is based on research evidence from clinical trials. (4) Establishing criteria for accountability measures will challenge organizations to work even harder than in the past to ensure the development of measures that assess all major health care services.

These concerns should inform rather than delay development and implementation of uniform selection criteria for accountability measures. Through their linkage to reimbursement, accreditation, and other oversight functions, accountability measures are intended to influence patient outcomes. Criteria proposed for selecting accountability measures thus raise the same kinds of questions for quality measures that are routinely considered when evaluating a new clinical intervention: Has the change in a clinical process encouraged by a proposed accountability measure been studied and proven to provide benefit? Does the process measured have fidelity to the process shown to provide that benefit? And can these benefits be realized without causing undue harm?

---

Potential Conflicts of Interest
All Core Editorial Board members complete Conflict of Interest disclosure forms annually for the Agency of Healthcare Research and Quality (AHRQ). With regard to this viewpoint, Dr. Loeb notes he was an author on the 2010 Chassin et al., article. All other Core Editorial Board members declared no potential conflicts of interest with respect to this viewpoint.
References

1. NGC/NQMC Editorial Board. Promoting transparent and actionable clinical practice guidelines: viewpoint from the National Quality Measures Clearinghouse/National Guideline Clearinghouse (NQMC/NGC) Editorial Board. Available at: http://guideline.gov/expert/expert-commentary.aspx?id=24556.
2. Chassin MR, Loeb JM, Schmaltz SP, Wachter RM. Accountability measures -- using measurement to promote quality improvement. N Engl J Med 2010;363:683-688.
3. National Quality Forum. Measure Evaluation Criteria. January 2011. Available at: http://guideline.gov/disclaimer.aspx?redirect=http://www.qualityforum.org/docs/measure_evaluation_criteria.aspx . Accessed: August 7, 2011.
4. Goren JL, Parks JJ, Ghinassi FA, Milton CG, Oldham JM, Hernandez P, Chan J, Hermann RC. When is antipsychotic polypharmacy supported by research evidence? Implications for quality improvement. The Joint Commission Journal on Quality and Patient Safety 2008; 34(10):571-582.