National Guideline Clearinghouse | Expert Commentaries: NGC and NQMC Survey Responses Generate Enhancement and Growth

full-text ►
National Guideline Clearinghouse | Expert Commentaries: NGC and NQMC Survey Responses Generate Enhancement and Growth

July 1, 2013

NGC and NQMC Survey Responses Generate Enhancement and Growth

By: the National Guideline Clearinghouse and National Quality Measures Clearinghouse Teams
In November 2011, the Agency for Healthcare Research and Quality (AHRQ) launched a new user survey on the National Guideline Clearinghouse (NGC) and National Quality Measures Clearinghouse (NQMC) Web sites. The online survey, administered by ForeSee Results, Inc., helps AHRQ determine the extent to which NGC and NQMC are meeting the needs of their users and solicits ideas for future Web site and content enhancements. From November 2011 to October 2012, ForeSee surveys were completed by 1,564 NGC users and 280 NQMC users, many of whom provided constructive feedback. The average response rates for this time period were 46.7% for NGC and 42.6% for NQMC. Presented here are results from those user surveys with respect to who is using NGC and NQMC and their primary reasons for coming to the Web sites. Also considered is how the survey results have influenced current and planned enhancements to NGC and NQMC. Where relevant, we compare the most recent survey results with those of the 2008 NGC/NQMC user surveys described in User Perspectives: An Overview of the Clearinghouse Survey Results, which was published to the NGC and NQMC Web sites in June 2009. (1) In addition, in September 2011, AHRQ received the results of an independent evaluation of NGC from the perspective of key stakeholders such as guideline developers. (2) Again, where appropriate we will note how these evaluation results intersect with the most recent ForeSee Survey results and how they have affected Clearinghouse content and process.
Who Is Visiting NGC and NQMC?
The November 2011–October 2012 ForeSee survey results indicate that a wide variety of healthcare professionals and others turn to the NGC and NQMC Web sites as a primary information gathering resource. NGC survey takers were asked to pick from a list of 12 roles/job titles that best described them, e.g., Nurse/Nurse Practitioner, Consumer/General Public, Media/Press. NQMC provided a list 16 roles/titles from which to choose. Among those who returned the surveys, three professional groups had the strongest showings for both Web sites, namely clinicians (which included Physicians, Physician Assistants, Nurses, Nurse Practitioners, and other Health Care Professionals); researchers; and educators/students. Interestingly, among the clinician group, Nurses/Nurse Practitioners accounted for the single largest number of respondents for both Clearinghouses. From November 2011 to October 2012, 28% of the NGC surveys were completed by Nurses/Nurse Practitioners. For NQMC, these nursing professionals provided 25% of the responses during the same time period.
Although NGC and NQMC survey takers were offered different choices for roles/titles, seven (7) roles/titles were consistent across both Web sites and can be used for comparison. The figure below provides a comparison of these seven user groups that returned the NGC and NQMC surveys from November 2011 to October 2012, broken down by quarter as follows:

• Quarter 1 (Q1) = November 2011–January 2012
• Quarter 2 (Q2) = February–April 2012
• Quarter 3 (Q3) = May–July 2012
• Quarter 4 (Q4) = August–October 2012

Figure description: The title of the figure is "Distributions of Roles/Job Titles for NGC & NQMC Survey Respondents from Nov 2011–Oct 2012." The figure is a bar graph depicting the quarterly distribution of seven different roles/titles for respondents to the NGC and NQMC surveys administered from November 2011 to October 2012. The quarterly percentage value for each category of role/title is as follows: Consumer/General Public Quarter 1: 1% for NGC, 9% for NQMC; Quarter 2: 1% for NGC, 0% for NQMC; Quarter 3: 1% for NGC, 2% for NQMC; Quarter 4: 1% for NGC, 1% for NQMC. Educator/Student Quarter 1: 11% for NGC, 11% for NQMC; Quarter 2: 16% for NGC, 6% for NQMC; Quarter 3: 9% for NGC, 10% for NQMC; Quarter 4: 14% for NGC, 12% for NQMC. Health Administrator Quarter 1: 1% for NGC, 6% for NQMC; Quarter 2: 1% for NGC, 2% for NQMC; Quarter 3: 1% for NGC, 8% for NQMC; Quarter 4: 3% for NGC, 9% for NQMC. Health Researcher/Analyst Quarter 1: 6% for NGC, 16% for NQMC; Quarter 2: 4% for NGC, 4% for NQMC; Quarter 3: 6% for NGC, 16% for NQMC; Quarter 4: 6% for NGC, 16% for NQMC. Nurse/Nurse Practitioner Quarter 1: 27% for NGC, 10% for NQMC; Quarter 2: 26% for NGC, 23% for NQMC; Quarter 3: 27% for NGC, 36% for NQMC; Quarter 4: 32% for NGC, 31% for NQMC. Other Health Care Professional Quarter 1: 5% for NGC, 10% for NQMC; Quarter 2: 6% for NGC, 2% for NQMC; Quarter 3: 6% for NGC, 6% for NQMC; Quarter 4: 6% for NGC, 3% for NQMC. Physician/Physician Assistant Quarter 1: 29% for NGC, 9% for NQMC; Quarter 2: 25% for NGC, 23% for NQMC; Quarter 3: 34% for NGC, 12% for NQMC; Quarter 4: 22% for NGC, 15% for NQMC.

Although the roles/titles provided in the 2008 NGC/NQMC user surveys differed from those provided in the surveys administered from November 2011–October 2012, it is possible to draw some comparisons regarding representation of clinicians. In NGC, for example, over 60% of respondents to both the 2008 and 2011–2012 surveys were clinicians. For NQMC, however, there has been a marked increase in the number of clinicians using the Web site. In 2008, approximately 25% of NQMC survey respondents identified as clinicians. For the surveys fielded from November 2011–October 2012, that percentage rose to 43% as illustrated in the chart below.

Figure description: Chart description: The title of the chart is "Comparison of Clinician Response to the 2008 NGC/NQMC Survey and the 2011-2012 NGC/NQMC Survey." The chart is a bar graph comparing the distribution of clinician response received by NGC and NQMC to the 2008 User Survey and the 2011–2012 User Survey, respectively. The proportion of response is provided for three categories of Clinicians: 1) Nurses/Nurse Practitioners, 2) Physicians, and 3) Other Health Care Professionals for both NGC and NQMC. In addition, the combined total 2008 response for all three clinician groups is provided for comparison to the same metric for the 2011–2012 survey. Nurses/Nurse Practitioners (2008): 30% for NGC, 11% for NQMC; Nurses/Nurse Practitioners (2011–2012): 28% for NGC, 25% for NQMC. Physicians (2008): 26% for NGC, 12% for NQMC; Physicians (2011–2012): 28% for NGC, 15% for NQMC. Other Health Care Professionals (2008): 10% for NGC, 2% for NQMC; Other Health Care Professionals (2011-2012): 6% for NGC, 3% for NQMC. Combined Clinician Response (2008): 66% for NGC, 25% for NQMC; Combined Clinician Response (2011–2012): 62% for NGC, 43% for NQMC.

Why Do Users Come to the NGC and NQMC Web sites?
During the November 2011–October 2012 time period covered by the ForeSee Survey, NGC and NQMC users were asked to indicate their primary reasons for visiting the respective Web sites. Four primary reasons were cited most often, although their order of importance to NGC and NQMC users varied. As shown in the table below, survey takers reported they came to the Clearinghouse Web sites 1) to find a specific guideline or quality measure, 2) to determine the guidance or quality measures available on a specific topic, 3) to perform research, 4) to stay current by reviewing newer guidelines or quality measures.

Top Reasons Given for Visiting the NGC or NQMC Web Sites		Average for 2011–2012
		NGC	NQMC
1.	Find a specific clinical practice guideline(s) or quality measure(s)	49%	33%
2.	Determine the guidance/quality measures available on a specific topic	14%	24%
3.	For research	15%	21%
4.	Review the newest guidance/quality measures to stay current	6%	9%

The figure below provides a snapshot of the survey responses to this question over the four quarters of 2011–2012 that the ForeSee survey was fielded.

Figure description: The title of the figure is "Top Reasons Given for Visiting NGC and NQMC Web sites from Nov 2011–Oct 2012." The figure is a bar graph depicting the quarterly distribution of the top four reasons for visiting the Web sites given by respondents to the ForeSee survey fielded from November 2011–October 2012. The quarterly percentage value for each reason given is as follows: Find a specific clinical practice guideline(s) or quality measure(s) Quarter 1: 50% for NGC, 23% for NQMC; Quarter 2: 50% for NGC, 47% for NQMC; Quarter 3: 51% for NGC, 30% for NQMC; Quarter 4: 43% for NGC, 30% for NQMC. Determine the guidance/quality measures available on a specific topic Quarter 1: 13% for NGC, 27% for NQMC; Quarter 2: 16% for NGC, 22% for NQMC; Quarter 3: 11% for NGC, 24% for NQMC; Quarter 4: 14% for NGC, 21% for NQMC. For research Quarter 1: 15% for NGC, 25% for NQMC; Quarter 2: 14% for NGC, 17% for NQMC; Quarter 3: 13% for NGC, 22% for NQMC; Quarter 4: 19% for NGC, 18% for NQMC. Review the newest guidance/quality measures to stay current Quarter 1: 7% for NGC, 9% for NQMC; Quarter 2: 4% for NGC, 4% for NQMC; Quarter 3: 9% for NGC, 8% for NQMC; Quarter 4: 5% for NGC, 15% for NQMC.

The 2008 NGC/NQMC user survey did not collect data on the reasons for visiting NGC and NQMC. However, the independent evaluation of NGC report provided to AHRQ in 2011 noted that key stakeholders such as guideline developers and researchers turn to the NGC repository to assist with research for guideline development, to locate comprehensive reviews of other current guidelines on a specific topic, and to augment the guideline dissemination process. This evaluation found that developers and other healthcare professionals review three to four sources when conducting research, yet report that the NGC Web site is their primary source to find guidelines. What Changes to NGC/NQMC Are Users' Comments Driving?
In 2008, NGC and NQMC survey response reinforced AHRQ's decision to upgrade the appearance and functionality of the Web sites, notably by moving to a new platform and adding a powerful search engine. From November 2011–October 2012, the ForeSee Surveys results continued to inform current and ongoing efforts to improve the Search and Navigation of the Web sites. Some of the changes made to improve the experience for NGC and NQMC users include the following:

• Improved design and layout of the home pages by relocating Find, Learn, and Compare sections of both the NQMC and NGC Home pages to call attention to areas of the Web sites that were more difficult to discover through the standard navigation, such as Browse options, Tutorial pages for quality measures, guideline resource pages, and the Compare features.
• Updated Search Tips/About Search Web pages.
• Enhanced Search function to allow searching by vocabulary codes (e.g., International Classification of Diseases, Ninth Revision, Clinical Modification [ICD-9-CM]), and refining the search results.

The above elements were also important to the key stakeholders who responded for the 2011 NGC Evaluation. In addition, two-thirds of these stakeholders stated they would be inclined to use "Subject-specific email alerts" if available.
In July 2012, AHRQ launched the Personalization Feature, which allows users to find a record of their most recent searches and recently viewed summaries simply by accessing the NGC or NQMC home pages. For readers interested in more individualized navigation, there is the option to create a My NGC or My NQMC account and select summaries and organizations as "favorites." Those who create these free accounts are notified by email whenever a relevant summary update or addition is posted to the Web sites or when a summary of interest is withdrawn. In addition, users can elect to be notified when new summaries or updates are available on up to 33 NGC topics and 46 NQMC topics. Ideally My NGC and My NQMC will streamline the visitor's navigation and search experience and provide an opportunity to maintain a virtual "filing cabinet" from which to draw resources previously viewed. AHRQ has provided a video Tutorial on personalization to walk users through this new feature.
Finally, the 2011 NGC Evaluation provided guideline researchers and developers with the opportunity to suggest improvements which are now planned enhancements, including revising NGC's inclusion criteria to encourage development of clinical practice guidelines that reflect the recommendations of the May 2011 Institute of Medicine Report on Trustworthy Guidelines . (3) As noted at the Web site, NGC posted the revised Inclusion Criteria in June 2013 and will begin implementing them a year after that.
AHRQ is translating the input of those who visit the Web sites and take the surveys into changes designed to facilitate the user experience. NGC and NQMC users who have not yet taken the survey, or have not taken it for some time, are urged to do so. It is a direct way to assist AHRQ's efforts to improve the Web sites by making them more responsive. We're listening — let us hear from you!

---

References

1. The National Guideline and National Quality Measures Clearinghouse Management Team. User perspectives: an overview of the clearinghouse survey results. In: National Guideline Clearinghouse (NGC) [Web site]. Rockville (MD): Agency for Healthcare Research and Quality (AHRQ); [cited 2013 Mar 29]. Available: http://www.guideline.gov.
2. Tregear M. Evaluation of the National Guideline Clearinghouse^™. PowerPoint. In: AHRQ Annual Conference. Laurel (MD): AFYA Inc. 2011 Sep.
3. Institute of Medicine. Graham R, Mancher M, Wolman DM, Greenfield S, Steinberg E, editor(s). Clinical practice guidelines we can trust. Washington (DC): National Academies Press; 2011. 2 p. Available: http://www.iom.edu/Reports/2011/Clinical-Practice-Guidelines-We-Can-Trust/Standards.aspx .