Volume 21, Number 9—September 2015
Improving Accuracy of Influenza-Associated Hospitalization Rate Estimates
In the United States, surveillance for influenza-associated hospitalizations relies on laboratory-confirmed diagnostic testing (1–3). Influenza testing modalities have expanded from traditional viral culture to include rapid influenza diagnostic tests (RIDTs) and molecular assays, such as reverse transcription PCR (RT-PCR) (4,5). RIDTs are point-of-care tests that provide results within 30 minutes; however, with reported sensitivities of 10%–80%, negative test results can be unreliable (6–9). RT-PCR exceeds viral culture in sensitivity for detecting influenza, but its widespread use is limited by cost and complexity of the assay (10,11).
Researchers have examined rates of influenza-associated hospitalization during different influenza seasons (1,2,12,13). However, comparing rates between seasons can be inaccurate without accounting for changes in the sensitivity of diagnostic testing used. In particular, after the 2009 influenza A(H1N1) pandemic, hospitals and state public health laboratories expanded diagnostic capabilities with high-sensitivity molecular assays to better detect influenza viruses and other respiratory pathogens (5). Particularly for nationally based surveillance, the use of different testing platforms by health care facilities and the variability in sensitivity of these diagnostic tests could lead to underestimation of rates of influenza-associated hospitalization and limit comparisons of severity across influenza seasons (3,4,6,7).
We used data from the Centers for Disease Control and Prevention (CDC) Influenza Hospital Surveillance Network (FluSurv-NET) from the 2003–2013 influenza seasons (3,14). FluSurv-NET conducts population-based surveillance for laboratory-confirmed influenza-associated hospitalizations among children <18 years of age (since the 2003–04 influenza season) and adults (since the 2005–06 influenza season). The FluSurv-NET system and protocol have been described previously (Technical Appendix[PDF - 313 KB - 8 pages]) (1,3,15).
CDC determined that data collected through FluSurv-NET were for routine public health surveillance and not subject to institutional review board approval for human research protections. Participating sites submitted the surveillance protocol to their state and local institutional review boards for review.
Case Definition and Data Collection
A case of influenza-associated hospitalization was defined as hospitalization of a catchment-area resident who was hospitalized in a catchment-area hospital during a designated influenza season (October 1–April 30) with a laboratory-confirmed influenza test within 14 days before or 3 days after hospital admission. Laboratory-confirmed influenza was defined as a positive result from RT-PCR, viral culture, direct fluorescent antibody staining (DFA), or RIDT or a positive result for an unspecified laboratory test documented in the medical chart. RT-PCR could be performed at the participating hospital or at the state public health laboratory depending on test availability in the hospital laboratory. The frequency of identified cases by diagnostic test type (observed case count) by patient age and by influenza season was evaluated. When an identified case had >1 type of positive influenza test, we used the test type with the highest sensitivity—RT-PCR, viral culture, DFA, RIDT (ordered from highest to lowest sensitivity)—for the analysis. If an identified case had no other test type and a positive result from an unspecified laboratory test documented in the medical chart, we assumed 100% sensitivity for that test.
Diagnostic Test Sensitivity
We reviewed the literature to obtain sensitivity ranges for influenza diagnostic tests. We searched PubMed with a strategy containing search terms for influenza disease or virus combined with search terms for RT-PCR, viral culture, DFA, and RIDTs and search terms for sensitivity. Search terms for influenza were as follows: “influenza, human” [Medical Subject Heading (MeSh)] OR “influenza A virus” [MeSh] OR “influenza B virus” [MeSh] OR “influenza” or “flu.” Search terms for the tests included “RT-PCR,” “reverse transcription polymerase chain reaction,” “culture,” direct florescent antibody,” “DFA,” “rapid diagnostic test.” Search terms for clinical sensitivity included “sensitivity,” “test characteristics,” “diagnostic test characteristics,” and “test performance characteristics.” We hand-searched bibliographies of included studies and recent narrative reviews of influenza diagnostic tests for additional relevant studies. We included only studies describing the clinical performance of the different diagnostic test types and did not use the manufacturer’s package insert or subtype-specific assessments. We identified studies describing the clinical sensitivities of different diagnostic test types in the system and focused on the periods before and after the 2009 influenza pandemic. The diagnostic reference standard used in the studies was either viral culture or RT-PCR. The sensitivity of influenza diagnostic tests varies by age because of factors, such as differences in viral shedding (16–19); therefore, we collected characteristics on each test type by age group (children <18 years, adults 18–64 years, and adults >65 years). Because we categorized the influenza diagnostic test type by method, we preferentially selected studies, such as meta-analyses, that could evaluate multiple brands of a particular influenza diagnostic test. We attempted to select studies based in hospitalized or emergency department settings when available.
We abstracted sensitivity values from the literature by age group as a range of minimum to maximum values or as a point estimate with a 95% CI, depending on how the data were reported (Technical Appendix[PDF - 313 KB - 8 pages] Table 1). To create a summary empirical distribution across all included studies for each age group and test type, we applied bootstrap techniques (20). All ranges were evaluated as a single observation and equally weighted in the analysis. We resampled 1,000 times from each reported distribution of test sensitivity (uniform distribution when only a minimum and maximum sensitivity were reported or a normal distribution when the midpoint and 95% CI were reported). To summarize the resulting empirical distribution, we calculated a median estimate and 95% CI for each diagnostic test type by age group.
We calculated rates of influenza-associated hospitalization per 100,000 population using the National Center for Health Statistics (NCHS) population estimates for the counties in the surveillance catchment area. We calculated observed rates per 100,000 population by age group for each season using the observed case count and dividing it by the NCHS population estimate for that age group and influenza season. To adjust the observed hospitalization rates for test sensitivity, we used the following formula to estimate an adjusted case count by age group for each diagnostic test:
(adjusted case count)test = (observed case count)test × (1/sensitivitytest)
We calculated the total adjusted case count for a season and age group by summing the test-specific adjusted case counts. Finally, we calculated adjusted rates per 100,000 population by dividing the total adjusted case counts by the NCHS population estimate for that age group and season.
To reflect the previously described distribution of test sensitivity, this series of calculations was performed within the previously described bootstrap for each resampled value of test sensitivity. Reported here are the median estimate and 95% CI for each season and age group. All analyses were performed in SAS version 9.3 (SAS Institute, Cary, NC, USA).
During 2003–2013, the distribution of influenza diagnostic tests among identified cases changed, particularly after the 2009 pandemic (Figure 1). Before 2009, RIDTs were the most common test type, accounting for ≈70% of cases identified in FluSurv-NET. After the 2009 pandemic, RT-PCR became the most frequent test type for all age groups (Technical Appendix[PDF - 313 KB - 8 pages] Figure). The proportion of RT-PCRs among identified cases increased from <10% before 2009 to ≈70% after 2009.
The Table summarizes the diagnostic test performance characteristics by age group obtained from the literature review and the bootstrap analysis. Influenza diagnostic tests are generally most sensitive when performed on specimens from children <18 years; RT-PCR has the highest sensitivity in this age group (sensitivity estimate 95%, 95% CI 82%–98.7%). The sensitivity of influenza diagnostic tests in adults 18–64 years is similar to that in children <18 years except for RIDTs, which are less sensitive in this age group. Overall, influenza diagnostic tests have poor sensitivity in adults >65 years. RIDTs have the lowest sensitivity in this age group (sensitivity estimate 20.1%, 95% CI 8.8%–41.4%), and although RT-PCR is more sensitive in this age group than are other test types, the midpoint sensitivity estimate for RT-PCR is still <90%. DFA sensitivity appears higher than that of culture and RIDTs in this age group; however, these results were extrapolated from studies that primarily included a younger population (27,28). Additionally, DFA was seldom performed in this age group (Technical Appendix[PDF - 313 KB - 8 pages] Table 1).
Observed and adjusted rates of influenza-associated hospitalization per 100,000 population varied by season for all age groups, indicating a particular influenza season’s severity (Figure 2; Technical Appendix[PDF - 313 KB - 8 pages] Table 2). Observed hospitalization rates ranged from 7.3 during 2011–12 to 50.5 during 2009–10 for children <18 years of age, 3.0 during 2006–07 to 30.3 during 2009–10 for adults 18–64 years, and 13.6 during 2008–09 to 181.8 during 2012–13 for adults >65 years. Hospitalization rates were highest for adults >65 years of age and lowest for adults 18– 64 years of age.
Adjusting for test sensitivity increased hospitalization rates across all age categories (Figure 2). Adjusted rates showed that the number of hospitalizations was higher than previously reported in all seasons for all age groups, regardless of the severity of the season; however, rates increased more in earlier seasons. The magnitude of hospitalizations during severe influenza seasons during earlier surveillance years (2003–04 for children <18 years and 2007–08 for adults >65 years) increased substantially after the adjustments, better highlighting the morbidity associated with influenza infections during those earlier seasons (Technical Appendix[PDF - 313 KB - 8 pages] Table 3). The wide CIs in the adjusted rates for adults >65 years in all seasons reflects the poor sensitivity of influenza diagnostic tests in this age group.
When adjusted for test sensitivity, observed rates of hospitalization underestimated influenza-associated hospitalization rates for all age groups but especially for adults >65 years (Figure 3). Observed hospitalization rates underestimated adjusted rates by ≈30% during 2003–2008 versus 15% during 2009–2013 for children <18 years; by 40% during 2005–2008 versus 20% during 2009–2013 for adults 18–64 years; and by 75% during 2005–2008 versus 55% during 2009–2013 for adults >65 years.
Adjusting for influenza diagnostic test sensitivity reveals that observed rates of influenza-associated hospitalization currently reported from surveillance data underestimate influenza-associated hospitalizations, particularly for adults >65 years. The increased use of high sensitivity tests, such as RT-PCR, after 2009 for all age groups has substantially reduced the degree of underestimation for children <18 years and adults 18–64 years of age. However, FluSurv-NET surveillance data still underestimate rates of influenza-associated hospitalization by 55% for adults >65 years without adjustments for influenza test sensitivity. Accurate influenza diagnostic testing can have a major impact on monitoring and guiding public health interventions for the control, prevention, and treatment of influenza.
Studies relying on administrative data alone to estimate rates of influenza-associated hospitalization may underestimate rates because influenza is seldom listed as a discharge diagnosis without laboratory-confirmed testing (32–35). The best way to ascertain influenza-associated hospitalization incidence rates in real time is to perform prospective surveillance that uses the most sensitive testing criteria (i.e., RT-PCR). Indeed, studies that have relied on active surveillance and testing, most often with RT-PCR, can improve estimates of influenza-associated hospitalization rates (32,34–38); however, as our study shows, failing to account for diagnostic test sensitivity can result in continued underestimation of influenza-associated hospitalizations, especially among older adults. Influenza diagnostic tests, regardless of test type, have poorer sensitivity in older adults than in younger persons. The methods used in our study account for case underascertainment resulting from varying testing sensitivity and provide opportunities to better compare the severity among different influenza seasons and age groups.
Although the degree of underestimation for the hospitalization rates reported here may seem high, the adjusted rates per 100,000 population for adults>65 years of age of 155.2 during 2005–06, 67.3 during 2006–07, and 314.4 during 2007–08 are still lower than the rates estimated in the literature using models of administrative data (rates per 100,000 for adults >65 years were 291.9 during 2005–06, 136.9 during 2006–07, and 380.9 during 2007–08) (13). This difference may be due to patients who had an influenza-associated hospitalization but were missed by our system because they were not tested. Nevertheless, sensitivity adjustments enable us to further improve the accuracy of estimated rates of influenza-associated hospitalization and provide timely results that account for changes in diagnostic test sensitivity over time.
Our analysis has limitations. First, our sensitivity adjustments do not reflect differences in detection by type or subtype of influenza viruses. Although this is a limitation of our analysis because diagnostic test sensitivity can vary on the basis of type or subtype of influenza viruses (7,21,23), differences in sensitivity based on type or subtype would have been difficult to assess, especially before the 2009 pandemic, when those data were not routinely available because of lack of RT-PCR or viral culture data in our network. Second, we did not adjust for any further variation in sensitivity measures by individual diagnostic test, but sensitivity measurements obtained from the literature enabled more generalizable estimates across the entire surveillance system. Third, we did not account for diagnostic test specificity. Influenza diagnostics tests generally have high specificities ranging from 96% to 100% regardless of age group (7,10,11), and the specificities of the tests used in the surveillance system have remained relatively constant over the study period, unlike test sensitivity. Although accounting for false-positive test results might decrease our estimates, the impact on overall rates would be minimal because test sensitivity covered much wider ranges. Fourth, although we conducted an extensive literature review, we did not conduct a formal systematic literature review. Additionally, published data on test sensitivity in adults >65 years of age are sparse; however, most studies demonstrate the poor sensitivity of influenza diagnostic tests in this particular population. Studies with larger sample sizes that focus on adults >65 years of age would improve understanding of diagnostic test sensitivity in this population with greater precision than is currently known. Finally, diagnostic testing in FluSurv-NET depends on a health care provider’s decision to order diagnostic testing on an individual patient. Therefore, we were unable to account for patients with influenza who were not tested. Multipliers based on the probability of an influenza-infected patient’s being tested have been estimated from the 2010–11 and 2011–12 seasons to correct for underascertainment (39; Technical Appendix[PDF - 313 KB - 8 pages]). Rates were adjusted for diagnostic test sensitivity and frequency of influenza testing (Technical Appendix[PDF - 313 KB - 8 pages] Table 3); however, because these results derive from estimates from 2 influenza seasons after the 2009 pandemic, our ability to determine whether the propensity to test has truly changed over time remains limited.
In conclusion, despite the increased use of highly sensitive molecular assays, current FluSurv-NET data still underestimate rates of influenza-associated hospitalization, particularly in adults >65 years of age. The primary reason for this underestimation is that diagnostic test sensitivity is imperfect, so true cases of influenza are missed. Furthermore, test sensitivity varies with patient age, and all types of influenza diagnostic tests, but especially RIDTs, have comparatively poor sensitivity in older persons. Adjusting hospitalization rates on the basis of diagnostic test sensitivity enables more accurate and timely comparisons of associated disease activity in hospitalized patients over time.
Dr. Millman is a physician and an Epidemic Intelligence Service Officer assigned to the Influenza Division, CDC. His research interests include influenza, emerging infections surveillance, and advanced molecular diagnostics.
We thank the following persons for their contributions at CDC and the FluSurv-NET sites: Alejandro Perez, Michelle Leon, Hallie Randel, Maria Rosales, Darcy Fazio, John Palumbo, Kimberly Yousey-Hindes, Olivia Almendares, Delmar Little, Kyle Openo, May Monroe; Dave Boxurd, Craig Morin, Sara Vetter, Minnesota Team Flu, Meghan Fuschino, Nancy Spina, Kirsten St. George, Gary Hollick, Maria Gaitan; Matthew Laidler, Patricia Newman, Katie Dyer, and Karen Leib.
FluSurv-NET is a collaboration of state health departments, academic institutions, and local partners and is funded by CDC. This publication was supported in part by cooperative agreement nos. CDC-RFA-CK12-1202 and 5U38HM000414.
- Dao CN, Kamimoto L, Nowell M, Reingold A, Gershman K, Meek J, Adult hospitalizations for laboratory-positive influenza during the 2005–2006 through 2007–2008 seasons in the United States. J Infect Dis. 2010;202:881–8.
- Dawood FS, Fiore A, Kamimoto L, Bramley A, Reingold A, Gershman K, Burden of seasonal influenza hospitalization in children, United States, 2003 to 2008. J Pediatr. 2010;157:808–14.
- Chaves SS, Aragon D, Bennett N, Cooper T, D'Mello T, Farley M, Patients hospitalized with laboratory-confirmed influenza during the 2010–2011 influenza season: exploring disease severity by virus type and subtype. J Infect Dis. 2013;208:1305–14.
- Takahashi H, Otsuka Y, Patterson BK. Diagnostic tests for influenza and other respiratory viruses: determining performance specifications based on clinical setting. J Infect Chemother. 2010;16:155–61.
- Jernigan DB, Lindstrom SL, Johnson JR, Miller JD, Hoelscher M, Humes R, Detecting 2009 pandemic influenza A (H1N1) virus infection: availability of diagnostic testing led to rapid pandemic response. Clin Infect Dis. 2011;52(Suppl 1):S36–43.
- Uyeki TM, Prasad R, Vukotich C, Stebbins S, Rinaldo CR, Ferng YH, Low sensitivity of rapid diagnostic test for influenza. Clin Infect Dis.2009;48:e89–92.
- Chartrand C, Leeflang MM, Minion J, Brewer T, Pai M. Accuracy of rapid influenza diagnostic tests: a meta-analysis. Ann Intern Med.2012;156:500–11.
- Talbot HK, Williams JV, Zhu Y, Poehling KA, Griffin MR, Edwards KM. Failure of routine diagnostic methods to detect influenza in hospitalized older adults. Infect Control Hosp Epidemiol. 2010;31:683–8.
- Peterson S, Dugas AF, Rothman RE. Evaluation of 11 commercially available rapid influenza diagnostic tests-United States, 2011–2012. Ann Emerg Med. 2013;61:573–7.
- Mahony JB. Nucleic acid amplification-based diagnosis of respiratory virus infections. Expert Rev Anti Infect Ther. 2010;8:1273–92.
- Kumar S, Henrickson KJ. Update on influenza diagnostics: lessons from the novel H1N1 influenza A pandemic. Clin Microbiol Rev. 2012;25:344–61.
- Cox CM, D'Mello T, Perez A, Reingold A, Gershman K, Yousey-Hindes K, Increase in rates of hospitalization due to laboratory-confirmed influenza among children and adults during the 2009–10 influenza pandemic. J Infect Dis. 2012;206:1350–8.
- Zhou H, Thompson WW, Viboud CG, Ringholz CM, Cheng PY, Steiner C, Hospitalizations associated with influenza and respiratory syncytial virus in the United States, 1993–2008. Clin Infect Dis. 2012;54:1427–36.
- Pinner RW, Rebmann CA, Schuchat A, Hughes JM. Disease surveillance and the academic, clinical, and public health communities. Emerg Infect Dis.2003;9:781–7.
- Dawood FS, Fiore A, Kamimoto L, Nowell M, Reingold A, Gershman K, Influenza-associated pneumonia in children hospitalized with laboratory-confirmed influenza, 2003–2008. Pediatr Infect Dis J. 2010;29:585–90.
- Barker WH, Menegus MA, Hall CB, Betts RF, Freundlich CB, Long CE, Communitywide laboratory-based influenza surveillance focused on older persons, 1989–1992. Am J Prev Med. 1995;11:149–55 .
- Steininger C, Redlberger M, Graninger W, Kundi M, Popow-Kraupp T. Near-patient assays for diagnosis of influenza virus infection in adult patients.Clin Microbiol Infect. 2009;15:267–73.
- Long CE, Hall CB, Cunningham CK, Weiner LB, Alger KP, Gouveia M, Influenza surveillance in community-dwelling elderly compared with children.Arch Fam Med. 1997;6:459–65.
- Steininger C, Kundi M, Aberle SW, Aberle JH, Popow-Kraupp T. Effectiveness of reverse transcription–PCR, virus isolation, and enzyme-linked immunosorbent assay for diagnosis of influenza A virus infection in different age groups. J Clin Microbiol. 2002;40:2051–6.
- Bradley Efron RJT. An introduction to the bootstrap. London: Chapman and Hall; 1993.
- Novak-Weekley SM, Marlowe EM, Poulter M, Dwyer D, Speers D, Rawlinson W, Evaluation of the Cepheid Xpert Flu Assay for rapid identification and differentiation of influenza A, influenza A 2009 H1N1, and influenza B viruses. J Clin Microbiol. 2012;50:1704–10.
- Van Wesenbeeck L, Meeuws H, Van Immerseel A, Ispas G, Schmidt K, Houspie L, Comparison of the FilmArray RP, Verigene RV+, and Prodesse ProFLU+/FAST+ multiplex platforms for detection of influenza viruses in clinical samples from the 2011–2012 influenza season in Belgium. J Clin Microbiol. 2013;51:2977–85.
- Templeton KE, Scheltinga SA, Beersma MF, Kroes AC, Claas EC. Rapid and sensitive method using multiplex real-time PCR for diagnosis of infections by influenza A and influenza B viruses, respiratory syncytial virus, and parainfluenza viruses 1, 2, 3, and 4. J Clin Microbiol. 2004;42:1564–9.
- Gharabaghi F, Tellier R, Cheung R, Collins C, Broukhanski G, Drews SJ, Comparison of a commercial qualitative real-time RT-PCR kit with direct immunofluorescence assay (DFA) and cell culture for detection of influenza A and B in children. J Clin Virol. 2008;42:190–3.
- Walsh EE, Cox C, Falsey AR. Clinical features of influenza A virus infection in older hospitalized persons. J Am Geriatr Soc. 2002;50:1498–503.
- LaSala PR, Bufton KK, Ismail N, Smith MB. Prospective comparison of R-mix™ shell vial system with direct antigen tests and conventional cell culture for respiratory virus detection. J Clin Virol. 2007;38:210–6.
- Irmen KE, Kelleher JJ. Use of monoclonal antibodies for rapid diagnosis of respiratory viruses in a community hospital. Clin Diagn Lab Immunol.2000;7:396–403 .
- Rahman M, Kieke BA, Vandermause MF, Mitchell PD, Greenlee RT, Belongia EA. Performance of Directigen flu A+B enzyme immunoassay and direct fluorescent assay for detection of influenza infection during the 2004–2005 season. Diagn Microbiol Infect Dis. 2007;58:413–8.
- Uyeki TM. Influenza diagnosis and treatment in children: a review of studies on clinically useful tests and antiviral treatment for influenza. Pediatr Infect Dis J. 2003;22:164–77.
- Stout C, Murphy MD, Lawrence S, Julian S. Evaluation of a monoclonal antibody pool for rapid diagnosis of respiratory viral infections. J Clin Microbiol. 1989;27:448–52 .
- Fader RC. Comparison of the Binax NOW Flu A enzyme immunochromatographic assay and R-Mix shell vial culture for the 2003–2004 influenza seasons. J Clin Microbiol. 2005;43:6133–5.
- Nelson EA, Ip M, Tam JS, Mounts AW, Chau SL, Law SK, Burden of influenza infection in hospitalised children below 6 months of age and above in Hong Kong from 2005 to 2011. Vaccine. 2014;32:6692–8.
- Ortiz JR, Neuzil KM, Shay DK, Rue TC, Neradilek MB, Zhou H, The burden of influenza-associated critical illness hospitalizations. Crit Care Med.2014;42:2325–32.
- Poehling KA, Edwards KM, Griffin MR, Szilagyi PG, Staat MA, Iwane MK, The burden of influenza in young children, 2004–2009. Pediatrics.2013;131:207–16.
- Williamson DA, Huang QS, Roberts SA, Grant CC, McArthur C, Baker MG. Surveillance for influenza using hospital discharge data may underestimate the burden of influenza-related hospitalization. Infect Control Hosp Epidemiol. 2012;33:1064–6.
- Widmer K, Zhu Y, Williams JV, Griffin MR, Edwards KM, Talbot HK. Rates of hospitalizations for respiratory syncytial virus, human metapneumovirus, and influenza virus in older adults. J Infect Dis. 2012;206:56–62.
- Burton C, Vaudry W, Moore D, Bettinger JA, Tran D, Halperin SA, Burden of seasonal influenza in children with neurodevelopmental conditions.Pediatr Infect Dis J. 2014;33:710–4.
- Emukule GO, Khagayi S, McMorrow ML, Ochola R, Otieno N, Widdowson MA, The burden of influenza and RSV among inpatients and outpatients in rural western Kenya, 2009–2012. PLoS ONE. 2014;9:e105543.
- Reed C, Chaves SS, Daily Kirley P, Emerson R, Aragon D, Hancock EB, Estimating influenza disease burden from population-based surveillance data in the United States. PLoS ONE. 2015;10:e0118369.
Suggested citation for this article: Millman AJ, Reed C, Daily Kirley P, Aragon D, Meek J, Farley MM, et al. Improving accuracy of influenza-associated hospitalization rate estimates. Emerg Infect Dis. 2015 Aug [date cited]. http://dx.doi.org/10.3201/eid2109.141665