Vol 6: Concordance Between the Findings of Epidemiological Studies and Randomized Trials in Nutrition: An Empirical Evaluation and Analysis

Structured Abstract

Full Title: Nutritional Research Series: Advancing the Role of Evidence-based Reviews in Nutrition Research and Applications
To facilitate a better understanding of the challenges involved in conducting nutrition-related systematic reviews and in integrating these reviews with nutrition applications for which such reviews have not been previously used, the Office of Dietary Supplements (ODS) has sponsored the development of a series of technical reports via the EPC Program. The sixth volume of this series examines the discordance between the results of observational studies and of randomized controlled trials (RCTs) and provides empirical data on how often the summary results of epidemiological studies and of RCTs are concordant.
Background: In nutrition, there are several examples of discordance between the results of observational studies and of RCTs.
Objectives: To provide empirical data on how often the summary results of epidemiological studies and of RCTs are concordant, and to explore whether the probability of concordant findings is associated with quantifiable metrics of citation maps formed between studies belonging to the evidence base of the nutrient-outcome association at hand. Citation maps are an objective representation of the translational paths in each association, and may be a surrogate of the maturity of the relevant evidence base.
Methods: We searched MEDLINE® to identify meta-analyses of RCTs or of epidemiological studies on the association between nutrients and health outcomes. Summary findings from both research designs that were statistically significant and in the same direction were considered qualitatively concordant. We also calculated the statistical significance of the difference in the summary effects from epidemiological studies and from RCTs (a measure of quantitative concordance). For each nutrient-outcome association we defined an “evidence base” including all publications identifiable by MEDLINE searches on the nutrient and outcome of interest and constructed citation maps of all articles in the evidence base that were cited by the epidemiological studies or the RCTs in the meta-analyses, either directly, or through one or more intermediary papers. We then quantified the size of the graphs (number of vertices and citation relationships), and their connectivity (density of citation relationships, mean hub and authority scores, and mean number of citations made or received over the included papers). We tested for associations between these metrics and the probability that the summary results from epidemiological studies and from RCTs are concordant between them.
Findings: In 23 out of 34 associations the summary findings from meta-analyses of epidemiological studies and of RCTs were in the same direction. In 6 of 23 associations, meta-analyses of epidemiological studies and of RCTs had statistically significant findings. In the remaining 11 out of 34 associations, meta-analyses of epidemiological studies and of RCTs had summaries pointing in opposite directions. In 12 out of 34 associations the summary findings of epidemiological studies were statistically significantly different from those of RCTs, in 6 out of 12 point estimates were in the same direction, and in the other 6 in opposing directions. Despite the variation in the size and the connectivity of the citation graphs across the examined associations, we find no evidence of association between quantitative metrics of the citation graphs and the probability that the two research designs have concordant or discordant findings (using various definitions of concordance or discordance).
Conclusions: The examined quantitative characteristics of the citation maps in each association cannot predict the probability that the findings from the two designs agree or disagree. It is unclear whether there is a good way to describe the maturity of the evidence base on an association between nutrients and outcomes. At a minimum, purely bibliometric approaches are not a good way to prioritize which nutrient exposures merit further study, and for which health outcomes.

Download Review

Nutritional Research Series, Volume 6: Concordance Between the Findings of Epidemiological Studies and Randomized Trials in Nutrition: An Empirical Evaluation and Citation Analysis

Technical Review (Publication No. 13-EHC067-EF): ( PDF File - 451.89 KB)

Evidence-based Practice Center: Tufts EPC
Topic Sponsor: Office of Dietary Supplements, National Institutes of Health

Current as of May 2013

Internet Citation: Vol 6: Concordance Between the Findings of Epidemiological Studies and Randomized Trials in Nutrition: An Empirical Evaluation and Analysis: Structured Abstract. May 2013. Agency for Healthcare Research and Quality, Rockville, MD. http://www.ahrq.gov/research/findings/evidence-based-reports/nutritn6tp.html