Adverse Events

Objective: Accurate harms data are necessary to appropriately assess the balance between benefits and harms of interventions. Numerous deficiencies in the quality and reporting of harms associated with clinical interventions have been reported. Little is known, however, about whether such perceived methodological shortcomings are associated with lower estimates of harms.

Study Design and Setting: Studies reporting harms associated with:

1. Carotid endarterectomy (CEA) for symptomatic stenosis.
2. Rofecoxib for arthritis.
3. CEA for asymptomatic stenosis were identified using published systematic reviews.

A standardized abstraction form, including eight pre-defined criteria for assessing the quality of harms reporting, was used to extract data. Univariate and multivariate analyses were performed to empirically evaluate the association between quality criteria and estimates of harms. A quality-rating instrument for studies reporting harms was developed based on the results of the analyses of studies of CEA for symptomatic stenosis. The quality-rating instrument was tested on the other data sets.

Results: In 111 studies of CEA for symptomatic stenosis, meeting five of the eight quality criteria was associated with significantly higher rates of stroke or death. A quality-rating instrument with four of the five criteria predicted adverse events (5.7 percent in studies rated "adequate", compared to 3.9 percent in studies rated "inadequate" [p=0.0003]). In multivariate analyses, the quality rating assignment remained significant when controlling for other clinical and study-related variables. Different quality criteria, however, predicted lower estimates of risk for myocardial infarction in 16 trials of rofecoxib, and none of the quality criteria predicted lower estimates of stroke of death in 18 studies of CEA for asymptomatic stenosis. Evaluation of the latter two data sets was limited by small numbers of trials and low rates of evaluated adverse events.

Conclusions: The presence of methodological shortcomings can predict lower estimates of serious harms. Clinicians and researchers should carefully consider the potential effects of individual methodological shortcomings when evaluating estimates of harms associated with clinical interventions. However, we were unable to develop a generic summary quality-rating instrument for studies of harms because specific methodological shortcomings were not consistently predictive across data sets.

Download Report

Empirical Evaluation of the Association Between Methodological Shortcomings and Estimates of Adverse Events

• Technical Review (Publication No. 07-0003).

Evidence-based Practice Center: Oregon EPC
Topic Nominators: Agency for Healthcare Research and Quality (AHRQ)