U.S. flag

An official website of the Department of Health and Human Services

SHARE:

FacebookXEmail AHRQPrintShare

Evaluation Design and Methods

Evaluation Design

Each of the EvidenceNOW Cooperatives’ evaluation teams set out to determine the effectiveness of their external support interventions, using a range of mixed-methods designs.  The cooperatives were asked to capture a core set of measures of Aspirin use, Blood pressure control, Cholesterol management and Smoking cessation (ABCS measures) to allow for a robust evaluation of the effects of external support on cardiovascular health.  Additionally, the cooperatives were required to measure practices’ capacity using the Adaptive Reserve and the Change Process Capability Questionnaire (CPCQ) instruments.  Table 1. provides an overview of each Cooperatives’ evaluation designs and methods, as well as the national evaluation team’s approach.

Table 1. Overview of the Evaluation Designs of the EvidenceNOW Cooperatives and National Evaluation Team

Coop Design Framework Intervention
Duration		 Intervention
Components		 Core Outcome
Measures
MW Two arm cluster randomized comparative effectiveness design with four cohorts Practical, robust implementation and sustainability model (PRISM) 12 months Practice facilitation, HIT support, QI support Enhanced arms: Point of Care (POC) and POC + population health
NC Modified stepped wedge, stratified by readiness, randomized trial with six cohorts An organizational model of innovation implementation (Figure 1 in Weiner et al., 2015) 12 months PF, HIT support, QI support Same as above
NW Pragmatic 2x2 factorial design; matched external controls Practice Change Model (PCM) 15 months 1) PF only
2) PF plus learning
3) PF plus expert consultation
4) PF plus both		 Same as above
NY Stepped wedge with four cohorts for one practice network and three cohorts for another practice network Consolidated Framework for Implementation Research (CFIR) and Solberg framework for improving medical practice 12 months 1) Onsite facilitation which included but was not limited to QI and technical assistance on: performance feedback and benchmarking, alerts and CDS tools, dashboards, patient self-management, workflow assessment and redesign
2) Expert consultation
3) Peer-to-Peer learning collaboratives		 Same as above
OK Stepped wedge, cluster randomized trial (by county) with rolling recruitment and four waves PCM and CFIR 12 months PF, HIT support, QI support, academic detailing Same as above
SW Two arm cluster randomized trial (by region) with rolling recruitment; matched external controls RE-AIM 9 months Standard arm:  PF, HIT assistance, QI support, community connector, learning collaborative, eLearning modules Enhanced arm: standard arm intervention plus Boot Camp Translation-informed cardiovascular toolkit and materials, encouragement and support from the intervention team for the formation and active use of patient advisory councils Same as above
VA Stepped wedge with three cohorts Multi-faceted 12 months; 3-month intensive and 9-month maintenance PF, HIT support, QI support, workflow redesign, restoring joy in practice, online learning community Same as above
ESCALATES
https://escalates.org		 Prospective observational study CFIR and PCM see Cooperatives see Cooperatives Same as above

 

Coop Qualitative Data
Collection		 Cost Data Recruitment
Data		 Context
Data
MW
  • Interviews with practice representatives and PFs.
  • Focus groups with PFs.
 Cost data were collected as part of protocol Yes  
NC        
NW
  • Focus groups with PFs.
  • Online diaries provided by ESCALATES.
  • PF and practice stories.
   Yes 1) See PF documentation system
2) Control data: Practice characteristics and ABCS data from other small-medium practices in the US
NY
  • Interviews with PFs and PF managers.
  • Interviews with a sample of providers and staff.
  • Site visits/observations to understand PF intervention.
 Startup costs were calculated based on budget items Yes Practices participation in MIPS/MACRA, other state programs etc.
OK        
SW
  • Select key informant interviews.
  • Select practice visits and process observation.
  • Web-based PF and CHITA field notes.
      
VA
  • Interviews with Cooperative members.
  • Focus groups with clinicians and staff at a sub-sample of practices.
   Yes  
ESCALATES https://escalates.org
  • Online implementation diaries.
  • Interviews and observation with Cooperatives.
  • Interviews and observation with practices.
 Yes Yes Systematic assessment of context from the perspective of Cooperative team members
Page last reviewed March 2021
Page originally created March 2019

Internet Citation: Evaluation Design and Methods. Content last reviewed March 2021. Agency for Healthcare Research and Quality, Rockville, MD.
https://www.ahrq.gov/evidencenow/projects/heart-health/research-results/research/evaluation-design.html

Click to copy citation