Data Collection Instruments and Analysis

An overarching, cross-cooperative evaluation of the EvidenceNOW initiative was conducted by the national evaluation team (ESCALATES). Local evaluations were also conducted by each cooperative. As part of these evaluations, the cooperatives and ESCALATES collected not only the ABCS outcome measures, but also robust primary data.  The primary data consistently collected across the cooperatives were surveys—a survey of each participating primary care practice completed by one individual in a practice and a survey of staff members in each practice.  In addition to surveys, evaluation teams also collected qualitative data via a range of methods including: interviews, focus groups, observation and site visits, as well as from practice facilitator notes or other intervention tracking documents, and online diaries.  Descriptions of the available data collection instruments are provided below.

Evaluation teams are still conducting their main analyses, however, if findings based on primary data are published, they are linked below. 

Practice Survey

The practice survey consisted of a core set of items to capture practice characteristics and infrastructure and 15 items from the Change Process Capability Questionnaire (CPCQ) for all cooperatives to collect from their practices at baseline, post intervention, and 6-months post intervention.  There were also optional items for cooperatives to field including items related to patient-centered medical home components. The national evaluation team created standardized codebooks for the survey: Summary, Baseline, Post Intervention, 6-Months Post Intervention, CPCQ, and Electronic Health Records.

Examples of the cooperatives’ specific practice surveys are provided (New York City; North Carolina; and Virginia.  Several cooperatives also fielded surveys with additional items unique to their local interventions and evaluations, for example:

• The Midwest cooperative also asked practices about point–of-care strategies (e.g., electronic reminders, standardized treatment protocols) and population management (e.g., outreach to patients, referral to community resources) at baseline and post intervention.
• The Northwest cooperative asked practices about the practice’s QI efforts, including: whether they were part of a larger organization’s QI, autonomous in their choice of QI efforts, and engaged in discussion with other practices.
• In their 6-months post intervention survey, the Southwest cooperative asked practices about their experience in the initiative and how the practice supports patients in terms of  social determinants of health (e.g., food security, support with housing, employment, education).

There were several challenges that cooperatives encountered with the practice surveys, to which they tried a range of solutions.  One of the key challenges was due to the amount time it took to complete the survey and collect the information to answer the questions (e.g., infrastructure questions).  Some of the challenges and solutions tried are described in Table X.  While most evaluators had conducted baseline analysis of the practice survey, some evaluators published these results and describe their analytic approaches and assumptions (Balasubramanian et al., 2018, Shelley et al., 2018).

Table X. EvidenceNOW Practice Survey Challenges and Solutions Tried

Practice Member Survey

In addition to the practice survey each cooperative fielded a practice member survey at baseline, post intervention, and 6-months post intervention.  The survey consisted of a core set of items including the Adaptive Reserve item set, as well as questions about the measures, patient needs, psychological safety, burnout, and readiness for change.  The national evaluation team created standardized codebooks: Summary, Baseline, Post Intervention, 6-Months Post Intervention, and Adaptive Reserve.

Similar to the practice survey, each cooperative was able to tailor and augment the survey to their aims.  For example, the Northwest cooperative developed separate surveys targeted at physicians (Baseline, Post Intervention, and 6-Months Post Intervention) and staff (Post Intervention and 6-Months Post Intervention). The following are the practice member surveys from the other cooperatives: (Midwest; New York City; North Carolina; Southwest; Virginia; and Oklahoma.

Cooperatives encountered challenges with fielding and analyzing the practice member surveys.  The practice member survey was relatively short, but posed challenges given the practices were small and medium-sized.  That is, staff would not complete the surveys or would not indicate their role in the practice, in part, given concerns about confidentiality or sufficient anonymity. Some other challenges to which cooperatives tried a range of solutions are described in Table X. 

While most of the evaluators have conducted only limited analysis to date, some of the analytic approaches and assumptions they made are described in early articles (Cuellar et al., 2018, Henderson et al., 2018, Shelley et al., 2018)

Table X. EvidenceNOW Practice Member Survey Challenges and Solutions Tried

Qualitative Data Collection

Evaluations of complex interventions like those implemented across the EvidenceNOW cooperatives warrant mixed-methods approaches.  Thus, the evaluators from across EvidenceNOW employed a wide range of qualitative methods including interviews, focus groups, site visits and observation, and online diaries to answer a host of evaluation questions.  Additionally, tracking of a cooperative’s intervention would often entail qualitative data collection (e.g., notes).  The respondents included practice leaders, practice staff, and practice facilitators.  The national evaluation also collected data from members of the cooperative teams.

Brief descriptions and select protocols for these instruments are provided below.

• Interview protocols were used across EvidenceNOW, for example:
  • In the New York City cooperative, interviews were conducted with QI leads, Providers, and Staff (1 and 2) to understand practice’s experience on key implementation domains (i.e., CFIR framework).
  • At the start of the initiative, the Southwest cooperative conducted interviews with practices to understand the internal and external factors that could affect a practice’s ability to participate and improve cardiovascular care for patients.
    • Internal factors included: practice structure, staffing, priorities, change processes, and methods of patient input. 
    • External factors included: concurrent/competing QI projects and health IT infrastructure. 
• Intervention tracking and notes
  • The Midwest cooperative used an intervention Tracking Score Sheet that was used by practice facilitators to code, on a 1-4 scale, practices’ progress on the following domains: point-of-care clinical decision support (CDS), other CDS activities, practice workflows, reports on ABCS performance, population management and list of patients not meeting ABCS measures, population management outreach, and population management community resources.
  • In the Southwest Cooperative, after each in-person visit, the practice facilitation staff completed a field note to collect both objective and subjective data about the encounter, including: who participated, type of meeting (in person or telephone), duration, and which coaching/facilitation activities/services were provided.  In addition to this objective data, practice facilitators were asked to classify the level of engagement from practice staff and their level of confidence in the practice’s ability to make progress on important practice improvements.
  • In addition to the field note form, the Southwest practice facilitation staff completed implementation tracking notes at the baseline visit and the three, six, and nine month follow-up visits.  Each note captured progress on elements of practice transformation related to leadership, data-driven improvement, empanelment, team-based care, patient-team partnership, and population management.
• Additional contextual information
  • The national evaluation team asked each cooperative to answer a set of written questions about the environment in which they implemented their projects.  Topics included: state and local policies, market, key stakeholders/partners, and historical background.
  • They also asked cooperatives to report on recruitment strategy, practices’ motivations and expectations, cooperative organization, and technological and/or geographic factors that influenced the external support provided.
• Online diaries:  The national evaluation team asked the cooperatives to have at least five individuals from their teams participate in online diaries to “document their experiences during project development and evolution, including practice engagement, recruitment, and providing external support for implementation” (Cohen et al., 2016).  For more information on the national evaluation team’s approach to the online diaries, go to: Cohen et al., 2016.

Several articles from across EvidenceNOW have reported on findings related to the qualitative data.  For example, ESCALATES synthesized multiple qualitative data sources  to report findings on recruitment (Sweeney et al., 2017) and the abilities and challenges of using EHRs for QI (Cohen et al., 2018). The cooperatives also published on practice facilitators and leaders’ perspectives on PF support (McHugh et al., 2018) and the rate of major disruptive events in practices (Mold et al., 2018).

For an up-to-date list of EvidenceNOW articles, go to the Publications page.

Protocol to Collect Cost Information

The national evaluation team collected information from cooperatives regarding their costs associated with the project, including information on personnel (salary and benefits), external consultants, subcontractors, and direct and indirect costs. 

Evaluators Workgroup

Beginning in early 2017, the EvidenceNOW Technical Assistance Center (TAC) convened a monthly workgroup of the evaluators from cooperatives and the national evaluation team.  This was one element of the technical assistance provided to grantees regarding data collection and analysis. Over the course of the year, the workgroup discussed a range of topics and challenges specific to EvidenceNOW evaluations including: missing data, data quality issues, scoring the CPCQ and Adaptive Reserve instruments, and integrating the intervention tracking data.  The workgroup provided evaluators the opportunity to discuss challenges and share solutions with one another.