National Evaluation Team

EvidenceNOW: Advancing Heart Health in Primary Care is an initiative of the Agency for Healthcare Research and Quality (AHRQ) to transform health care delivery by building a critical infrastructure to help smaller primary care practices improve the heart health of their patients by applying the latest medical research and tools. EvidenceNOW establishes seven regional cooperatives composed of public and private health partnerships that provide a variety of quality improvement services typically not available to small primary care practices. The goal of this initiative is to ensure that primary care practices have the evidence they need to help their patients adopt the ABCS of cardiovascular disease prevention: Aspirin in high-risk individuals, Blood pressure control, Cholesterol management, and Smoking cessation. The initiative also includes an independent national evaluation designed to determine if and how quality improvement support can accelerate the dissemination and implementation of new evidence in primary care.

Project Name: Evaluating System Change to Advance Learning and Take Evidence to Scale (ESCALATES)

Principal Investigator: Deborah Cohen, Ph.D., Oregon Health & Science University

Cooperative Partners:
Oregon Health & Science University
Rutgers—Robert Wood Johnson Medical School, The State University of New Jersey
University of Texas Health Science Center at Houston
HealthPartners Institute for Education and Research
Case Western Reserve University
University of Nebraska, Lincoln
Lehigh Valley Health Network
Veterans Affairs, Ann Arbor

Project Period: 2015–2010

Specific Aims

1. Engage EvidenceNOW grantees in the overall evaluation initiative by harmonizing measures and working together to collect similar qualitative and quantitative data, identify lessons learned, and foster rapid-cycle learning.
2. Identify the practice, organization, and contextual factors associated with higher and lower levels of delivery of ABCS services at baseline for over 1,500 primary care practices engaged by EvidenceNOW grantees.
3. Identify which intervention strategies are most effective in improving the delivery of ABCS services over time in relation to practice, organization, and contextual factors, and identify why some strategies are more effective.
4. Engage, rapidly disseminate, and evaluate the impact of disseminating actionable findings to key external stakeholders.

Updates on Key Project Components

Collecting and Analyzing Data

The national evaluation team is collecting and analyzing quantitative data from each EvidenceNOW grantee (or Cooperative) to identify the most effective combinations of intervention strategies for various primary care practice types, contexts, and organizational characteristics. The evaluation also is collecting qualitative data from Cooperatives and selected practices to understand why and how those combinations of interventions are effective.

Year 2 Accomplishments

• The national evaluation team conducted a second round of site visits to each Cooperative where they shadowed more than 41 practice facilitators in participating practices to gain deeper insight into the context and infrastructure in the regions, and also to build relationships with Cooperatives beyond the Cooperative leadership.
• The national evaluation has worked closely with each Cooperative to retrospectively harmonize intervention tracking data and operationalize various domains of intervention characteristics, including exposure, duration, and mode.
• The national evaluation team received additional ABCS and baseline survey data and began to characterize EvidenceNOW practices at baseline in terms of ABCS scores, practice location, practice size, practice ownership, practice capacity, burnout, and EHR characteristics, among other characteristics.
• The national evaluation team has used qualitative methods to identify contextual factors in each region that may be influencing various elements of intervention implementation through a context assessment tool. The national evaluation team has completed two rounds of context assessment and developed summaries of findings, one focused on the context of recruiting and engaging practices and one focused on intervention implementation.
• The national evaluation team also engaged with eleven external stakeholder groups to build relationships, introduce and discuss the EvidenceNOW initiative, understand what these groups hope to learn from EvidenceNOW, and develop collaborative dissemination approaches.

Updates

• The National Evaluation Team continues to engage with Cooperatives on the online diaries with a focus on learning more about practices in the active and maintenance intervention phases as interventions begin to end in Year 3.
• The National Evaluation team has begun to engage with Cooperatives through the EvidenceNOW evaluation workgroup on issues related to missing data and approaches to analyses; and will also engage the group in order to start the planning process of practice site visits, including developing materials and timelines.

Strategies for Disseminating Study Findings and Lessons Learned

The national evaluation team is disseminating EvidenceNOW activities and findings through a range of communications strategies, including an interactive website, multimedia webinars, blogs, and academic and professional publications and conferences.

Update

• The national evaluation team launched its external-facing website in Year 1 of the project, and since then has posted 31 stories authored by leaders in the field as well as EvidenceNOW partners, which detail findings and perspectives about EvidenceNOW through infographics, videos, blog stories, and Quick Reads.
• The national evaluation team has worked in collaboration with the Cooperatives and the TAC to present 19 abstracts, including three plenary addresses at conferences. Meetings where we’ve shared research findings include:
  • Academy Health
  • APHA (American Public Health Association)
  • NAPCRG
  • NAPCRG PBRN
  • D&I (Dissemination & Implementation)
  • SGIM (Society of General Internal Medicine)
  • SIRC (Society for Implementation Research Collaboration)
• The national evaluation team works with collaborators from across the EvidenceNOW Cooperatives, and has eleven manuscripts that are currently in development, and two published papers.
• The national evaluation team also maintains a Twitter account with over 140 followers and has conducted one online public webinar to rapidly disseminate findings from the evaluation.

Meeting EvidenceNOW Challenges

Comment From Principal Investigator

Deborah Cohen, Ph.D.

"The third year of EvidenceNOW is an exciting time for the initiative. The Cooperatives are completing their interventions, and we, the national evaluation team, are preparing to visit many practices throughout the country. This is an enormous undertaking, but we are eager to listen and learn about this initiative from the practices' perspectives. On a different topic, everyone is working to rapidly disseminate the findings from the initiative. What we're finding is that for rapid dissemination in our online platforms, the key is to keep it brief. It is more difficult to create something short, meaningful and actionable, so you need to know your audience, have your message clear, and edit the piece over and over. Short meaningful stories take more work, but we’ve found the result is gratifying."

What is happening in the 1500 small- and medium-sized practices: Cooperatives have been submitting ABCS and survey data to the national evaluation team throughout the study, and in Year 3, it is time for the national evaluation to get their boots on the ground and get into the practices to gather qualitative data. Through in-person practice site visits, the national evaluation team will answer four key questions:

• What kind of support did practices receive?
• What were practices' experiences with the support they received?
• What changes did practices make?
• Why were practices able to make changes in their quality and capacity, or why not?

In this next year, the national evaluation will sample high and low performing practices based on their ABCS scores over time. Visiting so many practices located across the US presents both a challenge for the national evaluation team and an unprecedented opportunity to understand, in a rich way, how and why Cooperatives' interventions work in a diverse set of practices.

Collecting data and disseminating rapidly: One of the four aims of the national evaluation is to rapidly disseminate key findings throughout the four years of the evaluation period. To accomplish this aim in Year 3, the national evaluation and Cooperative teams are continuing to collect data while at the same time shifting focus to collaboratively work on both traditional and non-traditional dissemination efforts in parallel. The challenge lays in what to share, when to share, and how to most effectively share the important findings emerging from EvidenceNOW, which is building one of the most interesting and comprehensive datasets on primary care practices nationally.