A critical step is understanding and obtaining the resources needed for successfully planning and carrying out your evaluation. The resources for conducting an intervention and evaluation are shown in Table 1 and Figure 1. We suggest you take stock of these items during the early planning phase for your evaluation. Senior management and others in your organization may need to help identify and commit needed resources.
The resources available for the intervention are linked to your evaluation because they affect (1) the extent to which practices can transform care and (2) the size of expected effects. How many practices can be transformed? How much time do staff have available to implement the changes? What payments, technical assistance to guide transformation, and tools (such as shared decision making aids or assistance in developing patient registries) will practices receive? Are additional resources available through new or existing partnerships? Is this intervention package substantial enough to expect changes in outcomes? Finally, how long is it likely to take practices to change their care delivery, and for these changes to improve outcomes?
Inventory the financial, research, and data resources you can devote to the evaluation, and adjust your evaluation accordingly.
Similarly, the resources available for your evaluation of the intervention help shape the potential rigor and depth of the evaluation. You will need data, research skills and expertise, and financial resources to conduct an evaluation. Depending on the skills and expertise available internally, an organization may identify internal staff to conduct the evaluation, or hire external evaluators to conduct the evaluation or collaborate and provide guidance on design and analysis. External evaluators often lend expertise and objectivity to the evaluation. Regardless of whether the evaluation is conducted by internal or external experts or a combination, ongoing support for the evaluation from internal staff—for example, to obtain claims data and to participate in interviews and surveys—is critical. The amount of time available for the evaluation will affect the outcomes you can measure, due to the time needed for data collection, as well as the time needed for outcomes to change.
Table 1. Inventory of Resources Needed for Testing a Primary Care Intervention
|
Resource Type |
Examples |
|---|---|
|
Resources for Intervention |
|
|
Leadership buy-in |
Motivation and support for trying the intervention. |
|
Financial resources |
Funding available for the intervention (including the number of practices that can test it). |
|
Technical assistance |
Support available to help practices transform such as data feedback, practice facilitation/coaching, expert consultation, learning collaboratives, and information technology (IT) expertise. |
|
Tools |
Tools for practices such as registries, health IT, and shared decision making tools. |
|
Time |
Allocated time of staff to implement the intervention; elapsed time for practices to transform and for outcomes to change. |
|
Resources for Evaluation |
|
|
Leadership buy-in |
Motivation and support for evaluating the intervention. |
|
Financial resources |
Funding available for the evaluation, including funds to hire external evaluation staff if needed. |
|
Research skills, expertise, and commitment |
Skills and expertise in designing evaluations, using data, conducting implementation and impact analyses, and drawing conclusions from findings. Motivation and buy-in of evaluation staff and other relevant stakeholders, such as clinicians and staff who will provide data. Expertise in designing the evaluation approach and analysis plan, creating files containing patient and claims data, and conducting analyses. |
|
Data |
Depending on the research questions, could include claims, electronic medical records, paper charts, patient intake forms, care plans, patient surveys, clinician and practice staff surveys, registries, care management tracking data, qualitative data from site visit observations and interviews, and other information (including the cost of implementing the intervention). Data should be of adequate quality. |
|
Time |
Time to obtain and analyze data and for outcomes to change. |
