Introduction and Purpose
This Practical Guide was developed based on the experiences and lessons learned from the 15 AHRQ Estimating Costs grants. Final reports for these grants were reviewed when available. In addition, telephone interviews were conducted with each principal investigator in February and March 2015. Interviews focused on clarifying what costs of primary care transformation were estimated; what methods were used to estimate costs; what, if any, tools were developed based on the study; and key lessons learned. Interviews were recorded and transcribed. The information collected from these sources was used to identify successful approaches for measuring the costs of a primary care transformation effort and key lessons learned for the field.
In addition, AHRQ hosted a conference call in July 2015 with the grantees to discuss their advice for other researchers, based on their experiences, to be included in a Practical Guide that would be useful for 1) researchers examining the costs of primary care transformation, and 2) administrators in health care organizations who want to predict or report the costs of primary care transformation efforts.
The intention of this guide is not to provide detailed methodological instructions, but rather to list the key steps in an analysis of the costs of a primary care transformation effort, review the range of methodological options, and describe key considerations for each method. References and appendixes provide additional detail for readers who wish to learn more about each method.
Step 1. Develop a Detailed Description of Study Setting and Transformation Efforts
An important first step in estimating the costs of a primary care transformation effort is to describe the transformation effort and the setting in which it is taking place. The costs of implementing primary care transformation can vary widely by organization type and clinic characteristics. Contextual information about the setting of interest, including provider and other staff mix, patient demographics and health status, number of providers, number of administrative staff, number of patient visits per year, payer mix, indicators of health care quality, and PCMH recognition or certification status, are important to consider when making cost calculations.
Primary care transformation can take many forms; therefore, it is important to describe the nature of the transformation whose cost is being estimated. Research questions at this stage may include: What standards or aspects of care were addressed by the transformation effort? What specific changes were implemented to address each standard or aspect of care? How did quality of care improve after practice changes were made (e.g., as measured by patient satisfaction ratings or the proportion of hypertension patients with blood pressure of ≤140/70 mm Hg)? It is also important to consider, and report, whether PCMH certification was sought and what level and stages of transformation are included in the study, including planning, model development, and training; implementation of PCMH-related practice changes; and maintenance. Primary care transformation efforts can take a number of years to implement and are an ongoing process. Therefore, determining what stages to include and the timeframe for these should be done at the outset of the study.
In addition, detailed descriptions of study settings, practice change efforts, and what is included in the cost estimations can help others infer the applicability of estimated costs to other settings.
Step 2. Select and Apply a Cost Estimation Method
Cost estimation methods can be divided into two main categories: micro-costing methods, also known as activity-based costing (ABC), which are based on a detailed analysis of resource use and unit costs of each resource; and gross-costing methods, which are based on aggregate data.9 Most AHRQ Estimating Costs grantees used an ABC method. Exhibit 1 summarizes for each method the purposes it serves, data required, possible analysis methods, and key considerations. Additional details about each method are provided below.
Gross-Costing Methods
Gross-costing methods can be used to conduct retrospective cost analyses of a primary care transformation effort when a good source of aggregate data is available. Data sources can include insurance claims data, general ledger data (e.g., from a staff model health maintenance organization [HMO] with many primary care clinics), or general ledger data from a grant program funding a primary care transformation effort.
It is important to note that claims data reflect the costs and savings experienced by insurers. This information may not reflect the full costs of the primary care transformation effort incurred by clinics, because many of the costs related to practice redesign are not fully covered.
Aggregate sources of data can be used to produce a descriptive analysis of clinic cost evolution before, during, and after the transformation effort took place. Exhibit 2 provides an example of cost trend analyses completed using general ledger data from a grant program that funded the primary care transformation effort. The graphs compare grant expenditures by clinics that ultimately succeeded in transforming into PCMHs with those of clinics that did not. The graphs’ middle line shows the difference in the costs incurred by clinics that completed transformation to a PCMH with clinics that did not.10
Descriptive analyses alone cannot establish the cost of a PCMH transformation effort; factors not related to the PCMH transformation effort, such as changes in patient mix and co-occurring quality improvement or other initiatives, can differentially affect costs across clinics over time.
Linear regression models can be used to compare costs before and after a transformation effort, or the costs of clinics that transformed versus those that did not, controlling for other possible causes of trends examined. Control variables can include patient case-mix, demographics, clinic characteristics, and other factors. When patients are the unit of analysis, fixed effects and random effects models can be used to account for unobserved clinic characteristics that may affect results.
A strong study design uses a difference-in-difference method, which compares differences in costs between transformed and untransformed clinics before and after transformation. This design requires data not only about clinics that completed PCMH transformation, but also data about a comparable control group of clinics that did not undergo or complete transformation. Propensity score matching techniques can be used to identify a comparable sample. Control groups are also helpful in descriptive analyses.
Key considerations for gross-costing methods include:
- These methods require access to general ledger or claims data spanning periods before, during, and after the transformation effort took place.
- Analysis of claims data may not fully reflect the costs of PCMH transformation efforts incurred by clinics.
- Attribution of costs to PCMH transformation can be challenging. While trend analysis is informative, it does not differentiate between the cost of PCMH transformation and other unrelated initiatives and trends that can differentially affect costs across clinics over time. To address this challenge, note the following:
- Regression analyses can be used to adjust cost estimates for risk (patient case-mix), patient demographics, clinic characteristics, and other variables.
- Use of a comparable control group that did not undergo transformation can help to distinguish changes caused by transformation efforts versus other trends or co-occurring initiatives.
- The difference-in-difference method requires a comparable control group that did not undergo transformation.
- Propensity score matching can be used to identify intervention and control clinics that are comparable on multiple covariates.
ABC Method
The ABC method can be used to either prospectively or retrospectively assess the clinic-level costs of primary care transformation efforts. ABC estimates are based on a detailed analysis of resources used and the unit costs of each resource.
The ABC method is the most common approach to assess the clinic-level costs of PCMH transformation efforts. It is most appropriate for estimating the costs of a single practice or small group of practices. While it is theoretically possible for the ABC method to be used over a large number of practices or an entire health system, this method is very labor intensive, and therefore is generally not practical for this purpose.
The basic ABC method follows these four steps:
- Step 1: Identify key cost elements and the degree to which they were or will be utilized. This can be achieved by interviewing a wide variety of key informants (i.e., practice leaders, clinicians, information technology staff, transformation leaders, and administrative staff) to collect information about the activities, staff, investments, and purchases associated with the transformation or by asking practice leaders to complete questionnaires. Questionnaires can be structured according to predefined categories of transformation expenses drawn from the published literature,11-14 or based on investigators’ prior experience with primary care transformation.
- Step 2: Assess unit costs for each cost element. In most cases, clinic leaders can provide this information by completing a spreadsheet or questionnaire and reporting on actual costs, such as staff salaries and benefits, equipment and space purchases, or leases. This can also be determined by using external data sources (e.g., national or local labor rates from the U.S. Bureau of Labor Statistics) to calculate standardized staff costs based on labor categories.
- Step 3: Multiply unit costs by the quantities of each resource utilized (e.g., number of full-time equivalent (FTE) staff per job category) to produce total costs for each item.
- Step 4: Add total costs for each item to produce a total cost.
To make cost estimates derived through the ABC method more relevant to other settings and increase external validity, sensitivity analyses can be used to estimate the range of costs given a variety of circumstances. For example, cost estimates can be stratified by clinic size, rural/urban location, geographic location, organizational attributes (e.g., group vs. independent practice), transformation activity or component implemented, and level of PCMH recognition achieved.
Data Collection
For researchers using a micro-costing or ABC method, data collection may present the greatest challenge for generating accurate cost estimates of transforming care. Barriers to data collection using the ABC method include the following:
- Data collection can often be time consuming and burdensome for both researchers and respondents.
- Respondents may have limited expertise in cost data collection and a lack of familiarity with practice costs.
- Retrospective data collection is subject to recall and nonresponse biases due to staff turnover. Staff may not recollect all activities implemented as part of the transformation effort, and staff who have left the practice cannot report on the time they spent on transformation-related activities.
- Estimating the costs of maintaining practice changes may be difficult because of challenges of attribution. Once the transformation has been implemented and the changes have become part of the regular workflow, clinicians and staff may not be able to easily distinguish how much time is being spent on general practice activities versus transformation-related activities.
- These challenges can be mitigated by offering flexible options for data collection, developing easy-to-use data collection instruments designed to facilitate recall, providing technical assistance to participants, and allowing for ample time to estimate costs.
Selection of Cost Elements
An additional challenge of data collection is identifying cost elements to be collected. The costs of a transformation effort can be classified into direct and indirect costs. While what is considered a direct and indirect cost can vary by site, direct costs of primary care redesign efforts are those that are clearly attributable to these efforts, such as staff training on the PCMH or new staff hired to implement PCMH-specific aspects of care (e.g., care coordinators). Indirect costs are those that can be attributed to PCMH-related practice changes but can also be incurred as a result of other activities. This includes costs such as overhead expenses and expenses associated with staff turnover. Total costs can also be classified into fixed and variable costs. Fixed costs remain the same regardless of the level or intensity of redesign, such as the cost of a facility upgrade, while variable costs can change (e.g., staff time).
Transitions to electronic health records (EHRs) may or may not be relevant for inclusion when estimating the costs of primary care redesign, depending on whether efforts to implement or adapt EHRs occur as part of the primary care transformation initiative or are considered a parallel effort, with separate costs and funding. Researchers should consider whether the costs of implementing an EHR system are relevant for inclusion, and may want to keep these costs separate from other costs.
Similarly, some organizations may not include the costs of the technical assistance, such as practice facilitation, that they receive to implement or maintain a PCMH. Often, this is because the assistance came through participation in a demonstration program or training collaborative. When possible, it is helpful to include the costs of technical assistance in cost estimations. Otherwise, researchers should explain when it is not possible to identify and report these costs; in these cases, researchers should fully describe the types and level of technical assistance received. It is important for groups that are contemplating transformation, but are not participating in such a program, to plan for the costs of technical assistance that may be needed to assist with the implementation of PCMH-related practice changes.
AHRQ Estimating Costs grantees have identified key cost elements, based on the literature and interviews with primary care clinic leaders, that may be useful for other investigators. These cost elements have been integrated into a variety of tools, including user-friendly spreadsheets, questionnaires, and online forms, with detailed instructions on how to collect cost data. Some of these tools are quite detailed, while others focus only on key cost drivers (e.g., staff time for various activities). Examples of these tools are provided in Appendix B.
- Key considerations for the ABC method include:
- This method can be used to fairly precisely estimate the costs of practice changes at the clinic level.
- This method is computationally simple.
- Data collection using this method can be very time consuming.
- AHRQ Estimating Costs grantees found that it was most effective to include on the research team one or more junior staff members who focused on data collection; a clinician who practices in the clinics studied and has a good rapport with the staff who will be interviewed; and a multidisciplinary research team, including finance and systems operations specialists, to facilitate cost attribution and analysis.
- This method can be used to estimate costs either prospectively or retrospectively.
- In retrospective analyses, recall bias and staff turnover are barriers to data collection.
- Researchers must find or develop a tool for accurately collecting cost data. Examples of cost data collection tools developed by AHRQ Estimating Costs grantees are included in Appendix B. These tools provide a detailed breakdown of cost elements and may be helpful to estimate the costs of primary care transformation in other settings.
Step 3. Report Results
While readers may be very interested to see a “price tag” for a clinic or health system transformation effort, total cost estimates presented in isolation may be misleading. To help readers interpret study results, important contextual information should be presented along with results, such as:
- Whose costs were estimated (i.e., costs to a clinic, grant program, or payer)?
- What practice change activities were implemented?
- Are technical assistance and EHR implementation costs included?
- What stage of transformation was the focus of the cost study (i.e., model development, implementation, certification, and/or maintenance)?
- Was PCMH certification sought, and at what level?
Further, to help readers apply the findings, results should be presented in a standardized way. A variety of metrics can be used to adjust estimates for clinic size, staff mix, and PCMH-related practice changes implemented. These include:
- Cost per member or per patient, per month
- Cost per clinician FTE
- Cost per administrative staff person
- Cost per patient per encounter/visit
- Cost per accreditation standard or element
Another way to standardize results is to report them in standardized dollars (e.g., 2012 dollars).
Other Key Considerations for Estimating Costs of Primary Care Transformation
Some additional insights produced by the AHRQ Estimating Costs studies may be useful for future researchers, regardless of the cost estimation method employed:
- On average, AHRQ Estimating Costs grantees found that the transformation efforts they studied occurred over a 2- to 4-year period. Thus, future researchers planning to study the costs related to primary care transformation efforts should plan to examine data spanning a 2- to 4-year period.
- Several limitations of cost estimation studies suggest that the full costs of implementing or maintaining a PCMH-type initiative may be underestimated or overestimated. Recall and nonresponse biases are a particular concern in micro-costing studies, and challenges of attribution are a concern in all study types.
- Costs may be affected by co-occurring quality improvement or other initiatives whose activities overlap with transformation efforts.
- The definition of a PCMH has evolved and will likely continue to do so. As recognition standards evolve along with the definition, so will the costs of obtaining and retaining PCMH certification.
