To facilitate future cost estimation efforts, in this section we summarize the studies conducted by AHRQ Estimating Costs grantees. Short summaries of each team’s projects are available on the AHRQ Web site.
Appendix A summarizes key features of each study.
Settings and Practice Change Elements Studied
The 15 studies conducted by AHRQ Estimating Costs grantees included varying numbers of primary care practices, ranging from two to more than 500. Collectively, the studies obtained cost information from more than 700 primary care practices, including family medicine and safety net clinics located in rural, urban, and suburban areas. The organizational structures of the practices varied and included independent practices, for-profit and nonprofit health care organizations, Federally Qualified Health Centers, integrated medical groups, and staff model HMOs.
Most PCMH transformation efforts studied by AHRQ Estimating Costs grantees took between 2 and 4 years to implement and occurred between 2005 and 2015. Practice change efforts included redesigning senior care; implementing cardiovascular and diabetes care processes; integration of behavioral health services into primary care; expanded roles for physician assistants, medical assistants, and nurses; integration of virtual medicine; chronic disease management; and smoking cessation and depression screening initiatives.
Some practices developed and implemented their own PCMH-type transformation model to meet internal goals and standards (e.g., the Care by Design™ model developed and implemented by the University of Utah). Others followed a pre-existing model, such as the Lean methodology (also called the Toyota Production System), which was implemented at Group Health.
The majority of practice initiatives took place as part of an effort to achieve National Committee for Quality Assurance (NCQA) PCMH certification. Formal NCQA recognition is based on six standards, comprised of 27 separate elements (six of which must be passed), and is scored on a scale of 0 to 100. Depending on the total score, NCQA recognizes PCMHs as Level 1, 2, or 3, with Level 1 being the lowest level of recognition and Level 3 the highest. NCQA standards for achieving PCMH recognition have evolved since they were first developed in 2008, with updates issued in 2008, 2011, and 2014.3 Practices that sought NCQA recognition had to meet the standards of care necessary to obtain certification; however, other practices implemented just one or two standards or elements.
Cost Elements Estimated
AHRQ Estimating Costs grantees estimated a wide range of cost elements. In addition to estimating cost outlays, some grantees also factored the following into their cost estimates:
- Any savings achieved (e.g., from efficiencies achieved through adopting PCMH-type delivery models).
- Added revenue through PCMH incentive payments.
- Forgone revenue from reducing the number and insurance value of billable services.
Most grantees classified costs as direct or indirect costs, but definitions of direct and indirect costs varied across studies. For example, indirect costs variously included overhead expenses, forgone revenue, and unanticipated expenses resulting indirectly from care transformation (such as expenses associated with staff turnover). Some grantees did not distinguish between direct and indirect costs, choosing instead to report total costs, while others divided total costs into fixed and variable costs. Exhibit 3 shows items commonly reported by grantees as direct or indirect cost elements.
Some costs were typically not factored in by grantees. Transitions to EHRs were often excluded from cost estimates, either because the movement to implement EHRs predated many of the primary care transformation initiatives or because EHRs were viewed as a parallel effort, with separate costs and funding options. Some organizations received technical assistance through their participation in a PCMH demonstration program or training collaborative. However, most studies did not include the potentially significant expenses of technical assistance or other resources received through similar programs in their cost estimations.
Data Sources and Data Collection Methods
Most AHRQ Estimating Cost grantees collected information on transformation processes retrospectively to cover a past period of transformation activities; in one case, however, the information was collected both prospectively and retrospectively.
Data on the setting, PCMH-related practice changes, and cost elements were obtained from a variety of sources, including:
- Qualitative and mixed data collected from clinic representatives (e.g., clinic leaders, clinical and administrative staff): Structured and semistructured interviews with clinic leaders and staff, reviews of documents provided by clinic leaders and staff, and reviews of calendar entries and other documentation of transformation activities
- Quantitative data provided by clinics and insurers: Quality indicators; claims/billing data; organizational accounting/general ledger data, including overall financial records, payroll, billing, and expenditures; and data entered into study spreadsheets and questionnaires (online or not) by clinic leaders and staff
- External data: National and local labor rates provided by the U.S. Bureau of Labor Statistics
AHRQ Estimating Costs grantees identified key cost elements based on the literature and interviews with primary care clinic leaders to develop a variety of tools, including user-friendly spreadsheets, questionnaires, and online forms, with detailed instructions on how to collect cost data. Some of these tools are quite detailed, while others focus only on key cost drivers (e.g., staff time for various activities). Examples of these tools are provided in Appendix B.
In one example, Miller and colleagues integrated sensitivity analyses into their Web-based tool, allowing clinic leaders to estimate their costs under different scenarios, such as changing staff mix and adding or removing transformation activities. This tool can be used to estimate costs of transformation efforts either prospectively or retrospectively. Additional information about this tool is provided in Appendix B.
Data Analysis Methods
Grantees used qualitative and mixed methods to describe study settings and methods to estimate costs. The ABC method was used to calculate costs from the clinic’s perspective, while gross-costing methods were used to estimate costs from the perspectives of the insurer, staff model HMO, and grant program that funded primary care transformation efforts.
Analysis Methods to Describe Study Settings and Transformations Implemented
Most AHRQ Estimating Costs grantees used a case study method to describe key setting characteristics and practice changes. To develop narratives describing the transformation efforts, grantees used rich qualitative data collected through interviews and observations, as well as quantitative data about clinic size, populations served, and other practice characteristics.
Some grantees used quantitative data to compare measures of care quality and utilization before and after the transformation effort took place. For example, Kralewski and colleagues examined health care quality and access metrics such as ambulatory care sensitive hospitalization rates, blood A1c levels for diabetes patients, and patient satisfaction surveys.15 Martsolf and colleagues used claims data to construct an index composed of several quality indicators for diabetes, asthma, and cardiovascular disease care (G Martsolf, oral interview, March 2015).The index was then used to select primary care practices that achieved high levels of quality improvement associated with the transformation effort.
Cost Analysis Methods
Exhibit 1 provides a summary of the methods used to estimate the costs of primary care transformation efforts, including the purposes each method serves, the data required to use that method, possible analysis methods, and key considerations. How AHRQ Estimating Costs grantees used these methods specifically is described below.
ABC Methods
Most AHRQ Estimating Costs grantees used an ABC approach to retrospectively estimate the specific costs of primary care transformation efforts for their clinic or group of clinics. Grantees drew on several sources to develop ABC methods for their studies,12-14, 16-21 and then developed specific tools for data collection, several of which are included in Appendix B. A full description of how the ABC method can be used to estimate the costs of primary care practice transformation is provided in the Practical Guide section of this report.
Global Costing Methods: Trend Analysis and Econometric Modeling
Shi and colleagues10 used a global costing method including both trend analysis and regression modeling to estimate the costs incurred by a grant program to fund primary care transformation efforts across 110 clinics within 24 health care organizations in a large metropolitan area. Grant program expenditure data were used to estimate costs for each participating health care organization and clinic, including personnel, fringe benefits, travel, equipment, supplies, and alterations and renovations. Grant expenditure data were collected at 6-month intervals over a 3-year period. The trend analysis tracked total cost, cost per FTE physician, and cost per visit over the study period. The findings showed the incremental cost for clinics that achieved NCQA PCMH recognition versus clinics that did not. The incremental cost methodology is also discussed in Zuckerman et al.23
Shi and colleagues then used a Mann-Whitney test to determine if clinics that achieved PCMH recognition were significantly different from clinics that did not. Propensity score matching was used to identify and retain for linear regression analysis pairs of clinics that were comparable at baseline and differed primarily in whether they attained PCMH recognition during the study period. Linear regression models were used to control for factors other than PCMH transformation that may have affected cost, such as total number of visits, number of FTE physicians, patient demographics (e.g., sex, race, and age), and percentage of uninsured patients. Based on the results of a Hausman test, a fixed effects model was selected to control for unobserved clinic characteristics not varying over time that may have biased results. A difference-in-difference method was used to estimate the difference in cost changes before and after PCMH transformation, comparing clinics that did and did not ultimately achieve PCMH recognition.
Two studies used global costing methods to examine the costs of primary care transformation from a payer’s perspective. In one study, Kralewski and colleagues used claims data from a major health insurance plan to track patient-level costs of care. Allowed amounts paid were risk-adjusted (i.e., adjusted for patient case-mix) using the Johns Hopkins Adjusted Clinical Groups® algorithm plus patient age and sex. Patient-level, per member, per month costs for two clinics that underwent a major transformation were compared with those of 28 primary care clinics that did not transition to new models. Risk-adjusted costs were compared before, during, and after the transformation effort.14
In another example, Fishman and colleagues used general ledger data from a staff model HMO, which integrates coverage and care systems. Two methods were used to calculate the production costs of health care provided to members. Actual costs were measured using an internal cost model to allocate utilization and cost data, including overhead expenses. Standardized costs were also measured using the Resource Based Relative Value Scale, which is used by the Centers for Medicare & Medicaid Services to reimburse providers for services covered by Medicare part B, as described in O’Keeffe-Rosetti et al.24 To estimate the change in costs attributable to PCMH transformation, two regression models were used: 1) a linear regression model to estimate the change in health care costs before and after transformation in the HMO clinics that underwent PCMH transformation, and 2) a difference-in-difference method to compare cost changes for patients in those clinics compared with patients receiving care in the HMO’s statewide contracting network. Subgroup analyses were performed for patient subsets, including all adults, older adults, children, and persons with chronic conditions.
Results
Six of the 15 AHRQ Estimating Costs grantees were still in the process of analyzing data as of this writing; therefore, not all results were available.
The practices studied incurred a range of direct and indirect costs related to primary care transformation efforts, some of which were partially offset by PCMH incentives received. Across studies, staff time and benefits for existing staff and new hires, as well as lost revenue, were the main drivers of the cost of primary care redesign efforts. Similarly, reporting requirements (including staff time spent meeting them) were an important cost driver as well. One grantee reported that adapting an EHR to support transformation was a moderate cost driver. Some preliminary, illustrative findings from the AHRQ Estimating Costs grants are provided below.
- Halladay and colleagues found that the cost of initial transformation and attaining PCMH recognition was approximately $11,000 per clinician FTE in four small to medium practices (≤10 clinicians) in North Carolina (J Halladay, oral interview, February 2015).
- Magill and colleagues found that the average cost of sustaining transformation over time entailed significant monthly costs per FTE clinician (including physicians, nurse practitioners, physician assistants, and residents). Costs were also estimated per encounter and per member per month.25
- Kralewski and colleagues found that the cost of developing and implementing an advanced PCMH-type model (beyond PCMH certification standards) was greater than $1 million per clinic in two midsized high-performing clinics (5–9 clinicians) in Minnesota, but insurance companies saved more than $4 million in one year (about $31 per member per month). These gains were not sustainable, however, because the clinics were unable to bill for services provided by nurses and for telephone visits. After 2 years, patient costs for transformed clinics were higher than those of control clinics.15
- Miller and colleagues found that because of variation in personnel and integration activities, the total cost of mental and behavioral health integration ranged greatly (from <$30,000 to >$500,000) across six Colorado practices ranging from solo rural practices to large urban multispecialty primary care practices. Costs were lowest in practices that did not have behavioral health providers and utilized primary care providers and other staff to implement behavioral health screenings. Costs were highest in a large practice with multiple embedded behavioral health providers.26
- Shi and colleagues found that a large grant-funded PCMH transformation in 110 clinics in Louisiana had significantly higher overall and per patient per month costs during the transformation period in practices that achieved PCMH recognition compared with those that did not. Estimates of total incremental costs were sensitive to model specifications and sample size, and costs per FTE physician were deemed unreliable owing to the difficulty of quantifying FTE physicians due to staff turnover.10
Conclusions
The conclusions drawn from synthesizing the approaches, methods, and lessons learned from the 15 AHRQ Estimating Costs grantees were used to develop the Practical Guide at the beginning of this report. A summary of those findings is provided here.
Detailed descriptions of study settings and transformation efforts are important. Study settings and transformation efforts can vary widely. Robust descriptions of exactly what is being studied can help contextualize study results and clarify which audiences may find them helpful.
Consider the benefits and drawbacks of each cost estimation approach. A gross-costing method uses aggregated data, such as general ledger or claims data, to retrospectively analyze the costs of practice changes implemented. The benefits of this approach are that it does not require burdensome data collection and analyses can be adjusted for patient case-mix, demographics, and other variables such that the results are more generalizable to other settings. When trend analyses are conducted, problems with attribution can occur. Changes in cost trends can be caused by factors other than primary care transformation efforts. When claims data are used, they may not reflect all of the costs related to transformation efforts incurred by clinics (which are often not reimbursable).
A micro-costing (or ABC) approach creates a cost estimate based on a detailed analysis of resources used and the unit costs of each resource. This approach is appropriate to assess the costs of PCMH transformation efforts for most clinics. The benefits of the ABC approach are that it can be used either prospectively or retrospectively, does not require access to aggregate data, can account for all clinic-level resources spent as well as opportunity costs, and is computationally simple (i.e., it does not require statistical analysis software or skills). However, this method is time consuming for both investigators and clinic staff. This method is also susceptible to recall bias (for retrospective studies).
Report standardized results when possible. To help readers interpret and use the results from cost estimate studies, it is helpful to report results in standardized dollars (e.g., 2012 dollars) and provide metrics that adjust for clinic size (e.g., cost per member or per patient per month, per clinician FTE, or per visit).
Research Gaps and Future Directions
The Estimating Costs research team’s experiences highlight some research gaps and suggest promising directions for future research. The suggestions below build on input provided by AHRQ Estimating Costs grantees.
Standardizing measurements. The use of a variety of methods and metrics to report the costs of primary care transformation efforts makes meta-analysis difficult. It may therefore be important for the field to develop a consensus on standardized measurement strategies to measure costs on a larger scale. A useful industry model is the Medical Group Management Association’s use of annual cost benchmarking surveys of primary care practices.27
Examining understudied costs. Few grantees examined the cost of maintaining PCMH-related practice changes, and those who did encountered great difficulty disentangling the cost of maintaining practice change from other costs of providing health services. In addition, none of the grantees examined the costs or savings experienced by patients as a result of primary care transformation efforts. Studying these costs could be a useful contribution to the field.
Further research on the nature of primary care transformation. The definition of the PCMH has evolved and will likely continue to do so. In addition, the ways in which clinics implement PCMH standards vary greatly and will also continue to evolve. Understanding the PCMH practice changes implemented is an essential task to interpreting cost estimates; therefore, ongoing research on the nature of transformation efforts is needed. A useful industry model may be the Advisory Board Company’s ongoing Primary Care/Medical Home Benchmarking Survey.28
Measuring the value of primary care transformation. Beyond estimating the costs of primary care transformation efforts, several AHRQ Estimating Costs grantees commented on the importance of conducting research on the value of primary care transformation efforts, both to identify worthwhile practice changes that translate into measurable quality improvements and to identify other practice changes whose expense may not be justified by their contribution to PCMH goals. The use of large-scale benchmarking studies on the costs and components of primary care transformation, such as those mentioned previously, may be particularly helpful in this regard.
