How are States and evaluators measuring medical homeness in the CHIPRA Quality Demonstration Grant Program?

Authors: Stacey McMorrow, Anna Christensen, Brenda Natzke, Kelly Devers, and Rebecca Peters

Contents

• Key Messages
• Background
• Findings
• Conclusions
• Implications
• Learn More
• Endnotes

Many State Medicaid and CHIP programs and private health plans are pursuing medical home initiatives in an effort to improve the quality of care for children and adults, but varying conceptual definitions and measurement goals have led to the development of a number of different medical home measurement tools. This Evaluation Highlight examines the measurement of medical homeness in selected CHIPRA Quality Demonstration Grant Program projects. It also describes the development of the Medical Home Index-Revised Short Form (MHI-RSF), an adaptation of the short version of the Medical Home Index (MHI), for use in evaluating the demonstration projects. Using baseline MHI-RSF data, we present preliminary statistics on medical homeness for demonstration practices in six States.

Key Messages

Selecting a tool for assessing medical homeness is critical for implementing and evaluating efforts to enhance medical homes for children. The experiences of the demonstration States provide important lessons for other medical home initiatives.

1. CHIPRA demonstration States selected different assessment tools based on a variety of factors, including other medical home activities in the State, the target population for the medical home intervention, and properties of the tools themselves.
2. The MHI-RSF provides a low-burden option for collecting valid and reliable information on medical homeness for child-serving practices.
3. The MHI-RSF indicates that medical homeness varied across intervention practices in six States at baseline and suggests the opportunity for significant improvement as a result of the demonstrations.
4. The increased adoption of the MHI-RSF in child-serving practices across medical home demonstrations and evaluations could lead to improved generalizability of findings on the impact of medical homes on costs, quality, and outcomes.

Background

The patient centered medical home (PCMH) model originated in the pediatric community and has seen renewed popularity in recent efforts to improve quality of care for children.[1] The Agency for Healthcare Research and Quality (AHRQ) and several professional societies agree on the general characteristics of a medical home, which include accessible, coordinated, continuous, comprehensive, and patient-centered care.[2] However, each concept can be defined differently across diverse clinics, practices, and patient populations, and as a result, many medical home assessment or recognition tools have emerged over time.[3] Each tool includes unique questions to capture a given concept and places different emphasis on specific attributes. The use of different tools prevents “apples-to-apples” comparisons across demonstration projects, thus making any evaluation less useful for policymakers and researchers. Nonetheless, measurement of medical homeness—the extent to which a practice exhibits the attributes of a medical home—is critical to:

1. Evaluating whether demonstrations and pilots actually transform clinics and practices into medical homes.
2. Understanding whether such transformations affect key health care outcomes such as access, quality, or costs.

Twelve States are using CHIPRA Quality Demonstration Grant Program funding to design and implement PCMHs for child-serving practices, including pediatric and family practices and Federally Qualified Health Centers (FQHCs).[4] These projects represent a significant addition to existing medical home initiatives by Medicaid and private payers because prior initiatives have generally focused on adults.[5] Also, unlike other medical home initiatives that require practices to have already achieved a designated level of medical home recognition in order to participate, many CHIPRA quality demonstration projects are seeking to have participating practices formally recognized as medical homes by the end of the demonstration period. The intervention details vary by State, but broadly speaking, grantees are using quality improvement collaboratives and other strategies (for example, practice facilitation, workforce augmentation, and engagement of patients/families) to help child-serving practices develop or strengthen their medical home attributes and improve quality.[6]

The purpose of this Evaluation Highlight is to consider the issues and challenges involved in assessing medical homeness in the child-serving practices participating in the CHIPRA Quality Demonstration Grant Program. We describe the variation across demonstration States in their choice of medical home assessment tools and discuss the reasoning behind these choices. We also discuss the adaptation of the short version of the Medical Home Index to create the Medical Home Index-Revised Short Form (MHI-RSF) as a feasible, consistent measure of medical homeness across demonstration States and practices. Using baseline MHI-RSF data, we present preliminary descriptive statistics on the medical homeness of practices in six States. We conclude with the implications of our analysis for those interested in implementing and evaluating efforts to enhance medical homes for children.

Findings

Demonstration States adopted varied approaches to assessing medical home implementation

Each of the 12 States implementing medical home initiatives faced multiple options for assessing medical homeness within their demonstrations. While other tools were considered, the two leading contenders were the National Committee on Quality Assurance (NCQA) PCMH recognition tools and the Medical Home Index (MHI) developed by the Center for Medical Home Improvement (CMHI) (Table 1). Four States (Illinois, Maine, South Carolina, Vermont) chose to use a version of the NCQA tool as their primary measure of medical homeness, while five States (Florida, Idaho, Massachusetts, North Carolina, Utah) opted to use the MHI. Oregon, and its partner States, Alaska and West Virginia, chose components from both tools.

Table 1. Characteristics of Medical Home Assessment Tools

Sources: Burton R, Devers K, Berenson R. Patient-centered medical home recognition tools: A comparison of ten surveys’ content and operational details (2nd edition). Baltimore, MD: U.S. Centers for Medicare & Medicaid Services; 2012.
MHI: http://www.medicalhomeimprovement.org/knowledge/practices.html
NCQA: http://www.ncqa.org/tabid/631/Default.aspx.

States chose the different assessment tools based on a variety of interrelated factors, described below. The information presented in this section is based on conversations with State officials and other stakeholders during the first or planning year of the demonstration, as well as site visits to demonstration States in 2012, the second year of the demonstration and first year of implementation.[7]

States that chose the NCQA tool did so for some or all of the following reasons:

• NCQA is currently the leader in medical home recognition, having leveraged its prior work with commercial and Medicaid health plans to be one of the first to develop a medical home recognition tool. To date, the NCQA 2008 tool has been used most frequently in evaluations of medical home demonstrations.[8]
• Alignment with other medical home initiatives underway in their State using the NCQA tool was desirable. This consistency (1) sends a unified message to practices about what a medical home is and how it will be measured and (2) eliminates burden on practices that might arise if different tools are used in initiatives by Medicaid, CHIP, or commercial insurers, given that most practices serve children covered by multiple payers.
• Information obtained through a formal NCQA application and recognition process is verified by the NCQA and may provide more accurate information than self-reported data obtained via informal use of the NCQA tool or other instruments like the MHI.
• NCQA tried to make their 2011 tool more relevant and useful for child-serving practices, as some States felt the 2008 version was primarily adult focused.

Among States that chose the MHI, several specific factors played a role in the decision:

• The MHI is in the public domain and can be used at no cost and with minimal administrative burden, especially compared to formal NCQA recognition.
• The MHI was viewed as more pediatric-focused than the 2008 NCQA tool. While NCQA tried to address these concerns in its 2011 tool, the content of the new tool was not widely available when States were designing their demonstrations.
• The MHI was developed for practices serving children with special health care needs, and consequently, it was seen as being particularly attractive for States that were focusing their demonstration projects on this group. For example, compared with the NCQA tool, the MHI places more emphasis on care coordination, family-centeredness, and cultural competence and less emphasis on electronic health records and health information technology.
• Practices can complete the MHI as frequently as desired, thereby providing timely feedback for ongoing quality improvement purposes. The frequency for formal recognition application and renewal by NCQA is every 3 years.
• States felt comfortable asking comparison practices, which received few benefits from participating in the project, to complete the MHI because of its low burden.

Ultimately, each State planned to use the medical home assessment tool that met the needs of its intervention and was not deemed overly burdensome to participating practices. Not all States using a comparison group design, however, planned to collect medical home information from their comparison practices. Inconsistency in medical home measurement across the demonstration sites, and lack of measurement in some comparison practices, led to concerns that a rigorous cross-state impact analysis would not be feasible.

Modifications to the Medical Home Index produced a viable measure for cross-state comparisons

To enable a cross-state impact evaluation of the PCMH demonstrations, the national evaluation team asked selected States to collect supplemental data on medical homeness using a reliable and valid measurement tool that would be responsive to changes over the grant period. Project staff in many States were quite concerned about increasing the burden on participating practices by asking them to complete forms that would not be directly helpful to the practices. Such concerns underscored the need to use a brief measurement tool that practices would be willing to complete in the context of a demanding clinical environment. The national evaluation team chose to modify the MHI for this purpose because the MHI has been shown to be a reliable and valid tool to assess medical homeness at the practice level. Moreover, its six domains align with the American Academy of Pediatrics and the Maternal and Child Health Bureau definition of a medical home. This led to the creation of the Medical Home Index-Revised Short Form (MHI-RSF).[9]

The MHI-RSF consists of a subset of 14 questions from the 25-question MHI (Table 2), including the 10 questions from CMHI’s previously-developed MHI-Short Version (MHI-SV). In collaboration with CMHI, the national evaluation team selected four additional questions to supplement the existing MHI-SV. Two questions fall under the domain of Data Management, which was not included in the MHI-SV but has become more important in recent years as technology has advanced. The final two questions, Communication/Access and Family Involvement, were prioritized by CMHI as important additions. Taking two people a total of 20-30 minutes to complete, the MHI-RSF strikes a balance between being comprehensive and being low burden for practices. Ultimately, six States agreed to collect the MHI-RSF data to assist the national evaluation team, bringing the total number of States collecting MHI data to 10.

Our initial assessment indicates that the MHI-RSF is a reliable and valid measure of medical homeness. The MHI-RSF performed well on several tests of internal reliability and validity, and the MHI-RSF and full MHI tools rank practices similarly on medical homeness.[10] Future testing will consider the correlation between scores on the MHI-RSF and the NCQA tool, as well as the relationship between MHI-RSF scores and indicators of access and quality of care for children.

Table 2. Domains and Topics on the MHI and MHI-RSF

CSHCN= children with special health care needs; * indicates topic was not on the original MHI-SV.

Baseline MHI-RSF scores suggest opportunity for improvement in medical homeness across practices

As part of the demonstration projects, MHI or MHI-RSF data were collected from practices across several States. Six States (Alaska, Massachusetts, North Carolina, Oregon, South Carolina, West Virginia) provided baseline data to the national evaluation team for this analysis. The data presented here were collected between July 2011 and May 2012, and methods for selecting demonstration practices and collecting data varied across States. Each State sought to include practices that varied along key dimensions, such as size, ownership, and geographic location, but selected practices do not necessarily reflect the mix of practices in each State as a whole. As a result, the results presented in this Highlight should not be interpreted as representative of all practices in these States or child-serving practices more generally.[11]

Twenty-five intervention practices in two States (Massachusetts, North Carolina) completed the full MHI, while 38 intervention practices in four States (Alaska, Oregon, South Carolina, West Virginia) completed the MHI-RSF. In this analysis, we consider only the 14 questions in the MHI-RSF, even if the practice submitted the full MHI. Table 3 describes the standardized total scores for the MHI-RSF across 63 intervention practices. The standardized total score adds the scores across all questions on the MHI-RSF and assigns a value on a scale of 0-100, with 100 representing the highest degree of medical homeness.[12] Practice scores ranged from 32.1 to 91.1, and the mean of 56.1 reflects the average score across the practices. The median score of 53.6 indicates that half of all practices scored below this level. Overall, the data suggest that while some practices had a high level of medical homeness before the demonstration program, many practices have substantial room to improve.

Table 3 also includes the overall and domain mean scores. These scores average the scores across questions on the entire survey and within each domain.[13] On a scale of 1-8, the average overall score was 4.5 and the average domain scores ranged from a low of 3.9 for quality improvement to a high of 5.0 for data management. This indicates that there are larger opportunities for improvement in some domains, but that further progress is possible in all areas.

Table 3. Distribution of Baseline MHI-RSF Scores Across 63 Intervention Practices

Note: Overall standardized total scores are standardized to a scale of 1-100. The overall and domain mean scores are on a scale of 1-8. SD is standard deviation.

Conclusions

CHIPRA demonstration States selected different assessment tools based on a variety of factors, including other medical home activities in the State, the target population for the medical home intervention, and properties of the tools themselves. The choice of different tools creates numerous challenges for policymakers and researchers, but by developing the MHI-RSF, the national evaluation team was able to generate the consistent measure of medical homeness needed for its future cross-state quantitative evaluation. Preliminary analysis suggests that the MHI-RSF provides a valid and reliable measure of medical homeness, and baseline data indicate variation in the level of medical homeness across practices. This suggests that there is room for improvement as a result of the CHIPRA demonstration projects. Moreover, by using the MHI-RSF, the national evaluation will be able to provide policymakers and program directors with useful information about the relative success of different approaches to enhancing medical homes for children.

Implications

State program staff are often faced with difficult decisions about the best strategies for tracking their efforts to implement medical homes. Selecting an appropriate tool for assessing medical homeness is critical, but there is no consensus on the best tool for any given scenario. The experience of the demonstration States suggests the following:

1. NCQA is a leader in medical home recognition, and its tool is often used for consistency with other medical home initiatives and for the verified results provided by its documentation process.
2. The MHI was designed specifically to assess attributes of the medical home for children, particularly for children with special health care needs. Compared with the formal NCQA application process, the MHI’s low cost and ease of administration could make it attractive under certain circumstances.
3. Quantitative cross-state evaluations of medical home initiatives are strengthened by a consistent approach to measuring medical home attributes. Based on preliminary evidence, the MHI-RSF offers States, practices, and evaluation teams a feasible option for collecting information on medical homeness for child-serving practices.
4. Increased adoption of the MHI-RSF in child-serving practices across demonstrations and evaluations could lead to improved generalizability of findings on the impact of PCMHs on costs, quality, and outcomes. A similar adaptation of the adult version of the MHI could also prove worthwhile.[14]

As the demonstration projects continue, the national evaluation will track changes in MHI-RSF scores in response to each State’s efforts at medical home improvement, as well as the correlation between total and domain scores on the MHI-RSF and health care quality and outcomes for children. The national evaluation is also collecting MHI-RSF data in the comparison practices selected by several States. Ultimately, we will use the MHI-RSF to measure medical homeness in the intervention and comparison practices across States and over time as part of a cross-state analysis of the impacts of the demonstration projects on medical homeness and child health outcomes.

Endnotes

[1] Sia C, Tonniges T, Osterhus E, Taba S. History of the medical home concept. Pediatrics 1992; 90(3):419-423.

[2] Agency for Healthcare Research and Quality. Patient Centered Medical Home Resource Center. http://www.pcmh.ahrq.gov/portal/server.pt/community/pcmh__home/1483/pcmh_defining_the_pcmh_v2

[3] Burton R, Devers K, Berenson R. Patient-centered medical home recognition tools: A comparison of ten surveys’ content and operational details (2nd edition). Baltimore, MD: U.S. Centers for Medicare & Medicaid Services, 2012.

[4] The list of states implementing PCMH models with CHIPRA funding and information on each state’s project can be found at http://www.ahrq.gov/policymakers/chipra/demoeval/demostates/index.html.

[5] Takach M. About half of the states are implementing patient-centered medical homes for their Medicaid populations. Health Aff 2012; 31(11):2432-40.; Devers K, Berenson R, Coughlin T, Macri J. Innovative Medicaid initiatives to improve service delivery and quality of care. Washington, DC: Kaiser Commission on Medicaid and the Uninsured, 2011.

[6] Devers K, Foster L, Brach C. Nine states’ use of collaboratives to improve children’s health care quality in Medicaid and CHIP. Acad Pediatr (forthcoming, 2013).

[7] Further details on the NCQA and MHI tools, each state’s tool choice, and qualitative data collection methods can be found at http://www.ahrq.gov/policymakers/chipra/demoeval/highlights/supplhighlight02.html.

[8] Bitton A, Martin C, Landon BE. A nationwide survey of patient centered medical home demonstration projects. J Gen Intern Med 2010; 25(6):584-592.

[9] The MHI-RSF instrument can be found at: http://www.ahrq.gov/policymakers/chipra/demoeval/resources/mhi-rsf.html.

[10] Further details on the creation and psychometric testing of the MHI-RSF can be found at http://www.ahrq.gov/policymakers/chipra/demoeval/highlights/supplhighlight02.html.

[11] Further details on the construction of a cross-state database and state-level analysis of the MHI-RSF can be found at http://www.ahrq.gov/policymakers/chipra/demoeval/highlights/supplhighlight02.html.

[12] The MHI-RSF includes 14 questions, each with a maximum score of 8 points, for a maximum total score of 112 points. The score is standardized by dividing the actual score by 112 and multiplying by 100 to provide a score on a scale from 1-100.

[13] The overall score is the average of the scores on a scale of 1-8 for all 14 questions. The domain score for an individual practice is the average of the scores for each question in that domain where the number of questions varies by domain.

[14] The adult version of the MHI can be found at http://www.medicalhomeimprovement.org/pdf/CMHI-MHI-Adult-Primary-Care_Full-Version.pdf.