The Deficit Reduction Act (DRA) of 2005 (PL 109-171, Section 6086(b)) directs the Agency for Healthcare Research and Quality (AHRQ) to develop home and community-based services (HCBS) quality measures for the Medicaid (Title XIX) program. AHRQ is to develop measures in the domains of client functioning, client satisfaction, and program performance in order to assess the quality of Medicaid HCBS programs nationwide. In addition, AHRQ is tasked with publishing best practices information as a result of comparative analyses. To lay the groundwork for meeting these requirements, AHRQ contracted with Thomson Reuters to conduct an environmental scan of existing and potential measures. The measure scan was AHRQ's first step in fulfilling this directive, through documentation of the available science to support measure development and use.
Per the legislative directive, measures were sought from a variety of academic, professional, online, and State sources in the three DRA domains, under the guidance of the Technical Expert Panel (TEP) assembled for this project. In addition, a formal Call for Measures was published in the Federal Register in June 2007. We used a very broad definition of HCBS services and populations, including populations such as adults with severe and persistent mental illness who are not traditional recipients of Medicaid HCBS. This approach yielded more than 200 measure sources, including survey instruments designed to yield performance measures, measure sets, and measure databases. Summaries of each, including information about psychometric testing, development, topics, and prevalence of use, were compiled in a compendium for public distribution.
A draft version of the compendium of measures (compendium) was discussed at an in-person meeting of the TEP in June 2007. During this meeting, TEP members began to define the broad DRA domains and align them with specific measure constructs. This process was complicated by the overlapping nature of the domains written into the DRA, particularly the fact that client functioning and client satisfaction outcomes are really measures of overall program performance. Through further rounds of voting and stakeholder input, 21 constructs emerged as a list of important dimensions of HCBS quality that would be generally applicable across HCBS populations. These were, by DRA domain:
- Client Functioning:
- Change in daily activity function.
- Availability of support with everyday activities when needed.
- Presence of friendships.
- Maintenance of family relationships.
- Employment status.
- School attendance (children only).
- Community integration.
- Receipt of recommended preventive health care services.
- Serious reportable adverse health events.
- Avoidable hospitalizations.
- Client Experience:
- Respectful treatment by direct service providers.
- Opportunities to make choices about providers.
- Opportunities to make choices about services.
- Satisfaction with case management services.
- Client perception of quality of care.
- Satisfaction and choice regarding residential setting.
- Client report of abuse and neglect.
- Availability of support for resilience and recovery (mental health service recipients only).
- Program Performance:
- Access to case management services.
- Availability of care coordination.
- Receipt of all services in the care plan.
It should be noted that many other constructs were considered, and some omissions do not reflect stakeholder consensus. In particular, as the project progressed, it was recommended by some TEP members that the constructs of pain, depression, and weakness/fatigue be added to the functioning domain, although not all stakeholders rated these as important.
A crosswalk of the compendium contents against these 21 constructs was used to populate matrices of measures that aligned with each construct and that had been subjected to some degree of psychometric testing. To further evaluate candidate measures, a set of scoring criteria were developed to rate the testing, target population(s), data collection requirements, and prevalence of current use (one component of feasibility) associated with each measure. Measures that did not meet certain thresholds in each of these areas were removed to create a second set of matrices. In addition, proprietary measures not formally submitted to AHRQ and those identified after the cutoff date for formal submission were not eligible for evaluation.
The final set of matrices, one per domain listed in the DRA, provided the data for identifying any relevant gaps in the current science in HCBS quality measurement. Looking at the compendium in consort with the matrices, certain broad themes from the scan emerge:
- Several consumer survey tools have been developed to assess consumer experience with HCBS, particularly for individuals with intellectual and/or developmental disabilities, that include constructs from the above list.
- Many surveys have been subject to psychometric evaluation and are currently used by several State programs. Some of these tools also have well-developed training programs to support their use, which enhances their feasibility.
- Other HCBS consumer survey tools were developed and tested specifically for one-time evaluation purposes and are no longer in use. However, they align well with the constructs of interest.
- No single survey tool or measure set addresses all the constructs included in the list above.
- Several other extant measures potentially align with these constructs, although relatively few formal measures have been developed and tested specifically for HCBS programs and supports.
- Many State-specific tools have been designed and implemented to solicit consumer feedback and/or report against the federally required assurances for Medicaid 1915c waiver programs. However, few have scientifically documented reliability and validity.
- Several academic studies have yielded well-tested measures with potential application to Medicaid HCBS programs. Most of these measures, however, are not in current use by any existing State programs.
With regard to the three domains articulated in the DRA, a more detailed analysis yielded the observations summarized below.
Client Functioning
Client functioning has a different connotation for HCBS program participants relative to recipients of other post-acute care services provided in community settings. HCBS programs provide services and supports that allow participants to function in their communities, regardless of disability, rather than aim solely to improve underlying functional ability. To this end, the DRA domain of client functioning was defined for this project to include the following three components: functioning in daily activities; social role functioning; and health promotion/adverse health outcomes. Together, these subdomains make up 10 measure constructs from the list of quality measurement priorities identified by the TEP.
In reviewing only the measures that meet the threshold evaluation criteria, we saw the following themes emerge:
- Because the goal of HCBS programs is typically to support participants' daily activity functioning, measures based on client reports of unmet need for assistance can be used to assess if a State's Medicaid HCBS program is providing appropriate supports. These types of measures are available and may be more relevant than measures of change in underlying functional ability.
- We found multiple measures, primarily in consumer surveys, to assess the identified dimensions of social role functioning.
- Consistent, HCBS-appropriate definitions of a serious reportable event are lacking, particularly for non-intellectually disabled populations. Most of the measures endorsed by the National Quality Forum, for example, apply to institutional or residential settings only.
- With regard to avoidable hospitalizations, the Prevention Quality Indicators (PQIs) developed by AHRQ would require additional testing and/or modification to determine their appropriateness for the Medicaid HCBS population. There are several challenges in adopting the PQIs, in both collecting data and linking hospitalization to HCBS programs.
- While the U.S. Preventive Services Task Force's (USPSTF) annual Guide to Clinical Preventive Services is considered the evidence-based standard for preventive health care services, USPSTF recommendations are designed for the asymptomatic population and are not disease—or individual—specific. Thus, there is no consensus set of preventive health care services for the Medicaid HCBS population and their complex care needs.
Client Experience
In our review, the broader concept of "client experience" was substituted for the "client satisfaction" DRA domain, to better capture both objective and subjective assessment of program services and supports. Innumerable dimensions of client experience can be measured, and many State Medicaid programs try to capture at least some client feedback. In our discussions with stakeholders and experts, eight dimensions of client experience emerged as clearly important, which are listed above.
Of the three DRA domains specified, client experience was the one with the best coverage, in terms of tested and prevalent measures. This was especially true of the population with intellectual and/or developmental disabilities. Much work has already gone into the development of consumer survey tools designed to solicit feedback on these constructs. The challenge is in comparing these instruments. In our preliminary review, we noted:
- There is a lack of standardization in the definition of abuse/neglect/exploitation in the items assessing client experience.
- There is an opportunity to assess (via psychometric testing) the applicability/reliability of existing experience measures to HCBS populations other than the one(s) for which they were developed. The need for such cross-disability testing depends in part on the future direction of AHRQ's strategy for meeting the DRA requirements and the use of cross-disability versus population-specific measures.
Program Performance
Although the DRA language specified measure development in three separate domains, they are not necessarily mutually exclusive concepts. There is overlap between measures of client function and experience and assessment of overall program performance. In fact, this final domain in some ways subsumes the previous two.
Currently, State Medicaid HCBS waiver programs are required to provide assurances to the Federal Centers for Medicare & Medicaid Services (CMS) of their compliance with six key dimensions of program performance. Assessing compliance with the full array of Federal program performance requirements exceeds the scope of this exercise. However, feedback from AHRQ's TEP members and others identified three measure constructs that address portions of program performance (although by no means fully define it): access to case management; receipt of all services in the care plan; and care coordination services.
Of the three DRA domains we examined, we found the fewest potential candidate measures for program performance, as defined for the purposes of the scan. We found few prevalent, common measures for the receipt of case management services (including care coordination or support broker services) or access to case management services that could be drawn from administrative data. Similarly, we did not find simple, standardized measures for the receipt of all services specified in the plans of care. In short, the greatest opportunity for measure refinement and development resides in the area of service delivery.
The measure constructs and associated measures that emerged from this measure scan process are by no means definitive with regard to HCBS quality. Many other areas of potential quality assessment did not make the consensus list, such as qualified providers and exercising individual rights, which may be worthy of consideration. Further, although measures developed for institutional settings were excluded from the lists of evaluated candidate measures, they may offer potential for modification and inclusion. Others with potential include the home health version of the Consumer Assessment of Healthcare Providers and Systems (CAHPS®), the CMS Continuity Assessment Record and Evaluation (CARE), and the Assessment of Health Plans and Providers by People With Activity Limitations, which is in development. During the public comment period of the final TEP meeting, participants also raised inclusion of new instruments, including the Universal Stakeholder Participation and Experience Questionnaires (U-SPEQ®) consumer interview tool recently developed by the Committee for Accreditation of Rehabilitation Facilities (CARF).
As AHRQ moves into the second phase of this project, other issues related to performance measurement will become increasingly important. The mode of data collection, for example, has bearing on both measure development and project implementation. Some measure constructs are best derived from consumer-reported data, particularly those related to consumer experience. Others may be calculated from one or more less costly administrative data sources.
Ideally, assessments of HCBS quality may necessitate triangulation of data sources. TEP members provided feedback suggesting that some of the constructs we examined, including avoidable hospitalizations, serious reportable events, school attendance, and receipt of all services in the care plan, might be obtained from administrative data sources. In particular, claims data can reveal a great deal about variations in care and potential quality issues. The mode of service delivery, namely in a traditional agency model or via a self-directed program, can also influence data items and data collection strategies.
Finally, in order to conduct the required State-to-State comparisons of HCBS quality, risk adjustment is needed. There are administrative measures that do not directly assess HCBS program quality but provide an important context for those that do, including severity of impairment and other assessment data. They also include participant demographics, such as age, sex, residential setting and size, and diagnosis. One TEP recommendation and possible future task is the development of a comprehensive set of individual demographic and disability measures related to individual functioning. Environmental service variables are also needed, which would enhance the value of any data collected by AHRQ in identifying best practices in Medicaid HCBS programs.
