Inventory and Prioritization of Measures To Support the Growing Effort in Transparency Using All-Payer Claims Databases

Technical Expert Panel

Panel composition and role. A Technical Expert Panel (TEP) was formed based on suggestions from the project team and the AHRQ project officer. The TEP consisted of State APCD representatives, researchers, and consumer advocates, as well as a learning network (Appendix A lists TEP and Learning Network members). TEP members provided input on overall project direction and check-in on key project milestones during three focused TEP calls and occasional email-based discussions.

Goals for the first TEP call were to garner feedback on the approach to the APCD measure inventory. TEP feedback was desired on the framework for organizing and prioritizing measures.

The second TEP call used a thought experiment to help increase the usability and relevance of the measure inventory. The third TEP call focused on high-priority clinical areas to focus a more intensive evaluation of select potential measures ("the deep dive").

Over the three TEP calls and emailed feedback, the TEP provided key information on the APCD uses and users, organization and prioritization of measures, ideal application of APCDs to improve population health, and limitations of APCDs. They also ranked APCD use cases from various condition-specific areas.

APCD measure audiences and potential uses. TEP panelists noted that APCD measures have a variety of potential audiences, ranging from policymakers to consumers. Potential audiences included State health and data agencies, policymakers, payers, providers, third-party service providers, data or business analytic companies, consumers, and researchers.

Panelists maintained that policymakers were the primary target audience for initial reporting of measures using APCD data. These policymakers can affect further development of APCDs, as they need population-based measures that use readily available data, and are less affected by limitations such as data delays, which can severely limit the use of measures in consumer choice and negotiation.

Panelists noted that among the uses discussed, including choice, negotiation, accountability, and policymaking, one of the biggest potential targets for APCD measures is the market at large. Price transparency can be a powerful tool to help even out variations and constrain health care costs.

To further prioritize potential measures for inclusion in the measure inventory, panelists discussed the usefulness of the National Quality Forum (NQF) list of 20 high-priority Medicare conditions as one potential resource. Panelists noted that it may not adequately capture the interests of diverse audiences and recommended that consideration be given to three distinct payer audiences: Medicare, Medicaid, and payers. Consumers, for instance, would need measures that are "shoppable," meaning they cover conditions for which consumers have time to investigate the value of provider options.

The consumer use case was repeatedly discussed because consumer audiences have increasingly become a focus in reporting and use of APCDs. Panelists felt that while it would be useful to provide consumers with information on out-of-pocket costs, this use case presented certain challenges. A weakness of APCDs is the lag between claims and what is currently happening in the market. Given this delay, using the APCD for current pricing may not be effective or appropriate. Furthermore, informing the consumer was noted to be difficult and it would be hard to reach large numbers of consumers to change their behaviors in meaningful ways. Ultimately, an informed decision was made to not integrate consumer choice into the measure inventory.

Finally, usefulness and use cases are affected by data consistency and the ability to apply the measures across different APCDs and for across-State comparison. An illustration of this issue is that not all APCDs use the same clinical groupers (level of major diagnostic categories, diagnosis-related groups [DRGs]), and some do not use groupers at all. Thus, measures based on these groupings cannot be applied across APCDs.

Framework for organizing measures. Panelists expressed support for a framework presented by the project team, which organized measures by three functions of measures: choice, negotiation, and accountability. However, they noted that this might not be the most effective way to organize measures within the inventory. Panelists expressed the need for a framework that was flexible and could encompass the many possible current and future uses of APCD data.

Ideas for improving the framework included:

• Adding uses or functions such as feedback to providers and transparency.
• Considering which audience or stakeholder a use case applies to.
• Considering who the individual users are and their unique needs.
• Integrating use (such as the Choice, Negotiation, Accountability framework) and audience.

Ideal application of the measure inventory. To guide the presentation of the measure inventory to maximize its usefulness for the target audience of States, TEP members were asked to engage in the following thought experiment: "Imagine you are a state developing and/or maintaining an APCD. What would be the ideal application of that APCD to improve population health?" In response, TEP members provided the following feedback:

• TEP members noted the power of an APCD dataset lies in its ability to look at a broad cross-section of patients from multiple different payer types for any given provider. It was suggested that there is great interest in using such data to spot variation (e.g., in utilization, spending, rates of preventive screenings) and determining why that variation exists.
• Panel members expressed interest in looking at children across commercial versus Medicaid and the Children's Health Insurance Program to analyze access and utilization rates to study disparities.
• TEP members found it valuable to make data available to researchers and to have a dataset that allows people to investigate a broad range of topics.
• Topics of great interest included high-cost populations and the opioid addiction problem.
• Finally, panel members suggested that it might be more manageable to classify different stakeholder groups, develop an understanding of how each contributes to population health, and determine what information might allow each to make more meaningful contributions to population health.

High-priority clinical areas for measurement. To assist with selecting measures that would be assessed indepth, TEP members discussed and ranked the importance of use cases from a predetermined set of clinical priority topics: Diabetes, Mental Health and Substance Abuse, Childbirth and Reproductive Health, Imaging, and Medications. These topics met the criteria of being common clinical concerns and having multiple identified potential measures.

• Diabetes: Some members ranked this condition as high priority due to the large proportion of health care spending and it being a broad public health problem. Diabetes also offers an opportunity for improvement by better understanding differences in patterns of care across geographic regions.
• Mental Health and Substance Abuse: Members had widely differing opinions on Mental Health and Substance Abuse. One member ranked it as high priority because the topic was of interest in his/her State, while others ranked it as middle or lower priority. There was also uncertainty among the panel about the collection of mental health and substance abuse data, especially, privacy laws and concerns or noncoverage of care. Although some saw this condition as a high priority, others raised concerns that measures identified in the measure inventory were not sufficient to understand the cost/quality paradigm, specifically due to the limited number of quality measures. Also, utilization and cost measures in the absence of quality measures could be misleading and thereby discourage people from seeking services altogether in areas with limited choice or alternatives.
• Childbirth and Reproductive Health: Some members felt this area was high priority, because from the consumer perspective, there is time to plan; from the provider perspective, there is an opportunity to inform and educate. It was noted that the frequency of the ongoing prenatal care measure would be a challenge and APCDs may not be able to support this measure, but the area crosses payers, thereby providing consumers the ability to take action.
• Imaging: This use case ranked low among TEP members. They noted a limited breadth of measures, along with issues such as sizable price variation in imaging; and different technologies used for the same procedure, which leads to unproductive conversations and disagreement on costs.
• Medications: One of the advantages of an APCD is the ability to highlight differences in amounts paid for common medications. There are opportunities to focus the measures around medication to inform public policy that could result in better care and lower costs.

More TEP members prioritized diabetes than other categories, stating that the topic has the prospect of improving care dramatically for a significant portion of the population. In addition to diabetes, members suggested expanding the topic to focus on high-cost, manageable, chronic conditions in general (e.g., asthma, mental health, substance abuse). Mental health was the second most commonly prioritized topic area; TEP members hoped that the focus would promote development of better measures.

Return to Contents

Environmental Scan

Objective. The objective of the literature review and environmental scan was to map an approach to creating an inventory of measures of quality, cost, and utilization of care across settings for potential use with APCDs, noting gaps or current barriers to APCD measurement. The literature review and environmental scan provided a foundation for the work, describing the breadth of available measures and generating a framework for choosing measures and organizing them in the final inventory.

A large number of measures are based on administrative health data, in particular, hospital discharge abstract databases. For the purpose of the environmental scan, the focus was on measures that leveraged the unique aspects of APCD data. These included longitudinal data from multiple sources that allow patients to be tracked across time and settings, pharmacy data, and data on dollar amounts paid by insurers and patients. These key characteristics of APCDs enable measurement not possible with hospital discharge claims data alone, in particular, measures of ambulatory care (including measures that require data from multiple settings), episode-of-care measures, and cost measures.

Guiding questions. Several guiding questions were used to focus the search strategy and data collection efforts. The guiding questions resonated with the objectives of the report:

1. What measures or outcomes (quality, utilization, safety, price, etc.) that leverage the unique data in APCDs have been reported in the scholarly literature or in online public reports using APCD data?
2. What measures or outcomes have been proposed for use with APCD data or claims data that are episode based or longitudinal in nature?
3. What important measure gaps have been noted in relation to transparency initiatives? Have APCD-specific measure concepts been proposed to fill these gaps (even if no fully specified measures yet exist)?
4. What potential barriers to using and reporting measures with APCD data have been identified in the peer-reviewed or grey literature, including issues around availability and access to data elements? What strategies for overcoming these barriers have been proposed in the literature?
5. What are some of the methodological considerations and issues pertaining to using APCD data for measurement that have been discussed in the peer-reviewed or grey literature?

Summary of key findings from literature review and environmental scan. The peer-reviewed literature search yielded two basic types of articles: overview papers presenting concepts related to measurement in the areas of focus and those presenting particular measures used in addressing the authors' specific research question. The team screened a total of 189 articles; of those, 98 articles were included in a full review, of which 17 were overview articles and 81 were research articles.

Overview articles presented concepts related to measurement using APCDs or other large claims databases. The research articles focused on measurement of cost, quality, or utilization using APCDs, multipayer claims databases, or other claims databases (Medicare or Medicaid). The measures found most often focused on a specific research question or discussed the application of specific measures for public reporting or price transparency initiatives.

The environmental scan yielded information on existing APCDs and the potential of using APCDs to measure quality, cost, and utilization of care across populations and settings. Of the 236 different sources of information identified using the initial search criteria, 127 sources were included that met the inclusion criteria for this report. Sources found in the environmental scan came from AHRQ expert materials, the APCD Council, reports, task force papers, policy briefs, trade papers, business journals, white papers, books, APCD public reporting Web sites, measure inventories such as the National Quality Forum (NQF), and other sources.

National and State-specific general resources most often described the basis for the national trend to develop APCDs, as well as State-specific issues related to building or implementing APCDs. State resources, in particular, focused on building the case for and the barriers to establishing APCDs and using them for measurement. Many States with active APCDs have also issued reports of statewide quality, utilization, and cost.

Finally, resources related to price transparency were also included, as this is a major application of APCDs. The resources included high-level summaries of current efforts and barriers from governmental and nongovernmental sources, as well as State-level price transparency reports.

Major organizations and key contributors to APCDs and measurement. Several national organizations have developed expertise with APCDs and measurement using APCDs and have created resources and provided support for States creating APCDs. Such organizations include AHRQ, which has supported several initiatives, including:

• The United States Health Information Knowledge (USHIK) database, a repository for State APCD file submission specifications and data elements;
• The Community Quality Collaboratives program, where States shared experiences and best practices in quality and efficiency measurement, public reporting, and transparency; and
• The National Quality Measures Clearinghouse (NQMC), an online, searchable inventory of evidence-based measures and measure sets.

Nongovernmental organizations have also played a key role in advancing the science and implementation of APCDs:

• The APCD Council and the National Association of Health Data Organizations have taken the lead in supporting and documenting current State efforts and legislative work around APCDs. Efforts include working to harmonize data collection and release across States and providing technical and policy support to States that have or are developing APCDs.
• Catalyst for Payment Reform is a nonprofit organization that brings the perspective of purchasers to APCD efforts. They also provide tools such as report cards on States' efforts on price transparency to help purchasers and other stakeholders understand issues related to payment reform and transparency.
• The Health Care Cost Institute is a nonprofit organization whose goal is to provide access to health care cost and utilization data to researchers and policymakers trying to understand the factors influencing health care costs. They create twice-yearly cost reports based on claims from four major insurers.
• The Robert Wood Johnson Foundation has also been instrumental in bringing together multiple stakeholders to improve the quality of health care. For example, their Aligning Forces for Quality initiative has resulted in public report cards about quality in some States.
• In terms of measurement science, NQF is a leader in endorsing and encouraging implementation of evidence-based, valid, reliable measures that are meaningful to stakeholders, including consumers.

Other major sources of measures for this report included the literature review, public reporting Web sites from APCDs, and other online reports of cost and quality. Individual organizations, such as the National Committee for Quality Assurance (NCQA), the Quality Alliance Steering Committee (QASC), and Bridges to Excellence, also had useful measures.

Major sources of measures. The table below gives a high-level overview of the key sources of measures identified through the environmental scan and literature review. The focus was on measures that leverage the strengths of APCDs, namely ambulatory measures, episode-of-care measures, and cost measures. This review formed the basis of the measure inventory for use with APCDs. Because numerous measures were identified, the first step of the measure inventory was to prioritize the measures.

Table 1. Number of measures or number of public reports for potential use with APCDs

NQF: National Quality Forum; APCD: All-payer claims database; NCQA: National Committee on Quality Assurance.

* These papers provide measures or potential measures by describing one measure that is specific to the study question, using claims data, or describing the use of a group of measures that are already in use and are described elsewhere (e.g., NQF-endorsed measures, CMS measures).
† From a list compiled in AHRQ's Evidence-based Practice Center Technical Brief (archived). Public Reporting of Cost Measures in Health.

The literature provides studies using individual measures, some with well-described technical specifications. NQF and NQMC are measure aggregators and provide access to structured technical specifications. Public reporting Web sites either explicitly use State APCD data or use a combination of several data sources and measures. Some of these can be adapted for use with APCD data, although technical specifications are not always easily available through the online resource. Lastly, several organizations contribute discrete groups of claims-based measures. For example, NCQA, QASC, and Bridges to Excellence have made the technical specifications publicly available or available on request.

Key categories and domains for measures that leverage APCD data. The team identified key concepts and measure categories described in the literature review and environmental scan. These key categories and domains were used to organize the measure inventory, as well as in prioritizing certain categories of measures.

For instance, the purpose of performance reporting (e.g., for choice, negotiation, or accountability) can help guide measurement choice in the following way: if the purpose of a public report is to support consumer choice of providers, then a cost measure that only shows the average insurance reimbursement rate without including the patient out-of-pocket cost for each provider will not be helpful. However, if the purpose of the measure is to assist in negotiations between insurers and providers, average insurance reimbursement would be potentially more useful.

In addition, measures may be used for population health and policy purposes. For example, population-level measures of utilization and cost are important to strategic planning to help eliminate health care disparities at the State, regional, and local levels. APCD data may also be used in State operations such as budgeting and rate review.

Barriers to using APCDs for measurement and potential solutions to overcome them. Although APCDs are meant to contain comprehensive claims data across settings and time, there are still many barriers to using APCDs for measurement. Issues with data completeness, quality, standardization, and access hamper such efforts. Identifying and resolving these barriers is critical to using APCDs to improve health care value. The table below summarizes key barriers to APCD data collection and use and potential solutions for overcoming them.

Table 2. Key barriers and potential solutions related to data availability, quality, and access

Methodological issues or barriers pertaining to using APCD data for measurement. As more States develop APCDs and public reporting Web sites based on APCD data, it is critical to find valid measures that are relevant to stakeholders and feasible to implement on a large scale. Multiple methodological issues have arisen in pursuit of this goal. Key issues include:

• Inadequate measurement science that may threaten the validity and reliability of measures.
• Lack of standardization of measure concepts and specifications.
• Difficulties in implementing measures due to privacy concerns, denominator deficiency, difficult or inaccurate provider attribution, inadequate risk adjustment, or provider reluctance to participate in public reporting initiatives.
• Measure gaps, including methodological gaps and gaps in existing measures.

These methodological issues and measure gaps will need to be addressed for States and others to fully realize the potential of APCDs in increasing health care value. Some States are using their APCDs initially for public reports of State-level performance and substate (e.g., county or ZIP Code level) variations before the release of more granular analysis (e.g., by provider or payer). This allows early public reporting of policy-relevant data from APCDs for State decisionmakers, while the barriers described above are being addressed.

Conclusion. While it is clear that there is much to be learned to maximize the potential of APCDs and to reduce the difficulty and cost of using them, there is also national momentum building behind developing measures to be used with APCDs and defining the business cases for maintaining APCDs. This environmental scan provides an overview of both the potential for APCDs to generate the information needed to improve care, as well as caveats to keep in mind while doing so.

Next Page