Advancing the Science and Implementation of Patient Narrative Elicitation and Reporting

Introduction

On September 8, 2016, the U.S. Agency for Healthcare Research and Quality (AHRQ) convened a stakeholder meeting to review what has been learned to date through experimental research and early field tests conducted by members of the CAHPS team regarding the collection and use of patient narratives. Consistent with AHRQ's mission to support the translation of critical health services research findings into widespread practice, the focus of the meeting was on moving activity focused on this highly relevant topic from research to implementation. Specific objectives of the meeting were to identify key methodological, operational, and policy issues that need to be addressed in order to advance the evidence-based use of patient narrative information as a complement to standardized patient experience survey scores, both to help clinicians understand what they can do to improve their care and to engage and inform patients about the care they receive.

The 56 participants attending the meeting included representatives from a broad spectrum of stakeholder groups, including health systems, public and private health plans (including Medicare), patient advocacy groups, regional quality improvement coalitions, health services research (both in and outside of AHRQ), and foundations.

Meeting Structure and Presentations

The meeting consisted of a series of panel presentations designed to lay the groundwork for what we know, followed by a facilitated group discussion focusing on key questions and issues to be addressed in future research and implementation initiatives.

Panel presentations included the following topics:

• Environmental Scan: A review of the current sites and markets within which patient narratives are collected, reported, and used, including online rating sites, health systems, health insurers, and regional quality improvement coalitions.
• The CAHPS Narrative Elicitation Protocol: A review of the five-question narrative elicitation protocol developed by the CAHPS team for use with the CAHPS Clinician & Group Survey.
• Elicitation Protocol Field Test Results: A review of key findings from field tests of the elicitation protocol in real-world data collection projects in California and Massachusetts.
• Public Reporting of Narratives to Consumers: A review of what the CAHPS team has learned about methods for analyzing and reporting patient narratives for use by consumers, with perspectives and insights offered by sponsors of public reports as well as a consumer advocacy group.
• Feedback Reporting of Narratives to Clinicians: A review by two health systems and one online rating site of their experience with analyzing and reporting patient narratives for use by clinicians to improve patient experience.

A summary of the presentation content may be provided in a future publication targeted to interested readers not present at the meeting. The following summary focuses on key questions and issues identified in the discussion.

Key Questions and Issues

Narrative Elicitation

• Framing of the invitation: Would the breadth and depth of information elicited be different if the language used to invite comments from patients was framed as an opportunity to provide information to patients/consumers for choice vs. to clinicians for improvement? How might differences in framing affect response rates?
• Representativeness of narratives: How can we assure that the representativeness achieved in experimental testing is also achieved in real-world implementation? What methodological adjustments might be needed to assure representativeness across different socio-demographic groups?
• Value of knowing what people are looking for:  How does an initial question about what people are looking for in a health care provider add useful information for interpreting their responses to later questions?
• Positivity of narratives: Why are patient comments so consistently and overwhelmingly positive? Is it because patients fear retribution for negative comments, patients want to express loyalty to their practitioners, or patients' experiences are in fact mostly positive? Does knowing what patients expect from their healthcare providers allow us to better discern among positive comments?
• Elicitation burden vs. motivation: With regard to the number of questions in the protocol, how can we best balance our desire to elicit balanced, complete, meaningful, and representative accounts of patient experience with the perceived burden to patients? For example, what information would be lost if we were to reduce the number of questions from, say, 5 to 4 or 3, and what would be the corresponding gain in terms of shortening response time and increasing response rates? How should the question of response burden be viewed in light of some early evidence that suggests giving patients the opportunity to provide comments can actually enhance survey engagement and response rates through the "power of storytelling"?
• Mode of data collection: Large-scale collection and analysis of comments may be accomplished most efficiently using electronic administration methods. However, provider organizations are not all collecting, verifying, or maintaining patient email addresses, and both patients and providers have privacy concerns about using email. As a result, email alone may not be a feasible option at this point, although it can contribute to increasing response rates. How can we accelerate the use of web-based modes such as email? What are the advantages/disadvantages to using phone and texting modes? What other administration methods can be used to make collection of narrative information "better, faster, and cheaper?"
• Patients vs. caregivers as respondents: How can elicitation methods be adapted to capture the experience of caregivers in addition to or as proxies for patients?
• Effect of the survey/narrative sponsor: Experience suggests that framing the survey sponsor as the respondent's specific clinician and/or medical practice is most effective in generating a response. However, might such a framing generate a bias toward being overly positive?
• Priming the audience: Efforts to promote the importance and/or value of the survey and narrative information in advance of fielding, e.g., through promotional campaigns in medical offices, appear to have a positive effect on response rates in some health systems. How generalizable is this effect and what types of efforts are most efficient and effective?
• Integrated vs. independent administration: What are the advantages/disadvantages of fielding the elicitation protocol as an integral part of the survey vs. as a stand-alone set of questions?

Narrative Analysis and Reporting

• Responding to urgent medical concerns: Because of the emphasis on maintaining the confidentiality of respondents, CAHPS surveys are not designed to capture and respond to pressing patient medical issues. What methods can health systems and medical practices use to effectively identify and triage urgent medical issues that are reported in patient narratives? How can patient privacy and confidentiality be protected while still enabling clinicians to respond to patients’ concerns?
• Large-scale analysis of patient comments: While narratives will likely have great value in being read verbatim, the collection of large volumes of narrative responses will require some type of processing in order to more efficiently extract the core meaning of the information. Can qualitative analysis methods using human coders realistically meet this challenge? Can methods based on computerized, automated analyses such as natural language processing (NLP) be effectively applied to patient narratives? How can we best process and categorize comments to encourage system-level analysis without discouraging clinicians and practice group leaders from carefully reading (and assessing) the comments verbatim?
• Distinguishing between clinician vs. teams vs. system: Most surveys and open-ended questions are focused on specific providers, yet patient comments may relate to the entire experience.  How do we address the challenges this may pose to achieve accurate and fair attribution and reporting?
• Value of public reporting:
  • Impact on improvement: Some health systems in the U.S. that have posted patient comments along with survey scores have seen dramatic improvements in survey scores, as well as an increase in website traffic. Some systems also find that public reporting has contributed to positive changes in organizational culture. What would it take for these kinds of improvements to occur in other health systems?
  • Impact on patient-provider relationships: Public posting of comments on the Patient Opinion site in the U.K. has encouraged patient communication and trust as well as provider communication. The focus of eliciting comments on this site is to foster change and build relationships, as well as to confer benefits to the person telling the story. Is this a model that has applicability in the U.S.?
• Value of feedback reporting to clinicians:
  • Clinician acceptance of patient experience measures: Does providing comments along with survey scores contribute to physician acceptance of patient experience measures, for example, by helping to validate and explain survey scores?
  • Impact on improvement: In what ways do patient narratives complement and add value to survey scores that can support clinicians’ efforts to identify and address specific processes and behavior changes needed to improve patient experience? What narrative analysis and distribution methods are most effective in supporting the use of this information?

Topics for Future Research and Development

Participants at the meeting noted several topics for future research and development.  These topics were generally regarded as important focus areas that will help advance progress in the elicitation, analysis, and reporting of patient narratives.

• Developing elicitation protocols for other healthcare settings
• Testing alternative narrative invitations/framing language
• Improving efficiency of data collection modes
• Strategies for taking analysis of narratives to scale (e.g., NLP)
• Assessing alternative feedback methods for clinicians
• Exploring regulatory/legal issues related to public reporting of narratives