Development of the CAHPS Child Hospital Narrative Item Set

The CAHPS Child Hospital Narrative Item Set enables users of the CAHPS Child Hospital Survey to elicit a short, coherent story from the parents and guardians of hospitalized children about the important aspects of their experiences with inpatient care. The items were developed and tested by researchers affiliated with RAND, the University of Wisconsin-Madison, and the Yale School of Public Health.

Development of the initial protocol. The CAHPS team began the development process by researching the relevant literature to identify important domains of care and consulting with experts in the measurement of pediatric care experiences. Building on the development of the Clinician & Group Narrative Items, the team then designed a sequence of questions that prompted respondents to comment on their experiences with a child’s inpatient care.

Pilot testing of the draft items. The research team conducted an initial pilot test of the draft items with 30 participants. The draft items in English and Spanish were also vetted by a large pediatric hospital’s parent advisors. Based on the feedback, the team revised the items for the evaluation study.

Evaluation study. To evaluate the revised items, the CAHPS team conducted a study involving participants in the Ipsos-administered KnowledgePanel, a probability-based online panel of about 60,000 U.S. adults. The first step was to screen the panel members who were parents of children under 18 years (N = 8,037) for eligibility for the study. To be eligible, a panel member had to have a child with an overnight hospital stay in the past 12 months (excluding stays for healthy newborns and stays entirely in the emergency room). Of the 393 panel members found to be eligible, 163 participated in the study (a 41% participation rate).

Study participants responded to a survey about their child’s hospitalization experiences either online or by phone in English. This survey was an abbreviated version of the Child HCAHPS Survey followed by six open-ended items. The team then conducted intensive interviews with 47 of these participants about two weeks after they completed the survey. The team coded and analyzed the open-ended data collected from the pilot test and evaluation study as well as the intensive interviews. The analysis was focused on:

• The scope of experiences reported in response to the six narrative survey questions (i.e., the number of domains of care mentioned and the frequency of those mentions).
• The balance of positive and negative feedback.
• The actionability of information conveyed, when compared to the intensive interviews.

Key findings from these analyses include the following:

• Participants completed the narrative items in about 5-7 minutes (the median time to complete online was 5.4 minutes; the median time by phone was 6.7 minutes).
• The average narrative was 248 words (the equivalent of about 12-15 sentences).
• On average, narratives contained descriptions of one or two actionable events.
• Most of the evaluative content provided by parents was positive (63% positive when gathered by phone, 68% positive online).
• Approximately eight in ten narratives mentioned topics covered in the Child HCAHPS Survey. Approximately nine in ten mentioned topics not covered by the survey.
• Narratives gathered by phone were substantially longer than narratives gathered online (749 vs. 127 words), contained more codable content, and corresponded more closely to information provided by parents in an hour-long interview (75% correspondence vs. 59% correspondence).

A detailed description of the analyses and findings from this study are available in the following paper: Martino SC, Reynolds KA, Grob R, et al. Evaluation of a protocol for eliciting narrative accounts of pediatric inpatient experiences of care. Health Serv Res 2023 Apr, 58(2): 271-281. https://pubmed.ncbi.nlm.nih.gov/36645204/

Based on the findings from the evaluation study, the CAHPS team identified two issues for further exploration. The first was whether the item set works better with or without an initial item asking parents what they would most like to say about their child’s hospital stay. The second was whether the final item, which focuses on care coordination, could be clarified. Item sets with and without the initial item and with different versions of the care coordination question were evaluated in cognitive testing.

Cognitive testing. To ensure that respondents understand and interpret the items as intended, the team conducted cognitive testing of the items in English and Spanish, with an emphasis on recruiting respondents with limited education. This testing was completed in late summer of 2022. Based on the findings from this testing, the team made a decision to drop the initial item asking parents what they would most like to say about their hospital stay and chose the clearer of the care coordination items that were tested.

Approval and release of the final version of the Child Hospital Narrative Item Set. In March 2023, the CAHPS Consortium approved the use of the final item set for use with the CAHPS Child Hospital Survey. The items are available in English and Spanish.  

Research Articles Published by the CAHPS Team

• Martino SC, Reynolds KA, Grob R, et al. Evaluation of a protocol for eliciting narrative accounts of pediatric inpatient experiences of care. Health Serv Res 2023 Apr, 58(2): 271-281. https://pubmed.ncbi.nlm.nih.gov/36645204/
• Quigley DD, Predmore Z. Parents have more to say: comments from the Child HCAHPS single question versus a narrative item set. Hosp Pediatr 2023;13(4):345–356. 
• Grob R, Schlesinger M, Barre LR, et al. What words convey: The potential for patient narratives to inform quality improvement. Milbank Q 2019 Mar, 97(1): 176-227. https://pubmed.ncbi.nlm.nih.gov/30883954
• Grob R, Schlesinger M, Parker AM, et al. Breaking Narrative Ground: Innovative Methods for Rigorously Eliciting and Assessing Patient Narratives. Health Serv Res 2016 Jun;51:Suppl 2:1248-72.
• Martino SC, Shaller D, Schlesinger M, et al. CAHPS and Comments: How Closed-Ended Survey Questions and Narrative Accounts Interact in the Assessment of Patient Experience. J Patient Exp 2017 Mar;4(1):37-45.
• Schlesinger M, Grob R, Shaller D, et al. Taking Patients’ Narratives about Clinicians from Anecdote to Science. N Engl J Med 2015 August;373(7):675-679.