Development of the CAHPS Outpatient Mental Health Survey
In 2016, the CAHPS research team began developing a new survey of patient experience with mental health care provided in outpatient settings. The goal of this survey was to replace the Experience of Care and Health Outcomes (ECHO) Survey and respond to changes in the way mental health care services are perceived and provided. These changes include reduced stigma associated with the term “mental health,” and more integration of mental health services with other healthcare services as a result of technological advances.
Key Steps in the Survey Development Process
Initial gathering of information. The CAHPS research team began the process in 2016 by conducting a series of focus groups with a revised outpatient mental health survey to assess whether it adequately assessed the way contemporary patients experienced care. The team used information drawn from these activities to develop a revised mental health care survey that focused on obtaining prescription medicines for mental health and getting counseling and treatment.
Input from key audiences. In 2018, the team convened a Technical Expert Panel to review preliminary findings and make recommendations for revisions. Based on that information, the team revised the survey and conducted additional rounds of cognitive testing with a revised questionnaire. The research team collaborated with Research Support Services. Inc (Chicago, IL) to conduct cognitive interviews in Spanish. Cognitive interviews showed that patients do not always understand their mental health benefits, while the focus groups provided insight into how mental healthcare has changed. Using findings from the focus groups and cognitive interviews, several iterations of the survey were developed. Additionally, a Technical Expert Panel reviewed ongoing survey developments and offered recommendations.
Field testing. Due to logistical challenges related to finding partners for field testing, the team conducted multiple field tests with the revised survey from 2018 to 2022 to test the psychometric properties of the new questions asking about mental health care received in outpatient settings. The CAHPS research team first conducted online field tests among participants who were members of an online panel convened by Marketing Systems Group, a survey vendor. All participants reported receiving outpatient care for mental health in the last six months. The team then collaborated with the Veteran’s Administration (VA) for an online pilot of the instrument’s questions. After making further adjustments to the survey, the team conducted two additional field tests in the northeastern United States—one in cooperation with four Medicaid health plans in 2021, and the other with four family practice providers in 2022. Findings affirmed the content and construct validity of the revised instrument.
Development of the final version. The final 37-item survey was finalized and approved by AHRQ for the CAHPS trademark in 2024.
