Webinar #1: Fuzzy Set Analysis (Slide Presentation)

Presenter: Marcus Thygeson, MD
Discussant: Jodi Holtrop, PhD
Moderator: Michael I. Harrison, PhD

Sponsored by AHRQ's Delivery System Initiative in partnership with the AHRQ PCMH program.

Slide 1

Advanced Methods in Delivery System Research –Planning, Executing, Analyzing, and Reporting Research on Delivery System Improvement Webinar #1: Fuzzy Set Analysis

Presenter: Marcus Thygeson, MD
Discussant: Jodi Holtrop, PhD,
Moderator: Michael I. Harrison, PhD

Sponsored by AHRQ’s Delivery System Initiative in partnership with the AHRQ PCMH program

April 15, 2013

Slide 2

Speaker Introductions

N. Marcus Thygeson, MD is Senior Vice President and Chief Health Officer, Blue Shield of California, where he leads the healthcare services team.

Dr. Thygeson’s presentation today will draw on his paper with Deborah Peikes and Aparajita Zutshi on Fuzzy-Set Qualitative Comparative Analysis and Configurational Comparative Methods. This AHRQ PCMH Research Methods Brief is posted on the AHRQ PCMH website. Details will be provided at the end of this webinar.

Jodi Holtrop, PhD has been an Associate Professor in the Dept. of Family Medicine at the Clinical Center of Michigan State University. In July she will become a faculty member of the Medical School at the University of Colorado at Denver.

Dr. Holtrop is currently the principle investigator on several funded grants including an AHRQ-funded demonstration grant on "A Comparison of Provider versus Health Plan Delivered Care Management in Michigan." Her presentation today is based on one part of the research conducted under that grant.

Slide 3

Fuzzy Set Qualitative Comparative Analysis (fsQCA): Application to PCMH Research

An AHRQ Advanced Methods Webinar
Presented by: Marcus Thygeson, MD MPH
April 15, 2013

Reference: Thygeson NM, Peikes D, Zutshi A. Fuzzy Set Qualitative Comparative Analysis and Configurational Comparative Methods: Powerful Methods to Study and Refine Patient-Centered Medical Home Models. Rockville, MD: Agency for Healthcare Research and Quality.

Slide 4

Acknowledgments

• Apu Zutshi (Mathematica Policy Research) for early input to this presentation.
• Apu and Deborah Peikes (also MPR), my co-authors on the AHRQ methods brief from which this presentation is derived.
• Leif Solberg (HealthPartners, MN), who has been a great mentor and gave me the opportunity to use fsQCA to study the relationship between quality outcomes and PCMH characteristics.
• Charles Ragin (UC, Irvine), who originated the fsQCA method, and has generously provided guidance and encouragement to countless investigators, myself included.

Slide 5

Intended Results for Today

Understand what fuzzy-set Qualitative Comparative Analysis is, and how it is different from more traditional biostatistical analysis.

Understand the kind of studies or questions for which fsQCA is well-suited.

Understand some practical considerations to keep in mind if you are going to try using it.

Slide 6

Presentation Outline (Part I)

• Context setting:
  • fsQCA as a set-theoretic method for analysis of social phenomena
  • Comparison between set-theoretic configurational methods and methods based on correlation and regression
• Performing a fsQCA study
  • Study design and set up
  • Performing the analysis
  • Sample results from a PCMH study
• Advantages
• Limitations
• (Other applications, useful references)

Slide 7

fsQCA—the method

• Set-theoretic method that matches the ubiquity of set relationships in social and natural phenomena:
  • "All PCMHs are primary care practices but not all primary care practices are PCMHs" → "PCMHs are a subset of primary care practices"
• Developed in the social sciences for comparative analysis of small-N ("5-50") case studies
  • Care delivery systems are social phenomena; each primary care practice is a potential "PCMH" case; most PCMH case series are small
• To study phenomena characterized by complex causality
  • Equifinality: multiple paths to the same outcome
  • Conjunctural causation: impact of one factor contingent on presence or absence of other factors
  • Causal and conceptual asymmetry: A → B does not imply not-A → not-B, and the absence of A does not necessarily imply not-A

Slide 8

A Middle Path

Contrast with Regression

Slide 9

fsQCA—based on (fuzzy) set theory

• The basic unit of analysis: sets and subsets.
• Researcher assigns each case (unit of observation) membership in sets representing different explanatory variables and outcomes.
  • For example, Practice A is in the set of "good team-care" (1) and Practice B is not (0) [This is a "crisp set" binary membership assignment.]
• "Fuzzy set" QCA allows cases to have partial (or ‘fuzzy’) membership, between 0 and 1. For example, the fuzzy set of practices with NCQA PCMH certification may look like:

Slide 10

Calibration

• Assignment of set membership requires "calibration"—an explicit decision about the substantive meaning of observed variation
  • For instance, temperature is calibrated against the physical properties of water: water freezes at 0 degrees Celsius and boils at 100 degrees Celsius.
  • Water at 37 degrees is "neither in nor out" of the set "warm water".
• The calibration process forces the investigator to think carefully about the concepts being investigated
  • A practice that should be routine in regression-based analysis, but is often overlooked.
• An example—calibrating "access".

Slide 11

Necessity and Sufficiency

• Necessary relationships occur when a characteristic, or combination of characteristics, is observed in all practices with the outcome. The outcome is only observed when the characteristic is present. That is, the set of cases with the outcome is a subset of the cases with the characteristic.
• Sufficient relationships occur when a characteristic, or combination of characteristics, is observed in a subset of cases with the outcome. The outcome is invariably observed if the characteristic is present.
• Correlations are symmetric "necessary and sufficient" relationships: presence of the characteristic implies the presence of the outcome, and vice versa.

Slide 12

Recognizing fuzzy set relationships in data

Necessity (the lower right triangle plot)

Actual output from fs/QCA graph capability demonstrating that higher socioeconomic status is a necessary but not sufficient condition for achieving better optimal diabetes care. In other words, ODC is a fuzzy subset of ~LOWSES (not low socioeconomic status). The consistency of the set relationship is 0.96; and the coverage is 0.62.

Slide 13

Sufficiency (upper left triangle):

Serum methylmalonic acid (MMA) and homocysteine (HC)levels in 491 episodes of B12 deficiency. (NEJM 368;2:158)

In patients with B12 deficiency, elevated HC is sufficient for elevated MMA. Cases with elevated HC are a subset of cases with elevated MMA.

MMA is a more sensitive test for B12 deficiency.

Slide 14

Set relationships in B12 deficiency

Slide 15

Consistency and Coverage

• Consistency:
  • Measures the degree to which one set is a perfect subset of another. Varies between 0 and 1.
  • Generally require consistency > 0.8 to establish a set relationship.
  • Analogous to correlation coefficient; measures the strength of the relationship.
• Coverage:
  • Measures the degree to which a subset condition "covers" or "explains" an outcome. Varies between 0 and 1.
  • Analogous to "relevance". A highly consistent condition with low coverage has low empirical importance.

Slide 16

fsQCA enables configurational analysis

• Exploring the relationship between combinations of characteristics and an outcome.

• In a study of 21 Level 3 PCMH clinics, low preventive service use (PSUTD) was found to be related to a variety of factors, including average patient BMI, poor communication (~C), caring for women or seniors (WS), and weak practice systems for promoting preventive services (~PS). The findings:

  BMI or (~C and WS) or (C and ~PS and ~WS) → low PSUTD

  • Prior statistical analysis had overlooked these relationships.
  • Importance of practice systems contingent on patient population.
  • Importance of patient population characteristics obscured when treated as control variables.
  • IMPLICATIONS: work on communication in WS practices, practice systems in non WS clinics.

Thygeson et al., 2012

Slide 17

High-level process overview:

Step 0: Study design, case selection and data collection.

Step 1: create a data set of cases with key characteristics and outcome of interest.

Step 2: Choose membership function approach and assign fuzzy set membership in each characteristic for all cases. Calibrate based on substantive or theoretical knowledge.

Step 3: Identify necessary and sufficient relationships between possible explanatory conditions and the outcome of interest

Step 4: Use findings from Step 3 to develop or refine logic models ; then use fuzzy set Truth Table Analysis to identify sufficient associations between combinations of conditions (configurations) and outcomes.

Slide 18

Truth Table Analysis Results

• Another example: factors associated with low performance on Optimal Diabetes Care (ODC) in 21 Level III PCMH clinics (Thygeson et al., 2012):
  • Patients with low socioeconomic status, OR
  • Women or senior patients, OR

  • Weak team care capabilities for diabetic patients.
     

    LOWSES + (W+S) + ~DTC → low ODC scores

    consistency = 0.83, coverage = 0.74

Slide 19

Advantages of fsQCA

• Well-suited to exploratory analyses and theory development.
• Useful for small-N studies (and also large-N).
• Provides systematic approach to conceptualizing and analyzing qualitative data.
• Identifies equifinality—multiple paths to the same outcome.
• Useful with non-linear, non-normal data.
• Can be combined with (complements) statistical methods.
• Identifies interactions between contextual and instrumental factors.
• Supports linkage between implementation and impact factors.

Slide 20

Limitations of fsQCA

• Sensitivity to case selection (especially with small N).
• Risk of bias due to subjective decision making (e.g., choice of membership function and calibration criteria).
• Limits to explanatory power
  • Case-to-condition ratio limits prevent over-fitting of results.
• Limits to causal analysis (like regression, association does not imply causality).
• Analysis requires considerable time and effort (iteration, and lack of fully featured software tools).
• Lack of familiarity means hard to find knowledgeable reviewers for grants and papers.

Slide 21

Transition to Jodi Holtrop

Appendix slides address more detailed description of the fsQCA process, a list of published health services research articles using the method; and some useful references.

Slide 22

fsQCA applications in health services

• Examine relationship between medical "home-ness" and quality of care (Thygeson, Solberg, Asche, et al., 2012).
• Identify high-performance work practices associated with high front-line health care worker job satisfaction and perceived quality of care (Chuang, Dill, Morgan, et al., 2012 ).
• Identify causal configurations of antecedents associated with heavy drinking (Eng and Woodside, 2012).
• Organizational designs for achieving high treatment trial enrollment (Weiner, Jacobs, Minasian, and Good, 2012).

Slide 23

Useful references

• The Comparative Method, Charles Ragin, 1987.
• Redesigning Social Inquiry: Fuzzy Sets and Beyond, Charles Ragin, 2008.
• Configurational Comparative Methods, Benoit Rihoux and Charles Ragin, 2009.
• Set-Theoretic Methods for the Social Sciences, A Guide to Qualitative Comparative Analysis, Carsten Schneider and Claudius Wagemann, 2012.
• Thygeson NM, Peikes D, Zutshi A. Fuzzy Set Qualitative Comparative Analysis and Configurational Comparative Methods: Powerful Methods to Study and Refine Patient-Centered Medical Home Models. Rockville, MD: Agency for Healthcare Research and Quality.
• http://www.u.arizona.edu/~cragin/fsQCA/

Slide 24

Performing the Analysis

• Step 0: Study design, case selection and data collection
  • fsQCA is both a research approach and an analytic method.
  • Often select cases that are exemplars (selection on the outcome), a "no-no" with traditional analysis
    • Supports "solution" oriented analyses and understanding why something didn’t work that was expected to.
  • Specify concepts and collect data so as to facilitate calibration and assignment of set membership.
  • Develop a logic model for expected relationships.
  • Expect to iterate based on preliminary findings—case selection, data collection, calibration of measures.

Slide 25

Performing the Analysis (cont’d)

Step 1: create a data set of cases with key characteristics and outcome of interest

Slide 26

Performing the Analysis (cont’d)

Step 2: Choose membership function approach and assign fuzzy set membership in each characteristic for all cases. Calibrate based on substantive or theoretical knowledge.

Slide 27

Performing the Analysis (cont’d)

• Step 3: Identify necessary and sufficient relationships between possible explanatory conditions and the outcome of interest
  • Necessary conditions are present in all cases with outcome
    Fuzzy membership in outcome < membership in condition
  • Sufficient conditions are present in a subset of cases with outcome
    Fuzzy membership in outcome > membership in condition
• Step 4: Use findings from Step 3 to develop or refine logic models ; then use fuzzy set Truth Table Analysis to identify sufficient associations between combinations of conditions (configurations) and outcomes

Slide 28

Confessions of a First Time fsQCA User: Using fsQCA to Understand Determinants of Care Management Program Quality

An AHRQ Advanced Methods Webinar
Presented by: Jodi Summers Holtrop, PhD, April 15, 2013
Co-Authors: Lee A. Green, MD, MPH; Georges Potworowski, PhD

Slide 29

Disclaimer

• I am NOT an expert on fsQCA; instead I share my experiences using this method; also I am but one of a three person core team.
• The study referenced is in process and the results presented are not final; at this point they are only shown for an example of the method.

Slide 30

Provider-Delivered Care Management Pilot (BCBSM) and Grant (AHRQ Study)

• Studied BCBSM pilot project to examine delegation of care management from health plan to physician organizations and their practices.

Aims:

1. Compare patient outcomes among those offered PDCM versus health plan delivered care management (HPDCM).
2. Describe the implementation of PDCM.
3. Important: Describe practice environments and contexts of chronic care delivery, and identify implementation, PO, and practice features associated with improved outcomes.

Slide 31

Focus for this Presentation

Provide lessons learned based on how fsQCA was used to explore these questions…

1. Is there one best way to structure care management or does it vary by context?
2. What are the necessary and sufficient ingredients for care management program quality?

Slide 32

Lesson #1: Set up data collection with fsQCA in mind

• Use a logic model as a conceptual guide to start
  • What are the features or conditions that will potentially affect the outcome of interest?
• Within the logic model, identify features and/or conditions, as well as to what extent they exist and to what extent they matter
  • Helps to calibrate in fsQCA.
• Specify outcomes in advance
  • Make sure you are able to get the data.
  • If qualitatively derived into a quantitative value, need to do that in advance and separate from your analysis.

Slide 33

Care Management and Context Features

1. Lit. review identified some key features of CM success; vetted by experts.
2. Created "scorecard" with Likert-type anchors.
3. Scored practices on each feature, then inserted notes and later quotations.

• Patient access to program.
• CMgr qualifications and personal attributes.
• Staff CM training.
• CM resources for staff and patients.
• CM program attributes (Health promotion, care transitions, disease management).
• CM Program duration, dosage, features and length of visit.
• Practice staff participation in CM.
• CM team use.
• Location of CMgr visits with patients.
• CMgr visit documentation.
• Incentives and barriers for CM use.
• PO ownership.
• Provider buy-in.
• Practice culture.

Slide 34

Lesson #2: Know your data

Garbage in = Garbage out

• QCA is not a "turn the crank" type of analysis – can’t just put the data in and get a result; have to go back and forth with it; understand it and what is happening.
• Spend time in practices/environments and get to know how systems work, how people interact; Observe and elicit insight from participants
  • This understanding will be necessary to identify key conditions and to calibrate.

Slide 35

5 PO’s, 8 Care Management Models

Slide 36

Lesson #3: Specify a reasonable outcome

EXAMPLE:

• Qualitatively derived overall "Quality Score"→ THIS PRESENTATION
  • Probably best indicator of quality program, but subject to investigator bias in some cases highly related to the conditions examined
• Qualitatively derived "Normalization Score"
  • Proxy measure of implementation; also subject to investigator bias; Normalization or routinization of poor quality program isn’t a good thing
• Quantitative result - "Patient Engagement"
  • Key problem – difficulties with the limitations of the data which we did not collect; some programs didn’t include all patients – highly dependent on targeting methods and intensity; ideal if we had rates of all patients eligible and solid denominator for targeted
• Quantitative result of "Patient Clinical Outcome"
  • Many factors in line to get to outcome; some we don’t know what they are; but this is likely the ultimate measure; not available yet

Slide 37

Raw Data Table

Note: These data have already been transformed (calculated in fs terms) to degree of set membership

Slide 38

Lesson #4: Careful on selection of cases and sets

Warning: LOTS of time and discussion

• Our team of three met weekly for about 3 months to determine conditions, examine them and run analysis
• Consider what to include as a "row" (case) – PO or practice or provider?
  • Data coded to that level? Outcome available at that level?
  • Nesting of practices within PO’s; Is there variation by practice within a PO? If all practices within a PO act similarly then use the PO level.
  • Our decision - Separate by program structure rather than practice (8 cases).
• Consider what to include as a "column" (set)
  • Analysis gets really messy when getting beyond 8; throwing in lots of extra conditions is likely to NOT yield explanatory value.
  • Be thoughtful about what to include; even if you have a lot of data, that doesn’t mean it is all important – step back and look at larger picture; If no meaningful variation do not include as a set – still write up in analysis, but do not include in the QCA (for example we thought incentives might be important, but no one had any incentives).

Slide 39

Necessity Analysis

• Necessary = condition that the outcome is a subset of
  • EX: quality (outcome) is a subset of physician buy-in (set or condition)
• Scatterplot – all in the bottom right triangle (if outcome on the Y axis)
• Consistency = high relationship strength between the condition and outcome
• Physician buy-in as a condition seems necessary, but not sufficient

Slide 40

Physician Buy-In: Necessary, but not Sufficient

Graph of physician buy-in vs. quality. Consistency=0.993103 and Coverage=0.798521.

Slide 41

Sufficiency Analysis

• Sufficient = condition that is a subset of the outcome
  • EX: PO ownership (set or condition) is a subset of quality (outcome)
• Scatterplot – at the top left triangle (if outcome on Y axis)
• Coverage = high relevance to the outcome
• PO ownership seems sufficient, but not necessary – meaning there are other ways to a good outcome

Slide 42

PO Ownership: Sufficient, but not Necessary

Graph of physician ownership vs. quality. Consistency=0.712644 and Coverage=1.000000.

Slide 43

Necessary and Sufficient

• Necessary
  • Condition must be present for a good outcome.
  • Having the condition does not guarantee a good outcome, can still have a bad outcome.
  • EX: physician buy-in.
• Sufficient
  • Condition(s) alone or in combination will result in a good outcome.
  • Can combine conditions together to create best coverage = combination of conditions that best align with good outcome; unusual to find a single sufficient condition, usually a combination.
  • EX: PO ownership (reason singly sufficient was a proxy for multiple conditions).
• Necessary and sufficient
  • Can be both necessary and sufficient.
  • Can be necessary, but not sufficient.
  • Can be sufficient, but not necessary.
  • Can be neither necessary nor sufficient.

Slide 44

All Patients Included in CM Program: Neither Necessary nor Sufficient

Graph of all patients vs. quality. Consistency=0.797701 and Coverage=.741453.

Slide 45

If you want Good Quality CM Program…

Either:

• Have a team-based approach, have an on-site care manager, have good provider buy-in, plus a good care manager OR
• Be owned by a PO
  • Owned practice comes up sufficient because they demonstrate the conditions (team-based, onsite and have good care managers) – proxy

Slide 46

Truth Table – Truncated (does not include all the unobserved cases)

**There were eight total cases, but two cases did not have >0.5 membership in any configuration so their configurations did not appear in the table; they were consistent with the necessity and sufficiency conditions identified.

Slide 47

Conclusions

• fsQCA can be a valuable tool for health services research if you have…
  • An appropriate study/question.
  • Variability in conditions and outcomes.
  • Meaningful data that tie to an outcome(s).
• QCA seems to work best when a multi-disciplinary team is involved.
• Gathering and analyzing QCA data for meaningful outcomes can be time-intensive and requires in-depth knowledge of your data.

Slide 48

Thank you for attending!

Webinar #2: Statistical Process Control

Presenter: Jill Marsteller, PhD, MPP
Discussant: Stephen Alder, PhD
Moderator: Michael I. Harrison, PhD

Sponsored by AHRQ’s Delivery System Initiative in partnership with the AHRQ PCMH program
May 14, 2013 1 p.m.

Register for this and other events in our series on advanced methods in delivery system research, here: http://bit.ly/EconometricaAHRQ

Slide 49

No Slide Title

For more information about the AHRQ PCMH Research Methods briefs, please visit:

http://www.pcmh.ahrq.gov/portal/server.pt/community/pcmh__home/1483/pcmh_evidence___evaluation_v2