Clinical-Community Relationships Evaluation Roadmap

The following priority questions and recommendations are offered as a resource and guide for future research and evaluation into the effectiveness of clinical-community resource relationships for the delivery of clinical preventive services, as well as to stimulate additional thinking about important questions that must be answered to build a sound body of evidence in this area. The questions and recommendations do not describe discrete gaps in evidence, which, if filled, would answer specific research or evaluation questions. Rather, given the current relatively underdeveloped state of research and dearth of measures, the Roadmap provides general guidance for needed next steps. The order in which the priority questions and recommendations are presented does not imply a relative importance. Following most priority questions is a list of illustrative examples of specific research questions relevant to the topic. These lists are not exhaustive, and questions in one topic area may also be applicable to other areas.

2.1 Priority Questions

2.1.1 How Do the Characteristics of Primary Care Clinicians and Clinics, Patients, and Community Resources Influence the Effectiveness of Linkages for the Delivery of Clinical Preventive Services?

Our conceptual framework describes each of the three fundamental elements of a linkage between a primary care clinic and a community resource to provide a patient with a preventive service: clinic/clinician, patient, and community resource. It is known that some characteristics of these three basic elements may influence the receipt or effectiveness of preventive services—for example, a patient's assessed stage of readiness for change, or a clinic's usual workload. However, little is definitively known about how different characteristics may influence the success of linkages between clinics and community resources for that purpose. Examples of relevant questions include:

Primary Care Clinics and Clinicians

• What is the role of information technology infrastructure, and how can it best be used in different contexts to support clinical-community relationships?
• What types of clinical organizations are best suited to participate in clinical-community relationships? Are clinics with certain organizational or financing structures more or less likely to make successful linkages?
• What are the key organizational roles that must be filled at primary care clinics, and how can they best be staffed?
• How does a clinician's level of comfort discussing potentially stigmatized behaviors (e.g., alcohol use, smoking, or sexual activity) influence the success of linkages?

Community Resources

• What types of community organizations are best suited to participate in clinical-community relationships?
• What are the key organizational roles that must be filled at community organizations, and how can they best be staffed?
• How does the organizational infrastructure of a community resource affect its ability to reliably provide feedback to referring clinicians?
• What external conditions in the community might facilitate or hinder the formation or success of clinical-community relationships for preventions?

Patients

• Which patient populations are more or less receptive to receiving clinical preventive services through a clinical-community relationship?
• Does patient health literacy influence the likelihood of a successful linkage?
• How does a patient's level of comfort discussing potentially stigmatized behaviors (e.g., alcohol use, smoking, or sexual activity) influence the success of linkages?
• Does a patient's cultural or religious background influence the effectiveness of a linkage? If so, does it vary for different preventive services?

The answers to questions such as these could begin to distinguish the characteristics that are essential for effective linkages. Understanding which characteristics are essential or not for successful linkages, and in which circumstances, could increase the effectiveness and efficiency of efforts to improve clinical-community resource relationships for delivering preventive services. 

2.1.2 How Do Characteristics of the Clinician-Patient Relationship, the Patient-Community Resource Relationship, and the Clinical-Community Resource Relationship Influence the Effectiveness of Linkages for the Delivery of Clinical Preventive Services?

As with the three basic elements discussed in the preceding question, our conceptual framework describes the three basic dyadic relationships as fundamental aspects of a linkage between a primary care clinic and a community resource to provide a patient with a preventive service. These relationships are also considered because of the conceptual plausibility that their characteristics may influence the success of linkages for delivery of preventive services.

While some characteristics of the clinician-patient relationship (e.g., continuity of care; mutual trust) are known to be associated with better patient outcomes, little is definitively known about how aspects of that relationship may influence the success of linkages between clinics and community resources for preventive services. Illustrative examples of relevant questions include:

• How do the duration and continuity of the clinician-patient relationship influence the success of linkages between clinics and community resources for preventive services?
• Is effective shared decisionmaking different in quality or importance when a preventive service is provided by linkage to a community resource rather than in the clinic?

More has been learned about the characteristics of successful relationships between clinics/clinicians and community resources. Recently, the Institute of Medicine (IOM) convened an expert committee to assess past and current efforts to integrate primary care and public health. This committee's report describes core principles for successful integration, including involving the community in defining and addressing its needs, sustainability, and the collaborative use of data (Institute of Medicine, 2012). The report also notes that integration of primary care and public health can occur on a continuum, similar to Himmelman's schema for a continuum of strategies for working together that includes networking, coordination, cooperation, and collaboration (Himmelman, 2002). While the IOM report addressed primary care and public health integration efforts for colorectal cancer screening, many of its principles might apply more generally to clinical-community relationships for prevention.

In another recent initiative, AHRQ and the Robert Wood Johnson Foundation (RWJF) Prescription for Health program (www.prescriptionforhealth.org ) funded projects to develop and test innovative ways to help patients improve their health behaviors. Many of these projects involved connecting primary care clinics and community resources for preventive services related to tobacco use, unhealthy diet, physical inactivity, and alcohol misuse. In an analysis of eight of these projects, Etz et al., identified promising strategies for connecting primary care clinics and community resources for delivery of these preventive services, including developing referral guides and using external intermediaries (Etz et al., 2008). These recent initiatives provide an important and valuable base for needed future research into the clinical-community resource relationship, which is central to establishing linkages. Still, current understanding of how aspects of the clinical-community resource relationship influence the success of linkages for delivery of preventive services is not definitive or complete. Illustrative examples of relevant questions include:

• To which circumstances are the various degrees of relationship on Himmelman's continuum (e.g., networking, coordination, cooperation, collaboration) best suited? Is this the most useful framework for characterizing the levels or intensity of the relationship?
• What level of relationship intensity is best suited for various preventive services?
• What level of relationship intensity is best suited for particular types of clinics, patients, and/or community resources?
• What qualities in the relationship make for more effective two-way communication about patients and their progress? Does this depend on whether the relationship is limited in terms of duration, service, or patient population?
• How can primary care clinics maintain current information on the existence and availability of relevant community resources?

Finally, little is known about the possible influence of characteristics of the patient-community resource relationship on the likelihood of successful linkages. Illustrative examples of relevant questions include:

• How might continuity of care and mutual trust between the patient and the community resource influence the effectiveness of linkages?
• Do the quality and importance of shared decisionmaking within the community resource setting differ from those of shared decisionmaking between patient and clinician?

As with the three basic elements, future research may begin to distinguish the characteristics of the three basic dyadic relationships that are essential for effective linkages. Understanding which characteristics are essential or not for successful linkages and in which circumstances, could increase the effectiveness and efficiency of efforts to improve clinical-community resource relationships for delivering preventive services. 

2.1.3 How Do the Relative Importance and Influence of Clinics and Clinicians, Patients, and Community Resources and Their Mutual Interrelationships, Vary in Different Circumstances or Contexts, Including the Delivery of Different Clinical Preventive Services?

The six interrelated components that comprise the conceptual framework include three basic elements (clinic/clinician; patient; community resource) and the three dyadic relationships between these three basic elements (clinician-patient relationship; clinical-community resource relationship; patient-community resource relationship). The effects of the six factors combine in the making of linkages for the delivery of clinical preventive services as described in more detail in Appendix A. It is not presumed that each factor exerts an equal influence on the presence or success of a linkage. In fact, the relative influence of factors is generally expected to vary according to the particular preventive service and the specific circumstances of individual clinics, clinicians, patients, community resources, and communities. Little is definitively known, however, about how the relative influence or importance of the factors depends on particular circumstances.

To illustrate, consider the potential variable importance of one factor—the clinical-community resource relationship. While the nature and intensity of successful clinical-community resource relationships may occur along a continuum, it is not known under which circumstances a particular degree of relationship is optimal. The type of community resource may make a difference in the importance of the clinical-community resource relationship. For example, a moderate level of interorganizational coordination might be required to link a patient with a face-to-face tobacco cessation program at the local public health department, whereas no real relationship beyond simple awareness may be necessary for a referral to a telephonic smoking cessation program. In both cases, the patient element may not exert a significant influence on the success of the linkage, aside from being at an adequate stage of readiness to change. Similarly, the type of preventive service may make a difference in the importance of the clinical-community resource relationship. For example, a referral to the same local public health department for counseling about sexually transmitted diseases may require a still higher level of interorganizational cooperation.

Similar scenarios can be described for each of the six basic factors:

• Does the influence of certain patient characteristics (e.g., cultural or religious background) on the success of a linkage vary according to the particular preventive service (e.g., breastfeeding counseling)?
• Are particular types of community resources more successful at linkages in particular types of communities?
• Is the type of clinic organizational structure more important for successful linkages for particular preventive services?
• Which clinical preventive services are more or less suited for delivery through a clinical-community relationship?

Future research that elucidates the circumstances under which the basic six factors are most important and/or influential—individually and relative to each other—could increase the effectiveness and efficiency of efforts to improve clinical-community resource relationships for delivering preventive services. 

2.1.4 What Are the Best Methods, Strategies, and Settings for Studying and Improving Clinical-Community Resource Relationships for the Delivery of Clinical Preventive Services?

With relatively few studies to date, the best approaches to studying and improving clinical-community resource relationships for the delivery of clinical preventive services are still not certain. However, earlier work, such as that conducted through the RWJF- and AHRQ-funded Prescription for Health initiative (www.prescriptionforhealth.org ), has shown the promise of some intervention and evaluative strategies (Cohen et al., 2008; Etz et al., 2008; Green et al., 2008; Holtrop et al., 2008; Krist et al., 2008). As the following limited examples illustrate, many methodological questions remain and much future research is needed.

The Prescription for Health projects were all conducted in practice-based research networks (PBRNs), a seemingly ideal setting for this work. Do PBRNs, as currently designed, adequately involve community and patient partners on an equal footing? How might these valuable networks best organize to conduct research into the effectiveness of clinical-community resource relationships for the delivery of clinical preventive services? Many studies were of complex approaches that combined multiple elements of clinical-community resource relationships and other interventions such as the "five A's" (Whitlock et al., 2002). Little is known about the relative effectiveness of individual elements and combinations of elements of complex interventions—knowledge that might inform more efficient approaches.

While it seems clear that effectiveness research in real-world settings is the most promising general approach, little is known about the best methods and approaches for understanding and accounting for the unique complexity of local circumstances, related to variation in clinics, patients, community resources, and communities. Relevant methodological questions include, but are not limited to:

• What are the best qualitative approaches to understanding the complexity introduced by differences in local context?
• Which qualitative approaches are best suited to which questions and contexts?
• Under which circumstances and for which questions are formal study designs such as cluster-randomized trials most valuable?
• Would studies comparing types and elements of clinical-community resource relationships to other types and elements of clinical-community resource relationships (i.e., comparative effectiveness research) be viable?
• How might methods of implementation science be best applied to research in this area?
• What are the factors that can motivate clinical and community organizations to form relationships for the delivery of clinical preventive services? How important are financial incentives and how can they best be structured?
• What are the barriers to forming and sustaining clinical-community relationships for prevention, and how can they be overcome?
• Given the diversity of local circumstances, what are the best methods for assessing and reporting on unique aspects and external validity of studies?
• What are the most appropriate methods for identifying and reporting on key lessons that may be useful for those doing research or implementing programs in this field?

2.1.5 What Are the Best Measures for Evaluating the Effectiveness of Clinical-Community Resource Relationships for the Delivery of Clinical Preventive Services?

The multifactor complexity of connecting primary care patients with community resources for clinical preventive services presents a myriad of potential factors that might be measured to assess the effectiveness of clinical-community resource relationships for delivering those services. Measures could relate to each of the three basic elements (clinic/clinician; patient; community resource) and each of the three dyadic relationships (clinician-patient relationship; clinical-community resource relationship; patient-community resource relationship) of the conceptual framework. Within each of these six basic factors, multiple measures could relate to numerous different domains. For example, measures related to the clinic/clinician element could include the domains of accessibility, delivery system design, information technology infrastructure, etc. In fact, the Atlas describes 56 possible measurement domains grounded in the conceptual framework.

However, few studies using relevant measures currently exist. It is not known which existing or potential measures would be the most useful, practical, or valid for evaluating the effectiveness and outcomes of clinical-community resource relationships for the delivery of preventive services. A critical goal of future research should be to better understand which are the most useful and relevant domains to measure and what are the best measures to use across various interventions, settings, and contexts.  

Return to Contents

2.2 Recommendations

2.2.1 Researchers Should Use Complex Systems Approaches to Best Understand the Influence of Contextual Issues on the Effectiveness of Clinical-Community Resource Relationships for the Delivery of Clinical Preventive Services

Primary care and communities are complex systems. As others have noted, primary care needs systems approaches to work (Thompson, 2008). Establishing relationships between primary care and community resources to provide patient services further increases the complexity of the resulting interrelated systems. This underscores the need for systems approaches to develop, understand, maintain, and evaluate these relationships. This project's expert panel strongly emphasized the need for research and evaluation using complex systems approaches to best understand the influence of contextual issues on the effectiveness of clinical-community resource relationships for the delivery of clinical preventive services.

The interactions of a clinic/clinician, a patient, and a community resource to create a linkage are inherently complex. Furthermore, these activities function within broader socioeconomic, health care systemic, and community contexts, all of which underscores the critical importance of considering contextual factors in the design, implementation, and evaluation of research into clinical-community resource relationships for delivering preventive services.

While not endorsing one particular approach, the expert panel emphasized that the overarching principles of complex-interactive systems approaches are essential to research and evaluation of clinical-community resource relationships. Useful approaches would entail far more than merely gathering and analyzing quantitative data on various contextual factors. Rather, the best approaches would be intensive and aim for a rich and deep understanding of the complexity of the relationships. Examples might include approaches informed by complexity science, Situational Analysis, Dynamic Systems Modeling, and Realist Evaluation. 

2.2.2 Research on Clinical-Community Resource Relationships Should Be Designed to Flexibly Accommodate Variability in Primary Care Clinic, Patient, and Community Resource Characteristics, Including the Use of Tailored or Semi-Tailored Interventions

As previously noted, local circumstances vary considerably, and the characteristics of clinics/clinicians, patients, and community resources may affect the relevance and success of a particular intervention to improve clinical-community resource relationships for delivering preventive services. Local circumstances may especially present challenges to the successful implementation and evaluation of complex interventions, such as those that would be used to improve linkages for delivering preventive services. In a report based on analyses of projects from the AHRQ and RWJF funded Prescription for Health initiative (www.prescriptionforhealth.org), Cohen et al., describe the importance of adapting interventions to accommodate local circumstances (Cohen et al., 2008). To improve the integration of complex interventions into clinics and community resources, researchers should tailor interventions to better fit with local needs, resources, organizational capacity, values, customs, priorities, and preferences. Cohen et al., emphasize that such adaptations should be done cautiously to maintain adherence to key components of the intervention while modifying components that facilitate integration (Cohen et al., 2008).

Tailoring interventions to fit local needs must be based on a good understanding of the local circumstances, which in turn is dependent on the input and participation of patients, clinics, and community resources. Studies should assess and report on adaptations that were made to the intervention or protocol and how the adaptations affected the success of implementation, integration, and sustainability of interventions. Adaptations are common and usually precipitated by important real-world needs and considerations, and hence could provide rich and useful information—information that is generally lost because it is not reported. 

2.2.3 Researchers and Program Evaluators Should Use and Develop Standard Measures of Relevant Characteristics of the Six Elements and Relationships That Influence the Effectiveness of Linkages for the Delivery of Clinical Preventive Services

The use of validated standard measures can improve the quality of individual research studies and evaluation projects while also improving the ability to assess a body of existing research or programs. A set of valid standard measures of the most relevant characteristics of the six basic elements and relationships would allow for more reliable results and for more accurate and meaningful comparisons across studies or programs. There is an important need for the development and use of such measures by researchers and evaluators. Few studies have used existing measures related to clinical-community resource relationships for delivery of clinical preventive services, and those studies have used common measures of proportion, not specially developed measures. There is both a need to use and test existing common measures that have not heretofore been used in this field of study, as well as a need to develop new specialized measures.

A forthcoming report listing ideas for "candidate" measures, is being produced for AHRQ in conjunction with the Measures Atlas and this Roadmap. The report, to be released later this year, describes potentially valuable measures for use and/or future development.

Given the multifactor complexity of connecting primary care patients with community resources for clinical preventive services and the consequently large number of possible measurement domains, a goal of future research should be to determine which domains are most important to measure for which objectives. The resulting measures should be practical and valid for use across various interventions, settings, and contexts. 

2.2.4 Research Into Clinical-Community Resource Relationships for the Delivery of Clinical Preventive Services Should Be Relevant and Rigorously Designed Using Deep Qualitative Methods as Well as Formal Quantitative Study Designs

Although the majority of the evidence gaps identified by our expert panel were principally due to the relative dearth of literature, many of the studies included in the literature review were inadequately designed, insufficiently rigorous, and/or incompletely reported. Studies using robust qualitative methods for a deep understanding of the effectiveness of interventions were especially lacking. There is a need for much research, which should be rigorously designed and employ a variety of relevant methods.

There is a role for rigorously designed traditional analytic studies, such as cluster randomized controlled trials. Although half of the literature review studies on tobacco use counseling and interventions were of a cluster randomized trial design, there is a relative overall lack of cluster randomized trials of the effectiveness of interventions to establish or improve clinical-community resource relationships for the delivery of clinical preventive services, especially for other preventive services. Although potentially useful for a variety of interventions, cluster randomized trials should especially be considered for testing of highly standardized interventions with high potential for implementation in a variety of settings. Future research should also use meaningful comparison groups and specifically define "usual care"—the relative absence of which was a notable deficiency in many studies included in the literature review. And, studies should include meaningful intermediate and process outcomes, as well as ultimate patient health outcomes.

While cluster randomized trials may be useful for answering many important questions related to clinical-community resource relationships for delivery of preventive services, the complexity of the topic and the variability in local circumstances necessitate the use of rich qualitative methods grounded in a systems perspective. There is a need for studies designed and evaluated with a thorough understanding of the nature and effects of contextual factors. The particular approaches may vary, and could include a range of qualitative data-gathering and evaluative methods, but the goal should be to get beyond a traditional quantitative assessment of an intervention's effectiveness to a deeper understanding of the contextual factors at play. In many situations, mixed-methods studies using quantitative and qualitative approaches together may be the best way to understand the effectiveness and contextual influences of an intervention.

A large range of possible interventions is available to improve clinical-community resource relationships for delivery of preventive services. Future studies should include innovative and effective uses of health care information technology in clinical-community resource relationships. As the evidence base grows, future research should attempt to systematically assess not only the effectiveness of individual interventions , but also their suitability to particular local circumstances by studying what types of interventions are best suited to different preventive services, and what types of interventions most effectively cut across multiple services.

In addition to interventions designed and evaluated as part of formal studies, a variety of quality improvement initiatives have given rise to various interventions to develop and maintain clinical-community resource relationships. The experiences of these "natural experiments" could be a valuable addition to our understanding of effective interventions and contextual factors.  Research is needed on existing successful programs; such research could include ethnographic or "naturalistic" qualitative methods in the relevant communities. 

2.2.5 Research Findings Should Be Reported More Thoroughly and in More Useful Formats

More thorough and useful reporting of research findings is needed. Studies should describe interventions and clinical-community resource relationships in greater detail, including details of distinct elements of the relationship that might be critical to its effectiveness. Given the current lack of consistency across studies, this would allow for better comparisons and conclusions about the effectiveness of different interventions. When relevant, researchers should use standardized methods for thorough and transparent reporting of study results, such as the CONSORT statement (Schulz et al., 2010; Moher et al., 2010) for randomized controlled trials and the STROBE statement (STROBE, 2007) for observational studies. To make the greatest use of their findings, future qualitative research on clinical-community resource relationships should be more thoroughly reported, perhaps with a standard set of qualitative contextual factors. When possible, qualitative researchers should also use standardized methods for thorough and transparent reporting, such as the Consolidated Criteria for Reporting Qualitative Research (COREQ) for interviews and focus groups (Tong et al., 2007). And, given the great variety of local circumstances in which this research occurs, researchers should consider using the Reach, Efficacy/Effectiveness, Adoption, Implementation, Maintenance (RE-AIM) framework (Glasgow et al., 1999) to improve adoption and implementation of effective interventions. 

2.2.6 Studies Should Include Assessments of the Feasibility and Sustainability of Interventions to Improve Clinical-Community Resource Relationships for the Delivery of Clinical Preventive Services, Including Effects on Clinics, Patients, and Community Resources

Many interventions that are found to be effective in formal studies ultimately may not be feasible or sustainable for clinics, patients, or community resources. This may be due to certain artificial or idealized conditions under which the studies are conducted, such as with study funds or extra staff time. Even effectiveness studies intended to reflect real-world conditions may not be successfully implemented by other clinics or communities due to unique contextual factors. Future research on clinical-community resource relationships should consider the effectiveness of implementation, including in particular the feasibility and sustainability of interventions. The Consolidated Framework for Implementation Research (CFIR) is a tool that may be useful for such assessments (Damschroder et al., 2009).

For example, studies should include assessments of the cost of interventions to the clinic practice (e.g., financial cost, staff time, extra roles assumed by staff members), including followup studies for practices that maintain the intervention. There is a related need to study and understand potential "ripple" effects (positive and negative) of a practice change intervention (e.g., medical assistants are used to the highest level of their training to facilitate the practice change intervention and subsequently find a different job). Studies should also assess the cost to patients, including financial cost and time demands, and understand how these factors affect feasibility and sustainability. And, there is a need to study the impact (positive and negative) of interventions on community resources. For example, an intervention to increase clinic referrals for nutritional counseling may increase referrals with the unintended effect of overwhelming the community resource's capacity. Assessing factors such as these, related to successful implementation, may help to improve the feasibility and sustainability of interventions. 

2.2.7 Future Research Could Consider the Conceptual Framework Developed for This Project as a Starting Point That Might Be Further Refined

The conceptual framework that underlies this Roadmap expanded on the idea of a bridge between primary care practices and community resources as described by Etz et al. (Etz et al., 2008). The Etz bridge characterizes the two "anchors"—the clinic/clinician and the community resource—as well as the relationship between the two anchors. Our conceptual framework (Appendix A) added the critical element of the patient to this bridge. Including the patient element with the clinic/clinician and community resource elements produces a framework that represents the combined interactive influences of all three basic elements and their three respective dyadic relationships. We believe that this conceptual framework may serve as a useful guide for next steps in understanding and improving the real-world process of placing actual patients on an existing bridge to connect them with community resources to receive needed preventive services. With the knowledge gained by future research and evaluation, we hope that the relevance and function of its components will be further elucidated and that the conceptual framework may be further developed.

Return to Contents