Reimagining Healthcare Teams: Leveraging the Patient-Clinician-AI Triad To Improve Diagnostic Safety

Ideally, members of the PCA diagnostic team will support and augment each other’s performance. As diagnostic PCA teams are envisioned, we recommend extending the team-based healthcare principles IOM proposed to include AI as a member of the diagnostic team. These principles include establishing shared goals, clear roles, mutual trust, effective communication, and measurable processes and outcomes (Table 1).¹

Table 1. Principles for patient-clinician-AI diagnostic teams

Shared Goals

In PCA teams, the process of establishing diagnostic goals moves from a bidirectional exchange between the patient and the clinician to a tridirectional exchange. In a tridirectional exchange, patients and clinicians receive information from AI systems, AI systems receive information from patients and clinicians, and patients and clinicians receive information from one another.

Each member of the PCA diagnostic team brings their own unique goals to the relationship. For example, clinicians’ goals may include providing safe, efficient, evidence-based care. Likewise, AI algorithms are ideally designed with the goal of effectively performing specific tasks (e.g. recognizing an arrhythmia), often with little consideration of the patients’ or clinicians’ context.

The patient’s values, preferences, and circumstances should provide the foundation on which team goals are established. Patients, their families, or other caregivers must be comfortable describing their unique situation and their wishes. Clinicians should be adept at patient-centered communication to effectively guide these conversations. In addition, AI should be designed and deployed in a way that is conducive to meeting specific diagnostic goals. Ultimately, clinicians will be responsible for ensuring that the goals of each team member are carefully considered and that the team’s shared goals coalesce around patients’ values, preferences, and circumstances.

Clear Roles

Clear, specific delineation of roles within the PCA team is essential to the team’s success. In most cases, the diagnostic process is triggered when a patient notes a specific sign or symptom and notifies their healthcare provider. Increasingly, the diagnostic process may be triggered by patients wearing direct-to-consumer AI-based technologies.

Ideally, these AI alerts will lead to the patient engaging with the healthcare system and providing the information needed for accurate and timely diagnosis and treatment of their health concern. For instance, a patient may be notified by their smart watch (AI role) that they are experiencing tachycardia or an abnormal heart rhythm, which leads to the patient notifying their healthcare provider (patient role). In this situation, a clinician would need to be familiar with the validity, reliability, and intended use of the AI algorithm and the patient’s context to determine the need for additional evaluation (clinician role).

Another use case involves the clinician recommending a specific wearable or other AI-based technology to perform a specific task (e.g., information gathering, integration of information,) for diagnostic purposes. For example, a clinician may recommend a wearable technology to assess a patient’s fall risk.²⁹ The patient, family member, or other caregiver would be responsible for ensuring the technology is worn, while the AI would provide daily summaries about fall risk and near falls and notify family members or the clinician if a fall occurs. This information could lead to interventions to prevent falls.

While patient and clinician autonomy should be preserved, AI will have variable levels of autonomy depending on the diagnostic problem. In addition, clinicians and patients will have varying levels of comfort with the level of autonomy assigned to AI systems. For example, in some cases AI will provide a diagnosis independent of a clinician. The AI-based IDx-DR system, which autonomously diagnoses diabetic retinopathy without clinician overreading, is an example of a fully autonomous model.³⁰

Alternatively, an algorithm may alert clinicians to an increased probability of a diagnosis, leading to more efficient triage and diagnosis of a problem. For instance, an AI-based system designed to identify intracranial hemorrhage on CT images may lead radiologists to prioritize earlier reading of CT scans designated as high risk.³¹

Finally, we acknowledge the rapidly evolving field of AI. Patients, clinicians, and developers will need to be nimble and capable of adapting as roles are likely to change. For example, referring to the previous example, one can imagine AI directly notifying the clinician of actionable information (responsibility moves from the patient to AI), leading to more efficient diagnosis. This type of change may require an adjustment of processes and workflow to accommodate the new stream of information from a different source that is presented in a different way and perhaps at different times.

Mutual Trust

Assigning roles may prove futile if trust between team members is not established. Within the PCA triad, mutual trust is needed between patients and clinicians, and patients and clinicians need to trust the AI system’s ability to meaningfully contribute to the diagnostic process. How to establish trust between AI and patients/clinicians is an area of active research.^32,33

Rojas and colleagues suggested that clinician trust in AI be informed by the system’s fairness, transparency, and performance.³⁴ To this end, the Office of the National Coordinator for Health Information Technology recently proposed the HTI-1 rule, which would require users of clinical decision support systems using AI to have access to answers to three basic questions³⁵:

1. What data were used to train the algorithm?
2. How should the predictive algorithm be used, updated, and maintained?
3. How does the algorithm perform using fairness metrics in testing and in local data, if available?

As described above, in some cases clinicians will introduce diagnostic AI algorithms to patients. In other cases, patients will bring new AI-based technologies to clinicians. Regardless of the initiator, clinicians will need to develop the skills to critically evaluate AI-based diagnostic systems and make recommendations about their use. Thus, prospective studies are needed that demonstrate AI systems’ reliability, validity, and positive effect on important patient outcomes.

As with other healthcare interventions, patients are unlikely to adhere to plans that include AI if they do not trust the provider recommending the technology or do not trust the technology itself. These feelings may be especially common among historically marginalized groups.^36,37 Biased algorithms can worsen health inequities, as described by Obermeyer and colleagues. They showed that a widely used algorithm demonstrated signficant racial bias, as it disproportionately recommended additional assistance for White patients compared with Black patients, despite Black patients being sicker.³⁸ To mitigage bias and increase patient trust, rigorous strategies must be used during the development and implementation of algorithms.³⁹

Similarly, clinicians will need to trust that AI-based technologies are safe and effective before recommending them to patients. In addition, clinicians must trust that patients are experts about their experiences, context, and values.⁴⁰ Effective communication between all team members will be essential to developing the mutual trust needed for optimal functioning of the PCA team.

Clinicians will need education and training related to the use of AI/ML in the diagnostic process, which is beyond the scope of this brief. However, we acknowledge that they will need to become adept at appraising information describing AI/ML interventions, applying the output of diagnostic models, and communicating the role AI/ML played in the diagnostic process.⁴¹ This education and training will also be essential to avoiding underreliance or overreliance on AI/ML in the diagnostic process.

Effective Communication

Communication between humans and AI comes in many forms. For example, patients may communicate with AI systems by speaking; entering text, photos, or other types of data into the system; or using wearable devices (e.g., smart watches) that provide continuous monitoring. The AI communicates by producing an output in the form of an alert, reminder, probability, diagnosis, recommendation, or intervention.

AI communication with patients will depend on its purpose, goals, and FDA classification. For example, non-FDA-regulated technologies should not provide a diagnosis and may instead provide an alert to a potential abnormality (e.g., bradycardia). Outputs of FDA-regulated technologies will depend on the risk classification of the technology, with higher risk models producing outputs within the context of high-risk situations (e.g., ML-based cardiac defibrillator).

In fall 2022, the FDA provided updated guidance on designation of software as a medical device (SaMD).⁴² Previously, most EHR software did not meet SaMD criteria. However, this situation is changing as EHRs, hubs for healthcare communication, are increasingly incorporating AI-based predictive algorithms designed to augment diagnostic decision making.”^34,43

Clinician interaction with diagnostic AI algorithms will most often occur via the EHR or patient-owned wearable devices. Developers and health system leaders should proceed with caution to avoid making an already overwhelming EHR workload unmanageable with the addition of AI. For example, clinicians could quickly become discouraged by AI if they open the EHR to find patient-generated data lacking clinical context for multiple patients. While the data may contain valuable diagnostic information, attention must be given to the form and frequency of presentation of these data.

Patients and clinicians should be able to seamlessly communicate with AI. Therefore, anyone developing or implementing these models should consider usability. Patients and clinicians are not likely to engage with these potentially useful technologies if they significantly disrupt lifestyle or clinical workflow, resulting in inconvenience and inefficiency. Furthermore, AI outputs should be presented to patients and clinicians at the right time, with appropriate frequency, in a clear, concise, and user-centered manner. Optimal output is likely to minimize alert fatigue, or desensitization to alerts, helping to avoid diagnostic errors that result from inappropriate or inadequate use.

Measurable Processes and Outcomes

Re-evaluation of the PCA diagnostic team structure and function should occur at regular intervals. Patients and clinicians may address this issue during followup visits or via patient- or clinician-initiated portal communications. Another option is to discuss the matter with other members of the healthcare team (e.g., nurses, medical assistants, pharmacists, care navigators).

Patients should have the opportunity to provide anonymous feedback to clinicians and healthcare organizations via patient satisfaction surveys. The team’s effectiveness should be measured based on the shared goals set at the beginning of the diagnostic process. The team should consider the patient, clinician, and AI-related factors that positively or negatively impact the diagnostic process. When possible, team members should share thoughtful, constructive feedback with each other.

Clinicians should be comfortable communicating with health system leadership about AI performance. This feedback may lead to improvement of model performance and usability, as well as iterative updates similar to upgrades for EHRs. Ultimately, we imagine systems in which feedback can also be provided directly to the AI systems as it learns and adapts over time. Similarly, as alluded to above, AI might provide feedback to clinicians about how they might improve delivery of patient care.

The goal is for the diagnostic team’s performance to be augmented by the collective contribution of each team member in a way that allows efficient and effective diagnosis of the patient’s medical problem. Therefore, consistent with the diagnostic process, health systems and clinicians should learn from diagnostic errors, near-misses, and accurate, timely diagnoses that result from the PCA diagnostic teams.