In this section, we present our assessment of the SATIS-PHI/CRC intervention using the PRISM framework. We begin with the context domain element of evaluating the intervention from the perspective of the organizations implementing and delivering it (the LVPHO acting as the central entity) and serving as the setting for delivery (the participating practices). We also examine the intervention from the perspective of the patients.
We next examine the characteristics of the practices and patients participating in the intervention and assess how these characteristics may have affected its management, receipt, and outcome. We then assess the external environmental setting and how it also may have affected the intervention's management, receipt, and outcome. Finally, we assess the LVPHO central entity's implementation and sustainability infrastructure for evidence of how it may have affected the PHO's ability to implement and maintain the intervention.
We then turn to the outcome domain and assess the five RE-AIM outcome elements of Reach, Effectiveness, Adoption, Implementation, and Maintenance. We present our assessment in a somewhat different order, however. As Figure 3.1 suggests, adoption, implementation, and maintenance are most directly affected by the context domain elements and then, in turn, affect the reach and effectiveness outcome elements. For this reason, we present the outcome assessment by first considering the "AIM" elements and then turning to the "RE" elements.
PRISM Context Domain
Organizational Perspective
Before the central entity decided to implement the intervention, it already knew that this intervention had a strong evidence base. The intervention was based on an intervention previously tested in studies conducted by project staff at Thomas Jefferson University (Myers, 2007; Myers, 2001; Myers, 2004). Components of these prior studies showed that the intervention improved CRC screening rates in a large urban academic practice. The intervention also improved rates of diagnostic followup for positive screens in practices affiliated with a large, for-profit managed care organization. We designed this study to build on these prior studies. We examined how well the intervention could be transferred to a network of community-based practices and achieve similar rate improvements for both CRC screening and followup in a setting distinct from the previous studies.
The intervention also followed the most current (2008) clinical recommendations and guidelines (Levin, et al., 2008; U.S. Preventive Services Task Force, 2008). During the academic detailing sessions to the intervention practices, the central entity distributed and discussed information pertaining to these new guidelines to increase awareness among the clinicians and practice staff. This helped to further emphasize the evidence base of the recommended screening modalities and many of the key intervention elements.
Several factors affected the central entity's readiness to implement the SATIS-PHI/CRC intervention. This was the first time that it embarked on a study of this size. It had conducted prior studies and interventions but none with so many practices. The size increase resulted in a greater time requirement for the central entity to manage and implement the intervention across the 15 practices than originally anticipated, especially in terms of the patient mailings.
This intervention also required a more standardized and systematized approach than the central entity had used previously. This problem was partially compounded by limitations with its health information technology (HIT) system. The HIT staff was more accustomed to conducting retrospective electronic record reviews of billings, claims, and EMR data than prospective queries to determine which patients were eligible and in need of colorectal cancer screening.
In addition, as the central entity's staff conducted subsequent electronic record reviews, they had problems using the electronic records to uncover evidence of screening. Clinicians could enter data on screening in either fixed-response fields or in progress notes or other text fields. Evidence of screening contained in fixed fields was relatively easy to extract. However, such evidence in text fields was more difficult to identify. Further, since specialists rather than the primary care providers in the SATIS-PHI/CRC intervention practices performed colonoscopies, there was only limited evidence of colonoscopy in the primary care EMR or billing records. On the other hand, evidence on FIT screening was provided directly by HNL, making it easy to identify who was screened by this modality.
The timing of OMB clearance also affected readiness. The central entity could not begin to review electronic records to identify eligible patients (Step 3 of the intervention) until OMB cleared the project's data collection. We received this clearance in early December 2008 in the midst of the open enrollment period for the central entity's insurance plan. This timing interfered with the central entity's HIT staff's ability to devote time, staff, and attention to the record review as opposed to updating enrollment records. Record review also had to be delayed until HIT staff finished updating the transition of patients into and out of insurance plans in order to ensure that they could produce an updated list of eligible patients for the intervention. The timing also overlapped with the winter holiday season, further delaying the record review.
Project staff at the central entity also reported that this intervention was ahead of the curve in terms of the readiness of the entity's leadership. At the time of implementation of the intervention, organization's leadership was generally supportive but did not fully appreciate it. The organization is now participating in efforts within Pennsylvania to transform affiliated practices into patient-centered medical homes and now sees how the SATIS-PHI/CRC intervention can be part of population-based screening efforts for a medical home.
To minimize the burden on practices participating in the intervention, the central entity played a coordinating role managing SATIS-PHI/CRC across the practices by working through the office manager of each practice. However, based on lessons learned from the study, the central entity found that it would have been helpful to have a clinical liaison at each practice to help facilitate the intervention's implementation. Whereas the office manger could distribute information and materials, a clinician acting as a liaison with and champion for the intervention may have allowed better and more direct communication with the full clinical staff at each practice.
There were a few aspects that the central entity could not manage, so these elements had to be managed by the individual practices. For example, because HNL required a physician order form for each stool test kit, patients had to return stool test kits to the practice. The practice then attached the order form and sent the kit to the lab for processing, rather than having patients submit the test kits to the central entity for forwarding to the lab or having patients submit kits directly to the lab.
The central entity also did not coordinate directly with colonoscopy practitioners. However, the central entity did coordinate with each practice to compile a list of colonoscopy practitioners to whom a practice's clinicians referred. The central entity included this list in the invitation to screen mailing for that practice's patients.
Because the central entity managed and conducted most of the intervention itself, participating practices incurred no costs or only minimal costs. The central entity also worked with the individual practices to resolve any issues or problems as needed. The central entity learned from the pilot that the practices needed guidance in what they were expected to do to participate in the intervention. Therefore, for the full intervention, the central entity created a process guide for the practices that outlined what the practice and provider should do during each step of the process. It also included a detailed description of how to submit stool tests to the lab for processing and a shortened form for the practices to use when sending the kits to the lab.
Patient Perspective
We designed the intervention focusing on a patient-centered approach. One way we did this was to provide the patient with screening modality choices. The choices were an at-home stool test or a provider-delivered colonoscopy, as well as other screening modalities recommended by providers. Based on existing literature, we understand that some patients are not comfortable with certain screening modalities, so we worked to ensure that several options were available. Based on the patient focus groups, many patients noted that they liked the ease of the at-home stool test kits.
In addition, we wanted to provide patients with a way to ask questions about the intervention. We provided all eligible patients with a phone number for the central entity on the SEA mailing. Through this phone number, patients could ask questions and they could opt out of the study. Patients could also opt out by completing and returning the SEA form. We learned that the central entity phone number and SEA form also provided patients with an opportunity to raise concerns they had with the intervention.
At times, some patients were angry about receiving the mailings and clearly expressed their desire to no longer receive information. Once patients opted out, we removed them from our list of patients to receive subsequent mailings. However, due to timing, materials could have already been in the mail to the patient at the same time that the central entity received their request to opt out. To make this process transparent to patients, our mailing materials noted that there could be delays in updating our contact list, based on mailing time lags.
While the central entity managed and implemented the intervention to minimize the burden on the practices, the intervention mailing materials to the patients appeared to be coming from the patient's doctor and doctor's office. We designed the intervention this way to make it more patient centered. We believed that patients would be more encouraged to screen if the recommendation came from their practice rather than an unknown central entity. Based on the patient focus groups, this was true. Patients reported that they had confidence in the source of the information, as it came from their providers. Some noted that their providers had previously noted the importance of CRC screening, but this intervention helped to encourage them to act.
To help address the seamlessness of transition between program elements and the feedback of results, the central entity provided feedback to clinicians about their patients' screening and followup results. We hoped that if we notified the clinicians, they in turn would notify patients. We also provided clinicians with forms to track the followup test results of their patients with positive screens. The central entity also provided each practice's office manager with a Screening Tracking Sheet to track all of the practice's patients. Based on the patient focus groups, however, we learned that many patients did not receive notification of their test results from their clinicians, which patients found frustrating.
From the perspective of the patients, the timing of the mailing of the intervention was likely not ideal. Due to receiving OMB clearance in December and needing several months for the initial electronic record review, mailing and review of SEA forms, and preparation of the Invitation to Screen mailing, we did not send the invitation and accompanying screening materials until the summer. By then, many patients were on vacation or planning a vacation. Had we been able to better time the intervention from the perspective of the patient, resulting screening rates may have been higher.
We designed the SATIS-PHI/CRC intervention to minimize burdens on patients and maximize accessibility. We supplied stool test kits free of charge to patients and provided information on colonoscopy providers to whom clinicians in their primary care practices referred. Still, from the patient's perspective, the method of obtaining stool test kits was not ideal.
Although we originally planned to have the central entity mail stool test kits directly to all intervention patients, the lab's financial constraints prevented us from being able to do so for all but a small subsample of patients. Most patients had to request a kit by mailing back a request card. This added burden on patients may have reduced screening rates. Uninsured or underinsured patients of LVHN practices were eligible to negotiate reduced pay options with the network, which helped increase their access to screening But this option required a patient to complete an eligibility form. This added burden may have acted as a barrier to their screening.
Recipient Characteristics
Organization
As noted, we conducted the intervention and its assessment in the Lehigh Valley of Pennsylvania through the Lehigh Valley Physician-Hospital Organization (LVPHO). LVPHO was formed by the Lehigh Valley Health Network (LVHN) and the Greater Lehigh Valley Independent Practice Association (GLVIPA) in 1993. LVHN is the region's largest hospital system and health network, primarily serving the two Pennsylvania counties that surround the Lehigh River Valley (Lehigh and Northampton) and that contain the urban centers of Allentown and Bethlehem.
The network of primary care practices affiliated with the LVPHO consists of a mix of primary care practice types and is supportive of practice-based research (as members of the EPICNet PBRN). These practices differ significantly from the sites where components of the CRC screening intervention were previously tested by TJU researchers. The LVPHO practices serve a smaller, less urban community and are members of a physician-hospital organization that offers a preferred provider organization insurance product to local employers.
Whereas many of the LVPHO practices are fully independent, others are members of either (1) Medical Associates of the Lehigh Valley (MATLV, an independent physician-owned professional corporation providing administrative and business services to solo and group practices in more than 25 separate locations throughout the Lehigh Valley); or (2) Lehigh Valley Physicians Group (LVPG, a distributed group of practices owned by the LVHN serving the general Lehigh Valley community). In addition, LVHN operates three hospital-based primary care clinics that help meet the needs of uninsured and underinsured patients in the region. These four entities (LVPHO, LVHN, LVPG, and MATLV) provided us with electronic data that we used for the initial and followup electronic record reviews and to track screenings and results.
Based on our baseline assessment of participating intervention practices through focus groups and surveys, we learned several new organizational characteristics. A complete summary of these findings are available in our Preliminary Report of Findings submitted to AHRQ in September 2009 (Harris and Borsky, 2009). With regard to management support and communication, we learned that when clinicians made screening recommendations they did not follow any specific screening guidelines; rather, they provided patients with screening options. A few indicated that they followed American Cancer Society/Multi-Society Task Force guidelines, and most were at least familiar with them. In addition, many tried to keep up to date with these changes.
We found that most of the larger practices held monthly staff meetings that provided an opportunity for practices to disseminate new information. Some of the smaller practices indicated that information was more often communicated as needed rather than through regularly scheduled meetings, due to staff and provider time limitations.
With regard to data and decision support and management support, overall none of the practices had formal policies for CRC screening; respondents felt it was up to the individual physician to discuss screening with the patient during an office visit. Some said there was a flowsheet in the paper records that recorded screenings, whereas others used an EMR. Some respondents said that they documented recommendations or results in the patient record; others reported using either a health maintenance flowsheet or an EMR. Some did not do any tracking. Many respondents reported following up with patients via the specialist who provided a followup colonoscopy; others reported that there was no followup.
Some of the larger practices said they also used their EMR to help identify who was eligible for screening. Practices without an EMR said they were eager to have an EMR, and they were especially excited to have one that provided popup reminders for screening. However, we also learned that several practices had an EMR but the central entity was unable to use their EMR for determining which patients were eligible for the intervention. The reasons these practices' EMRs were not usable varied, but for some it related to the fact that they did not have the experience of using the EMR for population-based interventions.
Based on the preintervention practice surveys, we learned some additional information about the organizational culture. Some clinicians would recommend screening modalities that were not concordant with the current guidelines (e.g., digital rectal exam), and some clinicians would not recommend screening modalities that followed the guidelines (e.g., FIT). There were also clinicians who recommended screening modalities they did not believe to be effective.
In addition, we learned that clinicians and practice staff were more often aware of the screening process steps that occurred as part of an office visit rather than the steps that required tracking or outreach to patients. However, one strong positive finding was that most clinicians recommended the appropriate followup to positive stool tests. We designed our intervention to address many of these findings by educating clinicians and practice staff about the guideline-recommended screening modalities, appropriate screening followup, importance of tracking patient screening and followup, and all the required steps of the screening and tracking process.
Patients
The population served by LVHN includes 620,425 people, of whom 177,078 (28.5 percent) are ages 50-79, the target age population for this CRC screening intervention. Data provided by the LVPHO, based on Medstat Demographics expert, indicate that 3.6 percent of the area's population was non-Hispanic black or African American, 2.3 percent was Asian, and 11.0 percent was Hispanic or Latino. The fastest growing segments of the population are Hispanic and Asian according to figures cited by the Lehigh Valley Economic Development Corporation and supported the Medstat data. A recent cancer mortality report (Pennsylvania Department of Health, 2009) showed 1,824 new cases of invasive CRC in Lehigh and Northampton Counties in 2002-2006, with age-adjusted annual incidence rates per 100,000 as shown in Table 4.1.
As noted, the central entity conducted an electronic record review to identify patients eligible for screening. After the subsequent SEA mailing, additional electronic record reviews, and chart audits, we obtained our final patient study population. Table 4.2 summarizes the demographics of the patient population. The table shows patients from the two intervention arms (kit and card) and from the control group.
As shown in the table, the control group patients are younger than the intervention group patients, but they have roughly the same proportion of males and females. We do not have insurance data for the control practices, but we assume that because the control group population is younger, they are more likely to be covered by commercial or Medicaid insurance or be self-insured rather than covered by Medicare. There is also significant variation between the intervention and control group patients for the practice characteristics. As there is such variation, we controlled for these factors when we conducted our statistical analyses.
We were able to gather information regarding the patient's knowledge and beliefs about CRC screening from our patient focus groups. Focus group participants clearly identified CRC screening with turning age 50 and with life milestones. Their knowledge sources were medical providers or family and friends. For family, this also related to the impact of having a family history of CRC or other cancer.
With regard to factors motivating people to get screened, respondents indicated that they sought to be proactive about their health. They also noted that family ties played a significant motivating role. However, fear can be a barrier to screening (e.g., colonoscopy prep). The focus groups also provided us with information about the patient's satisfaction with the intervention, which we describe later in this report.
External Environment
We implemented the SATIS-PHI/CRC intervention in Pennsylvania's Lehigh Valley. This setting primarily consists of two counties (Lehigh and Northampton) and contains the urban centers of Allentown and Bethlehem. LVHN is the region's largest hospital system and health network. Along with the GLVIPA, LVHN formed the LVPHO in 1993. The PHO offers a preferred provider organization health insurance plan to employers throughout the region.
Primary care practices in the PHO have varying degrees of affiliation with LVHN. The network owns and operates three hospital-based clinics that help meet the health care needs of uninsured and underinsured residents of the region. Two of these clinics are associated with residency programs. LVHN also owns and operates the practices of the Lehigh Valley Physician Group. Other practices are not owned by LVHN. Some, however, are members of Medical Associates of the Lehigh Valley, which is an independent, physician-owned professional corporation providing administrative and business services to solo and group practices throughout the Lehigh Valley. The remaining practices are neither owned nor operated by any larger entity.
Factors Affecting Implementation
Several notable conditions or occurrences in this environment affected the implementation of the SATIS-PHI/CRC intervention and perhaps its outcome as well. The downturn in the national and local economies that began in calendar year 2008 resulted in a narrowing of operating margins for many of the entities playing a role in the intervention effort. In particular, these economic conditions led to decisions and actions by LVHN and its affiliated PHO and clinical laboratory (HNL) that affected the intervention. The network was limited in its ability to provide in-kind funding and staff resources needed to supplement the funding available through the task order contract. This affected the ability of the LVHN/LVPHO study staff to meet deadlines for electronic record review, mailings, and screening tracking.
Perhaps more significantly, the economy affected the ability of HNL to supply stool blood test kits free of charge for mailing to all intervention patients, which necessitated changing our intervention implementation protocol. The laboratory needed a high enough response rate to the invitation to be screened by stool test so that the revenue generated by charging patient insurance plans for processing the test kits would offset the cost of supplying the kit free of charge. Based on a lower than expected return rate in the intervention pilot, the laboratory informed the project's implementation team that it could not afford to provide stool test kits free to thousands of patients included in the full intervention. The laboratory required either that we substantially reduce the number of kits mailed out, that we assure them of a higher response rate, or both. To meet this requirement, the study team revised the intervention protocol for the invitation-to-be-screened mailing to enclose a card for patients to mail back to request a kit rather than to enclose the kit. In this way, the laboratory only had to provide kits for those requesting one and the response rate was likely to be higher among those requesting a kit than experienced in the pilot.
We were concerned that the change in protocol would add a burden from the patient's perspective and could significantly reduce the screening rate resulting from the intervention. We wanted to be able to estimate the effect of changing from enclosing the kit to enclosing the request card, so we negotiated with HNL to supply up to 500 kits for us to enclose with the invitation-to-be-screened mailing for a sample of patients. We had already mailed the invitation to Wave 1 patients by the time we reached this agreement with HNL, so we selected the two largest practices in Wave 2 to participate in this substudy. We randomly selected a subsample of each of these practice's patients to receive the kit rather than the card.
The economic environment also led to increased unemployment in the Lehigh Valley, with an accompanying loss of employer-based health insurance coverage. The regional unemployment rate rose to a 25-year high of 9.3 percent in August 2009, which was the second highest rate for a Pennsylvania metropolitan area. Anecdotally, we heard reports that even those who maintained their insurance coverage were experiencing higher copayments and deductibles. Loss of insurance or higher out-of-pocket costs for insured patients may have affected the rate at which patients targeted by the intervention were screened.
Another environmental condition likely affecting the intervention is the number and availability of local colonoscopy providers. To the extent that demand for colonoscopies put pressure on the supply of these providers as a result of the intervention, waiting times for screening colonoscopies likely increased. This situation may have led to some screens being delayed until after the intervention observation period was over, thus decreasing the observed effectiveness of the intervention.
Screening delays may have been most acute for those who are uninsured and underinsured. Local colonoscopy providers offer screening procedures to this population on a limited basis (again, the economic conditions may have worsened this situation). As discovered in key informant interviews and focus groups with hospital clinic personnel, the waiting list for screening colonoscopy for uninsured and underinsured patients may have been as long as 10 months, exceeding our 8-month observation period.
Possibly exacerbating this condition was the initiation of a CRC screening program sponsored by a local Blue Cross Blue Shield plan offering several insurance products (including a Medicare product) to residents of the Lehigh Valley. This program was launched just prior to our SATIS-PHI/CRC intervention and likely increased demand for screening colonoscopies, which would have spilled over into the observation period for our intervention. Further, patients of practices participating in SATIS-PHI/CRC who had health insurance coverage through one of these Blue Cross Blue Shield products would have already received material from the other program before receiving our intervention material.
To avoid confounding our estimation of the effect of SATIS-PHI/CRC, we decided to exclude from the intervention all patients in participating practices who had insurance coverage through a Blue Cross Blue Shield products. This exclusion also reduced the potential of confusion among patients who would be receiving different screening invitations and protocols from both the Blue Cross Blue Shield program and our intervention.
The actions of the Blue Cross Blue Shield plan reflected findings of a study of commercial insurers throughout Pennsylvania conducted in 2006. This study found that they generally changed their policies and practices to be more supportive of increasing CRC screening rates after the National Committee for Quality Assurance included CRC screening as a measure in its Healthcare Effectiveness Data and Information Set in 2003 (Sarfaty and Myers, 2008). These changes included implementing or revising guidelines for screening, initiating measurement of screening rates, developing reminder systems for patients, and developing tracking systems. SATIS-PHI/CRC can be seen as an effort consistent with and complementary to these other efforts within Pennsylvania.
Changes in Primary Care Delivery
Two developments in the delivery of primary care throughout the Lehigh Valley also likely affected the SATIS-PHI/CRC intervention. Many primary care practices, including those participating in SATIS-PHI/CRC, were implementing, upgrading, or still learning how best to use various EMR systems. We found that we were unable to obtain needed patient eligibility information from several of these systems in a timely enough manner to include them in the intervention. Further, the delay in obtaining eligibility data from some practices necessitated our splitting the first mailout into two waves, placing the slower practices in Wave 2.
Elements of some EMR systems that would have facilitated our ability to differentiate between ineligible and eligible patients, as well as our ability to track screening and followup, were not being used at some practices. Two practices were transitioning to a new EMR system during the intervention period, requiring study staff to search both old and new systems at each practice. The fluid EMR situation at these practices also negatively affected their postintervention survey response rates. In general, the HIT environment in the Lehigh Valley was an issue for implementing SATIS-PHI/CRC.
The second primary care delivery development occurring in the Lehigh Valley at the time of the intervention was the transition to a patient-centered medical home model of care. There were statewide initiatives and regional initiatives in southeast Pennsylvania sponsored by both the Governor's office and insurance plans serving the region. The purpose of the initiatives was to transform primary care practices into patient-centered medical homes and to institute pay-for-performance programs tied to medical home status and performance on key quality indicators. These developments may have detracted from attention to or interest in SATIS-PHI/CRC among LVPHO leadership since CRC screening was not a focus or key indicator of the medical home initiatives. Going forward, however, these initiatives are likely to facilitate dissemination and uptake of SATIS-PHI/CRC throughout the LVPHO practice network because CRC screening is becoming a focus.
The incidence of seasonal flu and h3N1 and concern over them led to higher than usual volume at primary care practices in the Lehigh Valley during the intervention period. This higher volume left little time for either providers or practice staff to devote to SATIS-PHI/CRC concerns, such as preparing returned stool test kits for submission to HNL for processing. In addition, staff did not have much time to respond to patient inquiries about the intervention or CRC screening.
Implementation and Sustainability Infrastructure
The LVHN/LVPHO central entity had a dedicated team of four people devoting part of their time to implementing and sustaining SATIS-PHI/CRC. This team recruited practices to participate in the project, extracted or arranged for extraction of electronic data, conducted all of the patient mailings, input data into the master patient database, and acted as liaison to the practices. Members of this team also participated with other project personnel from Thomas Jefferson University in conducting the academic detailing sessions and the practice and patient focus groups. This team also had in-kind assistance from the LVHN in preparing material for and carrying out the patient mailings. Nevertheless, the team was understaffed for what it was expected to do and frequently missed deadlines. The team was adaptable to changing situations (e.g., the change in protocol to use request cards instead of distributing stool test kits to all intervention patients, as well as adding an academic booster when the preintervention survey results indicated the need for one).
The central entity team had ongoing established relationships with each of the participating practices through EPICNet. Specific to this project, the team arranged to have a practice management staff person at each practice serve as the primary point of contact for the intervention. A number of issues that arose during the intervention implementation were clinical rather than administrative or required direct communication with the clinical staff. A clinical point of contact at each practice, such as a nurse or clinician, would have been advantageous.
The participating practices by and large did not have any population health infrastructure for an outreach screening program such as SATIS-PHI/CRC. To a large extent, the role of the SATIS-PHI/CRC central entity was to be that infrastructure for the practices: to take the place of having such an infrastructure at the practices. However, in addition to an implementation infrastructure, interventions such as SATIS-PHI/CRC benefit from a public health culture within the practices. The central entity could not compensate for the lack of such a culture in many of the participating practices.
Even though the central entity provided the infrastructure for screening outreach, the practices still needed to cooperate by providing clean and up-to-date patient population data to feed into that infrastructure (for record review, mailings to patients, and tracking screening) and to use the feedback and tracking tools and aids made available to them by the central entity. To do so, the practices would have needed to place a higher priority on these activities than they appeared willing or able to do without a strong public health culture. As reported in our preliminary report (Harris and Borsky, 2009), the results of the preintervention survey indicated that the more public health-oriented steps in the screening process were not performed in many intervention practices. Academic detailing could perhaps be expanded in future implementation of SATIS-PHI/CRC to include inculcating a public health culture.
Health information technology personnel at LVPHO and at participating practices were not familiar with the types of queries required to identify eligibility for SATIS-PHI/CRC. They found it difficult to extract the required information from their electronic systems. The required queries were more prospective (who needs to be screened) than retrospective (who was screened) in nature, and they were more familiar and comfortable with retrospective queries. Those practices using billing data for eligibility found it difficult to extract population data for this purpose.
Several data analysis and programming personnel providing data for the electronic record review left their jobs shortly after the initial eligibility review. Their replacements chose not to use the data extraction programs written by them, resulting in the need for these programs to be rewritten. This task delayed our ability to obtain ongoing updated data (i.e., screening results).
