Setting the stage refers to preimplementation activities such as assessing readiness to change, identifying barriers and facilitators to implementing EBIs, selecting or developing strategies to support implementation, and identifying and acquiring resources. Implementation scientists often find that an effective strategy in one setting may not work well in other settings and that some may need more or different types of support (eg, hours of training, intensity of coaching support, or remote vs in-person training). As such, one of the biggest implementation science challenges is to identify what works, for whom, under what circumstances, and with what level of support.

Typical approaches for selecting and tailoring implementation strategies to fit the local context (eg, process mapping, intervention mapping, and coincidence analysis) address this challenge at the organization or population levels [25]. Often, data to inform the selection of an implementation strategy are limited to surveys, qualitative interviews, and organization-level data. However, clinical or public health data (eg, electronic health records [EHRs], administrative data, claims data, patient or disease registries, immunization registries, and health surveys), data linkages (eg, EHR data linked across practice sites, water quality, and air quality), and data related to implementation processes (eg, responses of patients, community members, and practitioners to a specific implementation science strategy from prior studies) are increasingly available. Implementation scientists could use ML to analyze large-scale, individual-level data to identify or predict who (individuals or subpopulations) is most likely (or least likely) to engage or respond to the intervention [26,27]. Specifically, the application of ML in the preimplementation stage could assist with the selection of the settings or actors, refinement of implementation strategies, and decisions about support activities. ML techniques could predict which sites, practitioners, or target populations will most likely respond well to certain implementation strategies (such as a training session or a health information technology tool), are most likely to need extra support, or might respond better to different strategies. These analyses could be based on prior engagement with strategies that led to increased adoption of EBIs or known characteristics of community (eg, census and environmental health), health systems (eg, geographic location), providers (eg, years of practice), patients (eg, race and ethnicity), and other targeted users.

There are currently no studies using ML approaches to tailor implementation strategies or support needs in the preimplementation stage. A few studies have used unsupervised statistical learning methods, such as latent class analysis and latent profile analysis [28], to identify subgroups of health care providers [27] and patients [26] responding differently to implementation strategies that promote provider-patient communication on critical illness or patients’ physical activities for weight reduction. For example, one study identified 3 groups (or phenotypes) of oncologists based on demographics, practice patterns, and patient panel information [27]. These phenotypes showed different responses to an EHR-based intervention (EHR nudges) aimed at improving advance care planning (ACP) discussion. Oncologists with the lowest volume of patients and a higher rate of baseline ACP discussion showed the greatest improvement compared to those with higher volume or lowest baseline ACP and intermediate volume or baseline ACP. One study used a supervised learning model to identify areas where the implementation of HIV prevention programs should be prioritized. Using state surveillance data on substance use, sexually transmitted diseases, and community characteristics (eg, percent living in poverty), ML modeling identified high-priority areas, of which 79% did not have implemented syringe services programs [29]. Similar modeling approaches could be used to better identify who will adopt what implementation strategies with what supports and tailor resource allocation before an implementation program is launched to improve the adoption and sustainability of EBIs.

Further, ML applications during the setting the stage could also facilitate monitoring when interventions are needed. For instance, using continuously collected clinical or public health data and ML-based phenotyping methods [27], it is possible to prioritize target populations who need the EBIs most at different time points or stages of the implementation of an intervention. Modeling could also trigger notifications to local clinics and public health departments about changes in quality metrics that require improvement, the resources needed to make an improvement (eg, additional staff), or changes in an environmental context (eg, climate change) [30] that could impact disease incidences and health care needs.

During the active implementation stage, strategies and support activities are implemented to promote the adoption of an EBI (eg, disease surveillance, prescribing shingles vaccination, and lung cancer screening). During this stage, ML techniques could be incorporated as an implementation strategy. ML-based algorithms relating to the active implementation stage are currently being used to support making accurate diagnoses, disease risk estimation and surveillance, public health campaigns, and clinical decision-making. One example is the use of an ML model to identify foodborne illness in real time (FINDER). This model was developed, implemented, and tested in 2 US cities. FINDER would provide a daily list of restaurants identified as unsafe (likely to have health code violation). Health departments would then conduct an inspection in the restaurants identified by FINDER. The model identified accurately more unsafe restaurants than the previous system or reported complaints [31]. Examples in palliative care include a deep learning model that incorporates patients’ EHR data to predict mortality (those patients most likely to die within 3-12 months). The model-generated estimates were used to inform providers’ care recommendations and decisions about referring patients to palliative care [32,33]. In the context of cancer screening, ML models based on reinforcement learning or ensemble learning are being developed to more accurately identify patients with high risk of cancer [34,35]. These models could be used for cancer screening to balance the benefits of early detection and the costs of overscreening.

Further, in clinical care, clinical decision support (CDS) tools [36,37], including EHR alerts, are common implementation strategies used to promote guideline-concordant practice. ML can be used to develop “smarter” CDS tools to reduce alert fatigue. For example, an ML model was developed to predict whether a provider would respond to shingles vaccination alerts based on the provider’s characteristics (eg, demographics and clinical roles), patient’s demographics, and history of the provider’s interaction with the alerts [38]. The ML model was shown to reduce over 45% of shingles vaccination alerts without reducing weekly shingles vaccination orders [38].

This stage focuses on activities that ensure the sustainability of an intervention. During the monitor, support, and sustain stage, ML can inform changes needed to ensure the adoption and sustainability of practice changes. ML-based methods can leverage vast amounts of data to inform more flexible and adaptive implementation strategies. ML can also facilitate the evaluation and adaptation of strategies and inform where deimplementation is needed. For instance, ML could be used to identify when public health campaigns have reached saturation, need to be refocused, or are missing the target population. For example, during the COVID-19 pandemic, studies use ML models to identify people at greatest risk for COVID-19 death and who should be prioritized for vaccination. Different studies using different populations showed variations in who should be prioritized in informing local public health efforts [39-42]. For example, in clinical practice, implementation scientists leveraged both EHR audit logs and innovative ML-based approaches to monitor the impact of implementing a tobacco control CDS tool in the EHR system [43-45]. According to the Health Information Portability and Accountability Act (HIPAA) [46] and the 2014 release of the Meaningful Use regulations [47], all the EHRs in the United States are required to implement audit logs to unobtrusively track users’ EHR use. In a recent study, a latent-variable statistical ML model was developed to infer EHR-use activities from EHR audit log data [44]. Specifically, the ML model identified topics from EHR log data, where each topic was represented by a probability distribution of microlevel EHR actions such as loading a flow sheet, viewing a problem list, and using a favorite phrase predefined in EHR. Domain experts (3 physicians and 1 EHR specialist) reviewed these topics (eg, the top-ranked microlevel EHR actions belonging to each topic and example EHR sessions representative for each topic) and assigned an EHR-use activity (eg, visit documentation with record review and address CDS alerts) to each topic. This domain expert–informed model was then applied to EHR logs for 3703 encounters (before CDS implementation: n=2633 and after CDS implementation: n=1070) in 4 cancer clinics to monitor changes in providers’ EHR-use between 2019 and 2020 [45]. This study found that clinicians spent more time addressing CDS (more than 32-35 seconds) during a patient visit after CDS implementation (vs before CDS implementation), with compensatory unintended reductions in time spent reviewing patient vital data (less than 61 seconds) and modifying EHR (less than 7-24 seconds) [45]. These findings pointed to potential adaptations of the CDS to improve efficacy and reduce burden [43]. These data-driven findings can inform qualitative studies that aim to understand the causes of the unintended consequences and further inform the decision on refining or deimplementing certain features of the CDS tool.

In summary, despite very few real-world applications of ML in implementation science, there are many options and opportunities to use ML at different stages of implementation; however, some factors are important to take into consideration.

Despite the large amount of clinical data and data from pragmatic implementation trials, there are many challenges associated with data access and data quality. Further, the tools and resources needed to extract and preprocess these data for developing ML may not be easily accessible. For example, extracting and harmonizing patient-level data from the EHRs from multiple health systems to develop a preimplementation ML model could be particularly difficult and time-consuming if these health systems have different EHR vendors. Furthermore, the application of ML in implementation science may result in unintended consequences, and issues related to the sustainability and scalability of the model need to be addressed.

Public health data and information systems vary with regard to data quality, completeness, collection methods by systems, sampling bias, and underreporting [62-64]. In addition, the collection and generation of public health data are often time-consuming, resulting in delays in data reporting. Similarly, clinical-related data, such as EHR or health insurance claims data, are not designed for research and as such may not be collected and recorded in a systematic standard way. For example, comprehensiveness, completeness, and availability of patient demographic information (eg, race or ethnicity), health insurance data, and clinician data vary greatly by health systems and EHR vendors [65-71]. Additionally, some information that can be critical in the accuracy of ML prediction may reside in unstructured data (eg, a scanned PDF and free text of an encounter note) and, therefore, would require additional preprocessing steps, such as natural language processing [72]. Missing clinical-related data are unlikely to be random [70]. Specifically, EHR data come from a combination of clinician notes, test orders and results, documentation of diagnoses, and patient-reported information. The accuracy and completeness of these data are dependent on the source of the information. For example, the history of a cancer diagnosis can be derived from clinician diagnosis, clinical exchange systems, and patient self-reported history. A study linked EHR data with cancer registry to assess the accuracy of cancer diagnosis in the EHR [66]. Authors found that approximately 45% of cases recorded in the registry did not have a cancer history in their EHR. This information may have been in unstructured data such as in encounter notes. Data used for training an ML model may underrepresent certain patient subgroups [71]. For example, the use of insurance claims data excludes patients without health insurance, and these patients are often socioeconomically disadvantaged individuals. Variation in data documentation and completeness impact not only predictor variables used in the ML models but also the outcome variables. For example, predictive models of emergency department admissions using claims data would miss patients who are uninsured and are more likely to rely on the emergency department for care [73]. Moreover, ML models designed to develop an intervention targeting health system, school system, or community-based organization change may require data on staffing, supplies, or organizational capacity, which could be challenging to obtain.

ML models, whether designed for predicting disease risk or for supporting clinical care management and decision-making, are susceptible to bias. Bias can be introduced at multiple points in the development and application process of ML [61,74,75]. As noted earlier, data sources and data representativeness (eg, the population, inclusion or exclusion of diseases, comorbidities, and health risk factors) can greatly influence the ML model and consequently the actions based on the ML model. Further, because ML models can generate data for other ML models, bias can be amplified and can lead to unintended consequences [76]. Char et al [77] proposed a framework for examining ethical considerations of ML models in health care settings, which poses questions about the values and ethics at multiple steps of the model development and implementation. This framework can guide decision-making to minimize bias and can promote accountability and transparency in model development.

Public health interventions and campaigns are moving targets. For instance, climate change is leading public health departments to adapt or develop new initiatives for disaster preparedness efforts, disease surveillance, and carbon footprint reduction [78-80]. For instance, there is growing evidence of the mental health toll of climate-related events [81], yet strategies to monitor and intervene climate-related mental health burden are scarce [78]. Analogously, health care systems are ever-changing [82] as they must adapt to new clinical care guidelines, changes in reimbursement policies, care delivery modality (ie, telemedicine), quality improvement efforts, and local, state, or federal law amendments. For example, in April 2020, the American Society of Colposcopy and Cervical Pathology released new guidelines to provide recommendations on cervical cancer screening frequency and follow-up tests for abnormal cervical cancer results [83]. These guidelines significantly differ from the previous 2012 version [84]. Any implementation strategies designed to facilitate the adoption of the 2012 guidelines became obsolete and needed to be revised. For another example, EHR-based patient portals are efficient systems for communication between patients and health care providers and platforms for health information exchange. These portals can be a platform for patient-centered implementation strategies to improve the uptake of evidence-based practice. Patient portal tools have been used to improve the uptake of ACP or lung cancer screening [85]. Patient portal adoption before the COVID-19 pandemic, however, remained relatively low and varied widely across patient subgroups (eg, by age and socioeconomic status), diminishing the effectiveness of strategies implemented within the portal [86,87]. The need for social distancing and the uptake of telemedicine during the COVID-19 pandemic led to a rise in patient portal use, which could improve the reach of such strategies [88]. The uptake in patient portal during the pandemic was also associated with a rise in “e-visits,” which were communications between patients and clinicians between in-person visits [89,90]. This led to health care systems to bill for these messages following existing federal rules [90,91], which in turn may limit the use of patient portals and impact their effectiveness as an implementation strategy. This example illustrates how the changes in the health care system can impact a specific implementation strategy. Consequently, the reach, adoption, and sustainability of the EBI it aimed to improve are also impacted. These ever-changing systems pose a significant complication when using ML models [92,93]. How frequently should an ML model be adapted or recalibrated to ensure that it has accurate predictions and is unbiased and ethical? This is a critical factor impacting the use of ML in implementation science and across the 3 stages of implementation and remains to be answered by future studies.