In 2022, an mpox outbreak occurred in countries where local transmission previously had not been observed, including the United States [1]. New York City was the first urban center in the United States to experience a rapid increase in cases [2]. The first case among New York City residents was diagnosed on May 19, 2022 [3]. The next day, New York City health care providers were notified to immediately report suspected cases to the Provider Access Line at the New York City Department of Health and Mental Hygiene (New York City Health Department) for potential testing through the Public Health Laboratory [3]. On June 21, 2022, the New York City Health Department Incident Command System was activated for a public health response, and on July 8, the New York State Department of Health notified health care providers of the availability of commercial laboratory testing for mpox [4]. The New York City Health Department declared a local state of emergency on August 1 [5], and the Secretary of Health and Human Services declared a nationwide public health emergency on August 4 [6]. As the outbreak subsided, the New York City Health Department partially deactivated mpox emergency response activities on October 31 and fully deactivated these activities on January 31, 2023, aligning with the expiration of the US public health emergency declaration [7].

Throughout the emergency response, the New York City Health Department tracked case counts internally and on a public-facing website [8]. Inherent delays (eg, from patient symptom onset to care seeking, laboratory testing, provider and laboratory reporting to the New York City Health Department, and phone interviews with patients to determine the date of onset) make it difficult to interpret recent epidemic trends and make timely decisions during an outbreak. In early August 2022, while reviewing daily epidemic curves with no accounting for data lags, the New York City Health Department leadership inquired whether the outbreak had peaked.

The burden of mpox diagnoses was inequitably distributed by race and ethnicity among patients in the United States [9] and in New York City [10]. Confirmed and probable mpox diagnoses [11] among New York City residents peaked first among White individuals (weeks beginning July 17 and July 24, 2022), then among Black or African American individuals (week beginning July 24, 2022), and then among Hispanic or Latino individuals (week beginning July 31, 2022; Figure 1). Cases then decreased most sharply first among White individuals, then among Black or African American individuals, and then among Hispanic or Latino individuals. Differences in the timing, magnitude, and duration of epidemic peaks by race or ethnicity could reflect, in part, true epidemiologic differences, such as sexual network effects including exposures while traveling early in the outbreak, before local transmission was established, and differences in access to vaccination and treatment [12-16]. In addition, systemic inequities, including heightened stigma, medical mistrust, and inaccessibility of health care services (including financial barriers, inadequate insurance coverage, not having access to a primary care provider, lack of transportation, and lack of convenient care locations) likely contributed to reduced or delayed case ascertainment among Black or African American and Hispanic or Latino individuals [16-21]. Additionally, public health messaging and outreach did not quickly and effectively reach all affected persons, due in part to insufficient accommodation for cultural nuances and linguistic diversity, further contributing to care-seeking delays [17,19,22].

“Nowcasting” refers to predicting the present, and “hindcasting” refers to predicting through the day before the present. Nowcasting and hindcasting methods can be applied to partially accrued reportable disease data to estimate the number of recent events that have not yet been reported [23,24]. Public health agencies have nowcasted various infectious diseases [24-26].

The New York City Health Department first used nowcasting to improve real-time situational awareness during the COVID-19 pandemic public health emergency [24], applying a method called Nowcasting by Bayesian Smoothing (NobBS) [23,27]. NobBS requires a case line list of “date of interest” and “report date” to assess the past delay distribution and epidemic trend and projects the number of cases during a user-specified moving window ending on a date representing “now” [23].

We applied lessons learned from an evaluation of nowcasting COVID-19 [24] to mpox, including (1) using a negative binomial distribution instead of the NobBS default Poisson distribution, (2) using a 2-week moving window length for diagnoses, and (3) removing the display of estimates of diagnoses on weekends, given lack of adjustment for day-of-week effects. Additionally, we wished to nowcast mpox by onset date and to stratify by race or ethnicity, neither of which was previously implemented for COVID-19 at the New York City Health Department. We sent daily automated nowcasting reports to surveillance data leadership starting September 19, 2022; implementation delays were driven by complexities in determining the onset report date and limited staff resources. To monitor differences in epidemic trends across groups, we started stratifying nowcasts by race or ethnicity on September 29.

First, we documented challenges in developing input files for daily hindcasts of mpox cases among New York City residents by diagnosis date and by onset date, overall and stratified by race or ethnicity. Our goal was to provide methodologists developing nowcasting tools with greater insight into how relevant data are collected locally during a public health emergency. Second, we conducted a retrospective evaluation of hindcasting performance for New York City residents diagnosed with confirmed or probable mpox [11] from July 8 through September 30, 2022, capturing the outbreak peak and decline, compared with fully accrued case counts as of September 1, 2023. We used a 14-day moving window for hindcasting by diagnosis date and a 21-day moving window for hindcasting by onset date in real time and assessed whether other moving window lengths or a weekly time unit would have performed better. Third, we assessed mpox hindcast accuracy when stratifying by race or ethnicity.

We used onset, diagnosis, and reporting dates, as well as race and ethnicity data from the New York City Health Department’s mpox surveillance database. Reports were imported electronically from laboratories via the New York State Electronic Clinical Laboratory Reporting System [28,29] and from health care providers via Reporting Central, through the electronic Universal Reporting Form [30]. Information from providers reporting by phone was entered into the surveillance database via on-call physician notes. We included patients who tested positive for either mpox virus (confirmed cases) or orthopoxvirus (probable cases), as detailed in standard case definitions [11].

The Surveillance and Investigations Unit of the Mpox Emergency Response Team at the New York City Health Department conducted patient phone interviews as soon as possible after the initial report of diagnosis to determine risk factors for exposure, identify contacts, and prevent further transmission. These interviews included questions on symptom onset date, self-reported race and ethnicity, and recent history of sexual contact. Responses were entered into the surveillance database.

We selected the relevant “dates of interest” (diagnosis or onset date) and their respective report dates. The diagnosis date was defined as the specimen collection date of the first positive laboratory test, which was ascertained via electronic laboratory reporting. The symptom onset date for mpox illness was elicited during the patient interview and manually entered. The respective report dates were the different dates that the New York City Health Department ascertained the dates of interest. The diagnosis report date was defined as the date the first positive laboratory result indicating a patient met confirmed or probable case criteria [11] was received by the New York City Health Department. The onset report date was calculated based on the source for establishing the onset date, which was most commonly patient interview (Table 1).

We reviewed cases with long (≥50 days) or negative spans between date of interest and its report date to identify cases requiring additional data cleaning. Patients with a missing onset date were excluded from onset nowcasting. Of 2493 patients diagnosed during the study period and with an available onset date, 2099 (84%) had different report dates for diagnosis and onset, with a median of 2 (IQR 1‐4) days between diagnosis report date and onset report date.

On September 1, 2023, we created a frozen analytic line list of mpox cases among New York City residents with the minimum necessary variables to evaluate nowcasting performance—diagnosis date, diagnosis report date, onset date, onset report date, and race or ethnicity. This dataset was separately filtered by patients with diagnosis (n=3305) or known illness onset (n=2278) during the study period, from July 8 through September 30, 2022. We started the study period on July 8, 2022, when commercial laboratory testing became available, and ended on September 30, 2022, because case counts were sparse thereafter (Figure 1).

We characterized the delay distribution from diagnosis to diagnosis report and from onset to onset report by median number of days, IQR, and 90th percentile. We assessed delays overall during the study period and stratified by month and race or ethnicity. We used Kruskal-Wallis tests to assess whether delay distributions varied across race or ethnicity.

We mimicked prospective surveillance on Wednesdays for case counts through Tuesdays by using the R package NobBS (The R Foundation) [27] and restricting to data that had been available at the time. We evaluated hindcast performance across moving window length (14, 21, 28, 35, 42, and 49 days and 2, 3, 4, 5, 6, and 7 weeks), time unit (day vs week), and stratification (overall or stratified by race or ethnicity). For the maximum delay value, we used the NobBS default of the moving window length minus 1.

We chose to mimic surveillance on Wednesdays to balance operational constraints. Hindcast estimates produced on Mondays and Tuesdays could be underestimated because of reduced care-seeking and laboratory reporting on weekends, and hindcasts conducted on Thursdays and Fridays might be received by decision makers too late in the work week to affect that week’s planned public health actions.

To evaluate moving window lengths at daily resolution, we retained the number of estimated cases for each of the prior 7 days. For weekly resolution, we aggregated cases to 7-day periods and retained the estimate for the most recent week. While data from diagnoses on all days of the week were included in model inputs, when conducting the performance evaluation, we evaluated only daily diagnosis estimates from weekdays. This was because diagnoses were reduced on weekends when health care provider availability was more limited. This exclusion did not apply to estimates of onsets or weekly time periods.

Each window length was assessed for periods ending on Tuesdays once the number of days or weeks of that window length had elapsed since the July 8, 2022, start date. For example, we assessed the performance of a 14-day moving window beginning the 14-day period from July 13 through 26, 2022, shifting forward 1 week from July 20 through August 2, and continuing to shift forward 1 week at a time until ending with the period from September 14 through 27, 2022, for a total of 10 models run. For each model, we retained diagnoses for the last 7 days in the window, then excluded weekends, for a total of 50 estimates (5 weekdays from each of 10 models with different end dates). These 50 estimates were used for the performance evaluation. Scenarios with longer moving windows or with weekly aggregation had fewer estimates available for evaluation.

When stratifying by race or ethnicity, we used the “strata” option in NobBS. This option estimated the delay distribution across all race and ethnicity groups and the epidemic curves separately for each group. These analyses were restricted to Black (including African American or Afro-Caribbean), Hispanic or Latino, and White patients because of low case counts in other groups, including Asian, Native Hawaiian or Pacific Islander, and Native American or Alaska Native. We suspected the delay distribution could vary across race or ethnicity groups given differential access to diagnosis and accessibility for interviews, motivating us to compare the accuracy of stratified and unstratified estimates.

For each date of interest (ie, diagnosis or onset), we evaluated groups of estimates—moving window lengths against the lengths used in real time, stratified estimates, and weekly versus daily time units. Drawing from prior evaluations, we evaluated hindcasting performance using the log score [23], mean absolute error (MAE) [24,31,32], relative root mean square error (rRMSE) [24], and 95% prediction interval (PI) coverage [24,32].

We used the log score to evaluate the accuracy of the posterior predictive distribution of each hindcast [23]. We assigned predictive distributions to bins of possible values of fully accrued case counts. For unstratified hindcasts, we used bin widths of 10 cases ranging from 0‐99 for daily hindcasts and of 50 cases ranging from 0‐549 for weekly hindcasts. For stratified hindcasts, we used bin widths of 5 cases ranging from 0‐39 for daily hindcasts and of 20 cases ranging from 0‐179 for weekly hindcasts. These bin widths were selected to yield similar numbers of bins (10, 11, 8, or 9 bins, respectively), to enable comparisons across scenarios with widely varying case volumes. The log score was the natural log of the probability assigned to the bin in which the true count fell [23]. If the probability assigned to the bin for the true count was 0, then we assigned a lower limit log score of −10; this was necessary for only one estimate, for hindcasting for August 23, 2022, by week of onset using a 4-week moving window, stratified among Hispanic or Latino patients. We calculated the average log score across all days or weeks retained for evaluation. We report the exponentiated average log score (ie, average score), which is the average probability NobBS assigned to the bin containing the true case count [23]. Higher average scores indicated more accurate performance.

We also calculated the daily or weekly MAE and average daily or weekly rRMSE across all individual days or weeks evaluated to compare point estimates of hindcasted cases with the final number of cases reported after data accrued. Lower MAE and lower rRMSE indicated better performance, with estimates closer to final counts. MAE is dependent on case volume, making it useful for comparing scenarios with similar case volumes, such as the same time unit and stratification. rRMSE was more useful than MAE for comparing scenarios with different case volumes, which allowed us to compare daily versus weekly and stratified versus unstratified estimates. The 95% PI coverage represents the percentage of estimates when the 95% PI included the final case count; the closer to 95%, the better the performance is. When the 95% PI coverage is near 100%, then PIs might be too wide to be informative.

We checked the dispersion ratio for the entire study period and for shorter periods of 14- and 21-day duration ending on Tuesdays to reflect the window lengths used in real time for diagnosis and onset. This was done using Poisson regression models of counts by each respective date to confirm whether a negative binomial data distribution was appropriate for this dataset.

The New York City Health Department’s institutional review board reviewed this work and determined it to be exempt human participants research under 45 CFR §46.104(d)(4)(ii) and (iii) (IRB No. 22‐097). Analyses were performed using R version 4.2 and NobBS version 0.1.0. The frozen analytic line list, evaluation code, and codebook are available on GitHub [33].

Among 3305 New York City residents diagnosed with mpox from July 8 through September 30, 2022, the median lag in days from diagnosis to diagnosis report was 4 (IQR 3-5, 90th percentile: 6). Lags decreased as the epidemic progressed, from a median lag of 4 days for patients diagnosed in July to 3 days for those diagnosed in September (Table 2). Of 3305 patients diagnosed with mpox, 2558 (77%) were probable cases, with a median lag in days from diagnosis to diagnosis report of 4 (IQR 3‐5, 90th percentile: 6). The remaining 747 (23%) were confirmed cases, with a shorter median lag of 3 (IQR 2‐4, 90th percentile: 5) days (Table 2). Of the 3305 diagnosed patients, 2429 (73%) had a fully or partially completed interview (Table S1 in Multimedia Appendix 1). Typically, the interview was conducted within a median of 1 (IQR 1‐4) day of when the Health Department was notified of a confirmed or probable case, and a median of 10 (IQR 7‐14) days of disease onset. The interview success rate was steady by diagnosis week, with a weekly median of 73% of patients successfully interviewed (range 64%‐80%).

Of patients who were not interviewed, 88% (n=767) had missing onset dates and 28% (n=248) had unknown race or ethnicity (Table S1 in Multimedia Appendix 1). Race or ethnicity distributions were similar between patients who were and were not interviewed, except 39% (n=943) of interviewed patients were Hispanic or Latino, compared with only 21% (n=188) of not interviewed patients (Table S1 in Multimedia Appendix 1). The lower interview success rate among Hispanic or Latino patients could have reduced hindcasting performance for this stratum.

Separately, during the study period, 2278 patients had a recorded mpox illness onset date, and the median lag in days from onset to onset report was 10 (IQR 7‐14, 90th percentile: 18). Lags decreased from a median of 11 days for patients with onset in July to 8 days in September (Table 2). Of 2278 patients with an illness onset date, 1736 (76%) were probable cases, with a median lag from onset to onset report of 10 (IQR 8‐14, 90th percentile: 18) days. The remaining 542 (24%) were confirmed cases, with a shorter median lag of 9 (IQR 6‐13, 90th percentile: 18) days (Table 2).

There was no statistically significant difference at α=.05 across race or ethnicity groups in the lag from diagnosis to diagnosis report or the lag from onset to onset report, overall or in any individual month based on the results of Kruskal-Wallis tests (Table 2). Of 2278 patients with an onset date, 53 (2%) purportedly had onset after diagnosis, representing recall or data entry quality issues. Of the remaining 2225, the median lag in days from onset to diagnosis was 4 (IQR 2‐7, 90th percentile: 10) (Table S2 in Multimedia Appendix 1).

Of 3305 patients diagnosed during this period, 812 (25%) were missing onset date (Table 3). Of these, 230 (28%) also had unknown race or ethnicity (Table 3). Onset date missingness increased with time, from 19% (n=278) for patients diagnosed in July to 31% (n=117) for those diagnosed in September (Table 4).

Counts of cases for the full study period and for 14-day windows by diagnosis date were consistently overdispersed in Poisson regression models by diagnosis date and less so for 21-day windows by onset date (Table S3 in Multimedia Appendix 1), supporting use in NobBS of a negative binomial case distribution.

In evaluating NobBS for the 2022 mpox outbreak in New York City, we faced challenges in developing input files using the onset date. In addition, no moving window length consistently performed best. Daily time units performed better than weekly, and stratifying by race or ethnicity improved performance.

A key challenge in developing input files was that the onset date was frequently missing, which is a common challenge for mpox data collected via patient interviews [34]. When the onset date was available, it was usually after a long delay; the 90th percentile of delay from onset to onset report was 18 days (Table 2), reducing the usefulness of shorter moving window lengths. Furthermore, the onset report date was not a standardized field in our disease surveillance database, which led to implementation delays during the public health emergency. Performance was better when hindcasting by diagnosis date than by onset date, as expected given shorter lags from diagnosis to diagnosis report than from onset to onset report and missingness in onset date.

The choice of moving window length and whether to stratify by race or ethnicity had less influence on hindcasting performance than the choice of aggregating to daily or weekly time units. We had anticipated that with sparsity from relatively few cases in this outbreak, nowcasting at weekly aggregation might improve performance. This was not borne out, possibly because of greater difficulty in estimating the epidemic trend using fewer data points. Hindcasting was more accurate when counts were low and stable, toward the end of the outbreak. Others have also found that forecasting performance metrics varied between early and declining mpox outbreak phases [32]. This underscores the need for nowcasting methods that will reliably perform well as epidemics grow, peak, and decline.

Stratifying by race or ethnicity improved performance, and the highest average scores were observed for White patients. Performance at shorter windows was lowest for hindcasts of Hispanic or Latino patients, possibly due to a lower interview success rate.

Several data quality limitations were noted during project implementation. First, a quarter of diagnosed patients had missing onset dates, which made onset dates less reliable than diagnosis dates for monitoring trends. Patient interviews were the primary source for the onset date. Some patients may have refused interviews due to the sensitive nature of revealing a sexual history in the context of their mpox diagnosis. Generally, surveys about sexual history have participant refusal rates of 25%‐35% [35]. Another reason for missingness is that some patients could not recall their onset date.

As onset dates and race and ethnicity data were often collected during interviews, the stratified and onset-based nowcasts relied on incomplete reports (Table 3, Table S1 in Multimedia Appendix 1). Unstratified, diagnosis-based hindcasts were the only type of hindcast evaluated that relied only on complete and timely laboratory reporting data. Additionally, the median delay from onset to report decreased rapidly from the study start until late August (Figure S4 in Multimedia Appendix 1). Shortening delay distributions could have led NobBS to overestimate onsets in August. Shorter moving windows started with input data from the peak and early decline of the outbreak, while delay distributions and epidemic trends were rapidly changing. Longer moving windows, which appeared to be associated with better average scores, only began once case counts had stabilized, limiting our ability to directly compare window lengths.

Additionally, we included both confirmed and probable cases. Delays for both diagnosis to diagnosis report and onset to onset report were slightly shorter for confirmed than probable cases. While differences in delays by case status were minor, accounting for case status might improve accuracy. Additionally, stratified estimates were limited to Black or African American, Hispanic or Latino, and White patients, while unstratified estimates were for all patients, regardless of race or ethnicity, reducing our ability to directly compare stratified and unstratified estimates.

Although NobBS accounts for reporting delays, it does not account for other limitations of reportable disease data, including underascertainment, underreporting, and misdiagnosis or misclassification [19]. NobBS also does not account for external determinants influencing epidemic trends, such as behavioral changes or public health interventions. Our study period began after commercial laboratory testing became available, which nearly coincided with the epidemic peak, so we were unable to evaluate nowcasting performance during initial epidemic growth. We observed trade-offs in evaluation metrics, for example, scenarios of improved PI coverage with decreased accuracy (Table 5, Tables S4 and S5 in Multimedia Appendix 1), which could be related to overfit models or overconfident PIs. Additionally, the maximum delay used in NobBS of the moving window length minus 1 meant that window lengths were longer than the 90th percentile of observed delays for almost all moving windows. This could explain why changing the window lengths did not have a major impact on performance. Also, lags from diagnosis to report were almost universally less than 1 week, and nowcasting at weekly resolution may not be warranted for such short reporting delays. We did not compare NobBS with other nowcasting methods, such as generalized additive models [34,36], nor did we assess methods developed for the purpose of estimating the time-varying effective reproduction number instead of observed case counts [31].

Accurate nowcasts can facilitate real-time trend monitoring and reporting to policymakers. Stratifying nowcasts by key demographic characteristics associated with inequities, including disaggregated race or ethnicity groups, can help public health authorities quickly identify and remediate inequities faster than monitoring epidemic curves, without accounting for data lags. For example, on November 10, 2022, in the context of declining overall case counts and a focus on ensuring equitable access to interventions, we presented stratified nowcasting results to the Incident Command System leadership, highlighting that the number of recent estimated cases, even with uncertainty, was disproportionately higher among Hispanic or Latino New Yorkers (Figure 3). This finding was borne out after data fully accrued (Figure 4); final daily case counts were within the narrow 95% PIs for estimated case counts.

We recommend stratifying nowcasts to monitor differences in epidemic trends across patient groups and to improve performance, as well as using diagnosis date rather than onset date. For future outbreaks, health departments can strengthen preparedness to rapidly initiate nowcasting during public health emergencies by populating a field for onset report date directly in the surveillance database. Imputing the onset date might be necessary to improve completeness [31].

Performance metrics were sensitive to NobBS implementation details, and no single moving window length emerged as best performing. Health departments need reliable tools to initiate daily nowcasting by diagnosis date within the first few weeks of a public health emergency, to conduct interim performance evaluations to assess accuracy, and to pinpoint which adjustments to make to improve performance while emergencies are ongoing. Tools such as the scoringutils R package [37] could facilitate rapid evaluations and adjustments. Additional practical guidance is needed for health departments on how to optimize nowcasting, including how to add robustness by using multiple distinct methods, and how to best evaluate performance.