Abstract
Objective
To communicate the detection capabilities of syndromic surveillance systems to public health decision makers.
Introduction
Increasingly public health decision-makers are using syndromic surveillance for real-time reassurance and situational awareness in addition to early warning1. Decision-makers using intelligence, including syndromic data, need to understand what the systems are capable of detecting, what they cannot detect and specifically how much reassurance should be inferred when syndromic systems report ‘nothing detected’. In this study we quantify the detection capabilities of syndromic surveillance systems used by Public Health England (PHE).
The key measures for detection capabilities are specificity and sensitivity (although timeliness is also very important for surveillance systems)2. However, measuring the specificity and sensitivity of syndromic surveillance systems is not straight forward. Firstly, syndromic systems are usually multi-purpose and may be better at identifying certain types of public health threat than others. Secondly, whilst it is easy to quantify statistical aberration detection algorithms, surveillance systems involve other stages, including data collection and human decision-making, which also affect detection capabilities. Here, we have taken a ‘systems thinking’ approach to understand potential barriers to detection, and summarize what we know about detection capabilities of syndromic surveillance systems in England.
Methods
Within the systems thinking approach all stages in surveillance (data collection, automated statistical analysis, expert risk assessment and reporting of any aberrations) were considered. Sensitivity and specificity were then calculated for the system as a whole, and the separate impact of each process stage.
To communicate these findings to decision-makers, we created an evidence synthesis. Evidence was synthesised from research involving PHE syndromic surveillance systems and retrospective incidents detected and/or investigated by PHE. We then summarized the evidence for different types of incident.
Results
We identified the following stages which influence detection:
The proportion of people who become symptomatic;
The proportion of symptomatic people who present to different types of health care;
The coding of symptomatic patients;
Coverage of different health care systems by syndromic surveillance;
Statistical algorithms used to identify unusual clusters within syndromic data;
Risk assessment process used to determine action resulting following automated statistical alarms3.
Stages 1 to 3 depend on the type of incident that is affecting peoples’ health or healthcare seeking behaviour: stages 3 to 6 depend on the capabilities of the syndromic surveillance system. In general, each stage increases the time until detection, and reduces sensitivity but should improve specificity.
Our evidence synthesis identified a wide range of threats to public health including: seasonal outbreaks of respiratory infections; allergic rhinitis; insect bites; gastrointestinal outbreaks; air pollution; and heat waves. We ranked the available evidence, giving more weight to actual events detected and validated against independent evidence, and less to purely descriptive epidemiology or modelled simulations. We created different measures for sensitivity, specificity and timeliness depending on the type of evidence available. Sensitivity ranged from 100% for seasonal influenza to 0% for seasonal adenovirus. Specificity also varied, with high specificity where we had a specific syndromic indicators, e.g. sunstroke, and lower for those associated only with more generic multi-purpose indicators e.g. acute respiratory infections. Timeliness varied from being able to provide early warning of up to seven days prior to traditional surveillance methods for some respiratory illnesses, to being able to detect and report on the health impact of air pollution within four days of a period of poor air quality.
Conclusions
This study has shown that a syndromic surveillance systems’ utility depends on more than just an algorithm’s specificity and sensitivity measure. We’ve identified the impact of the different surveillance stages and separately considered different types of incident. Thus, we can identify the impact of issues such as local population coverage and an individual investigator’s risk assessment practices. Furthermore, the evidence synthesis will provide a summary for decision makers, and help identify gaps in our knowledge where more research is required.
References
1. Colon-Gonzalez FJ, Lake IR, Morbey RA, Elliot AJ, Pebody R, Smith GE. A methodological framework for the evaluation of syndromic surveillance systems: a case study of England. BMC Public Health. 2018;18(1):544. http://dx.doi.org/10.1186/s12889-018-5422-9
2. Kleinman KP, Abrams AM. Assessing surveillance using sensitivity, specificity and timeliness. Stat Methods Med Res. 2006;15(5):445-64.
3. Smith GE, Elliot AJ, Ibbotson S, Morbey R, Edeghere O, Hawker J, et al. Novel public health risk assessment process developed to support syndromic surveillance for the 2012 Olympic and Paralympic Games. J Public Health. 2016. http://dx.doi.org/10.1093/pubmed/fdw054