Abstract
Purpose: To validate the utility and effectiveness of a standardized tool for prioritization of information sources for early detection of diseases.
Methods: The tool was developed with input from diverse public health experts garnered through survey. Ten raters used the tool to evaluate ten information sources and reliability among raters was computed. The Proc mixed procedure with random effect statement and SAS Macros were used to compute multiple raters’ Fleiss Kappa agreement and Kendall''s Coefficient of Concordance.
Results: Ten disparate information sources evaluated obtained the following composite scores: ProMed 91%; WAHID 90%; Eurosurv 87%; MediSys 85%; SciDaily 84%; EurekAl 83%; CSHB 78%; GermTrax 75%; Google 74%; and CBC 70%. A Fleiss Kappa agreement of 50.7% was obtained for ten information sources and 72.5% for a sub-set of five sources rated, which is substantial agreement validating the utility and effectiveness of the tool.
Conclusion: This study validated the utility and effectiveness of the standardized criteria tool and was used to identify five information sources suited for use by the KIWI system for a pilot project focusing on emerging and zoonotic diseases. The tool can be used in prioritizing a plethora of information sources to improve early detection of diseases.