I am a PhD student in health in disasters and emergencies in Tehran university of medical science, School of public health
Disasters, whether natural or man-made, are common events worldwide
The establishment of an effective SS in a disaster or emergency setting is a complex and difficult process requiring a large number of resources including manpower, equipment, and administrative facilities. To determine whether CDS meets target goals, it is necessary to measure the performance of these SS
An appropriate and unique assessment system that is established according to disaster characteristics is required for the monitoring of SS. An effective assessment system should include appropriate indices and should be conducted in a correct method. Such measures can improve health system response as well
Management experts believe that “what cannot be measured cannot be managed.” Therefore, the measurement of performance is considered one of the most important components of efficacy and effectiveness
With regard to the importance of performance assessment and its role in improving the performance of communicable diseases SS in response to disasters, there are 2 main questions:
What kind of performance assessment frameworks, indices, and criteria currently exist for CDS systems in response to disasters and other emergencies, including infectious diseases outbreaks? What are the characteristics of these frameworks, indices, and criteria? The aim of this systematic literature review was to find the answers to these questions.
Before starting this systematic review, a written review manual was developed for searching strategy, study inclusion, and exclusion criteria. The study was then conducted according to this review process.We have reported our review according to the PRISMA guideline.
The review aimed to answer the following questions:
1. What kind of performance assessment frameworks currently exist for CDS systems in disasters and other emergencies, including infectious diseases outbreaks?
2. What criteria and indices are used in the performance assessment of CDS systems in response to disasters and emergencies?
3. What are the characteristics of the articles on the performance assessment of CDS systems in terms of article type, study approach (qualitative or quantitative), study setting, results, hazard type, geographical location, and country or affiliation of first/corresponding authors?
For the purpose of this study a “communicable disease” is “a disease caused by living agents as infectious agents, or their products, that can be transmitted from 1 patient to another” (Synonym: infectious disease) In the literature, “surveillance” is “the ongoing, systematic collection, analysis, interpretation, and dissemination of data regarding communicable disease for use in developing preventive actions to reduce morbidity, mortality, and to improve health” A “disaster” is “a man-made or natural event that disrupts the affected community functions and results in widespread losses that are greater than community resources” An “emergency” is “a condition that needs urgent attention and could become a disaster if not managed effectively.” In this study, both natural and man-made disasters and emergencies were included. Smith defines “performance assessment” as “a systematic process that seeks to monitor, evaluate, and communicate the extent to which various aspects of a system meet its key objectives”
The following criteria were used to select relevant studies:
Articles that were published in peer-reviewed journals and had addressed the performance assessment of CDS in response to disasters and emergencies (as defined above). Articles in any format including editorials, case reports, reviews, and original research.
All non-English articles, unless an English abstract was available. Papers with abstracts that were not accessible or did not include enough information for extraction of the study variables.
We searched 5 electronic databases, including PubMed, Scopus, Google Scholar, ScienceDirect, and ProQuest. The databases were searched for articles published up to the end of 2013. In addition grey literature[1] was searched through the "New York academy of medicine grey literature reports"
[1] - Grey literature definition: "That which is produced on all levels of government, academics, business and industry in print and electronic formats, but which is not controlled by commercial publishers."
We chose key terms and developed a search strategy based on the National Library of Medicine "Medical Subjects Headings (MeSH)".
The following search strategy was applied in the PubMed database: “(disasters [Title/Abstract]) AND surveillance [Title/Abstract], (disasters [Title/Abstract]) AND communicable diseases [Title/Abstract], (emergencies [Title/Abstract]) AND surveillance [Title/Abstract], (emergencies [Title/Abstract]) AND communicable diseases [Title/Abstract]".
To search the other databases, the PubMed search strategy was adopted. We limited our search to titles and abstracts of articles.
First, the selected key words were entered into the database search boxes, and the search was limited to abstracts and titles. The results of the key words search were reviewed by a member of the review team (JB). If the study met the inclusion criteria, it was included in the review. If there was any doubt about meeting the inclusion criteria a decision was made based on the consensus of the review team. Articles unrelated to the aim of the present study were excluded. The remaining titles were entered into an Excel spreadsheet and sorted. Duplicates were excluded. Next, the abstracts of the related titles were screened for their precise relevance to the aims of the present study. If an abstract met the inclusion criteria, it was included in the review. Abstracts that were not precisely relevant were excluded. The remaining papers were included in the review. The full texts of these articles were downloaded from the databases. If an article was not available free of charge, we paid for access. Two papers did not have full text that was accessible to us, and the study variables were not extractable from the abstracts, so they were excluded from the review.
The finally included papers were evaluated by a member of the review team (AA) using a data abstraction sheet developed by the research team. This data sheet included the study variables: name of the journal; name of the first author; number of authors; publication year; type of potential hazard; model/framework used for the CDS performance assessment; indices/criteria and tools used for the CDS performance assessment; the study approach; and the location of study. In the extraction of CDS performance assessment criteria, indices, and the study approach, our first priority was the authors’ statement. If criteria, indices, and the study approach were mentioned in the article, the data were included in our data abstraction sheet. If not, the review team used a consensus approach to decide whether the data should be included.
Ethical approval was not required for this literature review.
The initial search strategy resulted in a total of 3928 articles/documents (3902 resulted from database searching and 26 documents resulted from Grey literature and websites searching). Of these, 3698 titles did not fulfill the inclusion criteria and were excluded, leaving 230 articles/documents that were considered potentially relevant. These papers/documents were entered into an Excel spreadsheet and sorted alphabetically. Duplicates (93 titles) were discarded. In the second phase, the abstracts of the remaining articles/documents (137 titles) were examined. In this step, 114 irrelevant abstracts were excluded and 23 papers/documents were considered for analysis. Two articles full texts were not accessible and their abstracts were not informative enough thus excluded. Four potentially relevant documents were not accessible too. In total, 16 papers and one book were included in the final review list for data extraction. Figure 1 outlines the literature search and the study selection process.
Sixteen papers and one book were included in the final review list. A total of 97 authors contributed to these 17 papers/documents. The mean number of authors per article was 5.7 (SD = 3.07). The affiliations of the first/corresponding authors of articles were the USA (n = 6, 37.5%), France (n = 2, 12.5%), the UK (n = 2, 12.5%), Australia (n = 2, 12.5%), Brazil (n = 1, 6.2%), the Netherlands (n = 1, 6.2%), and Turkey (n = 1, 6.2%). The 16 papers were published in 13 different peer reviewed journals.
The earliest article was published in 2007 and one article was published in each of 2007 and 2008. There was an increase in the number of published studies of CDS system performance assessment from 2009. For example, in both 2010 and 2011, 3 articles were published, and in 2012, 5 articles were published. The 16 studies focused on 6 hazards types including epidemics (n = 4, 25%), hurricane/cyclone (n = 2, 12.5%), heat waves (n = 2, 12.5%), mass gatherings (n = 2, 12.5), complex emergencies (n = 1, 6.2%), flood (n = 1, 6.25%), 4 article (25%) and the book included all hazards.
The studies used a quantitative approach (n = 10, 62.5%), a qualitative approach (n = 1, 6.25%), or a mixed approach (n = 3, 18.7%). Two (12.5%) studies were reviews. Six studies (37.5%) were conducted in the USA, 2 (12.5%) in France, and 2 (12.5%) in Australia. A single study was conducted in 5 countries (Brazil, Chad, Poland, the UK, and Turkey). The location of 1 (6.2%) study was not determined.
The 16 articles that were finally selected for review were divided into 5 groups according to the theme of the study. These 5 themes were: performance assessment of syndromic surveillance systems (31.2%); mortality/morbidity SS (25.0%); public health/disease surveillance (12.5%); the applications of cost analysis, efficacy, effectiveness, and usefulness in performance assessment of SS (25.0%); and the review of performance assessment indicators (6.2%).
The relevant book is about the communicable diseases control in emergencies, and it has a specific section for CDS in disasters.
In the reviewed articles and book, there was no specific performance assessment framework for SS in disasters. However, the CDC updated guidelines for public health surveillance system evaluation
Of the CDC public health surveillance evaluation attributes, the most widely applied was timeliness, which was used in 7 studies. Flexibility was used in 5 studies, data quality in 4, simplicity in 3, stability in 3, and usefulness and representativeness in 2 studies. In all cases, timeliness, data quality, sensitivity, specificity, PPV, cost, and representativeness were calculated quantitatively. Flexibility, usefulness, simplicity, and acceptability were calculated in a qualitative manner. Stability was calculated both quantitatively and qualitatively.
In the reviewed book
Overall, in the 16 articles and one book that were included, 31 criteria/measures and 35 indicators were used or suggested for the assessment/evaluation of the performance of CDS systems in response to disasters and emergencies.
Josseran L, Fouillet A, Caillère N, Brun-Ney D, Ilef D, Brucker G, Medeiros H, Astagneau Pv
CDC's updated guidelines for surveillance system evaluation
Data quality, cost, flexibility, stability, timeliness, effectiveness, sensitivity, specificity, positive predictive value
Heat wave
Zielinski A
Review study
Cost minimization, cost-effectiveness analysis, cost utility, cost benefit
Heat wave
Cinti S, Haas K, Paliani P, Newton D, Wright C, Zalewski C, et al.
Comparing with data from regional and national surveillance reports
Percentage of visits by established SS and national surveillance, percentage of samples with positive results
Pandemics
Elliot AJ, Hughes HE, Hughes TC, Locker TE, Shannon T, Heyworth J, et al.
CDC's updated guidelines for surveillance system evaluation (Incomplete)
Sensitivity, specificity, timeliness, data quality
Mass gathering
Hope KG, Merritt TD, Durrheim DN, Massey PD, Kohlhagen JK, Todd K, D'Este, et al.
CDC's updated guidelines for surveillance system evaluation(Incomplete)
Usefulness, flexibility, acceptability
Mass gathering
Schnall AH, Wolkin AF, Noe R, Hausman LB, Wiersma P, Soetebier K, Cookson ST
Comparing of diagnosis in developed form with ED discharge diagnosis
Agreement between discharge diagnoses and developed form
Natural hazards
Choudhary E, Zane DF, Beasley C, Jones R, Rey A, Noe RS, Martin C, Wolkin AF, Bayleyegn TM
CDC's updated guidelines for surveillance system evaluation
Usefulness, simplicity, flexibility, data quality, acceptability, representativeness, timeliness, stability, sensitivity, positive predictive value
Hurricane
Teixeira MG, Costa MCN, Souza LPF, Nascimento EMR, Barreto ML, Barbosa N, et al
Comparing Brazil's public health SS with international health regulation
Structure (legal framework and financial, human and physical resource), surveillance procedure (capacity to detect, assess, notify), response (investigate, intervene, and communicate)
Public health emergencies (reemergence of infectious disease)
Noha H. Farag, Rey A, Noe R, Bayleyegn T, Wood AD, Zane D
CDC's updated guidelines for surveillance system evaluation
Simplicity, flexibility, acceptability, timeliness, stability, data quality, sensitivity, positive predictive value, representativeness
Hurricane
Bowden S. Braker K, Checchi F, Wong S.
Simplicity, flexibility, appropriateness, timeliness, dissemination of data
Complex emergencies
Potter MA, Sweeney P, Luliano AD, Allswede MP
Practice, process, outcomes (review ofpublished records)
Process (first clinical observation , accurate diagnosis, laboratory confirmation, identification of exposure source, report to public health authority, report to law enforcement authority, initiation of emergency operation plan , initation of risk-mitigation activities, initiation of past exposure prophylaxis, initiation of public health education activities, initiation of risk advice to health care workers, last reported new case); outcome indicators (primary cases, total cases, secondary cases, HCW[1]s infected)
Outbreaks
Josseran L, Caillere N, Brun-Ney D, Pottner J, Filleul L, Brucker G, et al.
Comparing frequency of visits due to heat related patients in 'on alert' periods (ONAP) with 'off alert' periods (OFAP)
Percentage of emergency department visits, percentage of hospitalization, incidence of heat related diseases
Heat wave
Stoto MA
‘‘Triangulation’’ approach
Validity, utility
Pandemics
Hope K, Merritt T, Eastwood K, Main K, Durrheim DN, Muscatello D, Todd K, Zheng W
Comparing traditional SS with Syndromic SS
Timeliness
Natural disasters
Rosenkötter N, Ziemann A, Riesgo LGC, Gillet JB, Vergeiner G, Krafft T, Brand H
CDC's updated guidelines for surveillance system evaluation(Incomplete)
Validity, timeliness, sensitivity, specificity
Pandemics
Akgun D
Case report
Water sanitation, immunization, organization of health services, public education
Flood
Conolly MA (Editor).
Zero reporting, completeness, timeliness, the number of cholera cases for which samples were confirmed by the laboratory, the number of malaria cases confirmed by blood smear date of onset of the first case, date of reporting using outbreak alert form, date of investigation, date of response
All hazards
The aims of this research were to review and extract the frameworks and indices that have been used for assessing the performance of CDS systems in response to disasters and emergencies. An extensive systematic literature review was conducted using 5 popular international databases. Our initial search resulted in 3902 titles. Finally, 17 studies/documents met the study inclusion criteria and were considered in the data abstraction list.
The first disaster-related article was published in 1945, but the first article related to the performance assessment of CDS in disasters was not published until 2007. Surprisingly, there was a 62-year gap between the 2 publications.
Searching for the term “disaster” in PubMed resulted in more than 60,261 papers, but we could only find 16 papers about CDS systems performance assessment in response to disasters and emergencies. Therefore, performance assessment in this field is a new research focus, although in recent years there has been a growing interest in CDS system performance assessment in response to disasters. With a recent increase in the number of disasters, many countries have spent millions of dollars improving the response of their health care systems. Consequently, the assessment of the efficacy and effectiveness of the response to disasters has become a focus of investigation, to ensure that resources are used in the most efficient and effective way
According to current management knowledge, lack of an appropriate performance assessment system is a symptom of weakness and disorder in an organization or program
Furthermore, some economic assessment criteria have been suggested for use in the performance assessment of SS in disasters, but these criteria have not been used in practice.
The selection of appropriate criteria and indexes for CDS assessment is essential to meaningful research. In the included studies, 31 criteria/indices were used and the attributes suggested by CDC were the most widely used criteria. Some studies used other criteria and indices in addition to those of CDC
Different definitions have been proposed for the CDC criteria, and different scientific methods (qualitative and quantitative) have been applied for measurements. For example, Josseran divided the timeliness criterion into 4 steps and defined it as “the time that it takes for data to be collected, processed, analyzed, and publicized”
Despite the reasonably wide spread use of the CDC guidelines for systems performance assessment in response to disasters and emergencies, experts have emphasized that there are no generally accepted metrics
This review has some limitations. During the study period, the Web of Science (ISI) was not available, and was excluded from the search process. Studies were only included if the texts or abstracts were available in English. Therefore, there is a bias in the selection of studies. However, high sensitivity was used in the database search, yielding more than 3928 titles.
Another limitation was the identification of criteria and indices. We addressed this problem by using a consensus approach between reviewers. However, there were some errors in the determination of criteria and indicators.
Finally, there was limited access to the full text of some papers and four potentially relevant documents were not accessible. We were unable to extract the study variables from 2 articles and those results were excluded from the study.
Performance assessment is an integral component of the management of all organizations and the lack of performance assessment is considered a significant sign of weakness in an organization. Therefore, the lack of accepted mechanisms for the assessment of the performance of CDS systems in response to disasters is an important weakness. Although some studies attempted to assess surveillance systems, the results of this systematic literature review suggest that there is no clear and comprehensive assessment framework in place. While the CDC framework has been used in some studies, it is not specific for SS in response to disasters. Some criteria and indices were widely applied, but were not universally accepted, and researchers used different definitions and scientific approaches. Further research is required to identify and develop a universally accepted framework for the assessment of CDS systems in response to disasters and emergencies.
Ali Ardalan, MD, PhD. Department of Disaster Public Health, School of Public Health, Tehran University of Medical Science, Tehran, Iran.
Email: aardalan@tums.ac.ir
This study is part of a PhD thesis supported by Tehran University of Medical Science.