Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Public Health Surveillance Systems: Recent Advances in Their Use and Evaluation

Public Health Surveillance Systems: Recent Advances in Their Use and Evaluation Surveillance is critical for improving population health. Public health surveillance systems generate information that drives action, and the data must be of sufficient quality and with a resolution and timeliness that matches objectives. In the context of scientific advances in public health surveillance, changing health care and public health environments, and rapidly evolving technologies, the aim of this article is to review public health surveillance systems. We consider their current use to increase the efficiency and effectiveness of the public health system, the role of system stakeholders, the analysis and interpretation of surveillance data, approaches to system monitoring and evaluation, and opportunities for future advances in terms of increased scientific rigor, outcomes-focused research, and health informatics. INTRODUCTION Public health surveillance, and by extension the systems used to enable surveillance, is central to the practice of modern public health. Public health surveillance contributes data and information to assess and characterize the burden and distribution of adverse health events, prioritize public health actions, monitor the impact of control measures, and identify emerging health conditions that may have a significant impact upon population health. The core role of surveillance systems within public health practice, and their concomitant capacity to greatly influence the efficiency and effectiveness of the public health system, has stimulated research to strengthen the scientific basis of public health surveillance. In 1970, only 7% of PubMed articles about surveillance (20/277) focused on methods, but that proportion rose to 60% by 2015 (7,400/12,400). In the context of scientific advances in public health surveillance, changing health care and public health environments, and rapidly evolving technologies, the aim of this article is to review public health surveillance systems, including their current role, recent advances, and opportunities for future advances. This review is divided into three sections. In the first two sections, we review the contemporary use and evaluation of surveillance systems in public health practice and highlight some notable recent advances in their use and evaluation through case studies and other examples. In the third section, we discuss some promising opportunities for advancing surveillance systems in the future and highlight notable research activities. THE USE OF SURVEILLANCE SYSTEMS The Concept of a Surveillance System Surveillance, a core function of public health practice, is defined as “the ongoing, systematic collection, analysis, and interpretation of health data essential to the planning, implementation, and evaluation of public health practice, closely integrated with the timely dissemination of [this information] to those who need to know” and act upon that information (84, p. 164). A surveillance system, in turn, is a collection of processes and components that enable public health practitioners to conduct surveillance. Surveillance processes include data collection, data quality monitoring, data management, data analysis, interpretation of analytical results, information dissemination, and application of the information to public health programs. The enabling components of surveillance systems may include laboratory diagnostics to detect or confirm health conditions; information technologies to support the surveillance processes of data collection, analysis, and dissemination; clinician consultation and reporting; clinician, public health, and laboratory worker education and training; legislation, regulations, and policies that support the conduct of surveillance; systems and directories for disseminating alerts, bulletins, clinical guidelines, and prevention recommendations; program administration and management; and human factors (e.g., multisector communications and relationships) (54). Ultimately, public health surveillance systems should produce information to guide public health decisions in many areas, including disease prevention, prevention program planning and management, health promotion, quality improvement, and resource allocation. Using Surveillance Systems to Meet Public Health Objectives From a societal perspective, public health surveillance systems should increase the efficiency and effectiveness of the public health system, which is a primary determinant of population health. A surveillance system affects population health by capturing data and generating information that 58 Groseclose · Buckeridge public health practitioners and stakeholders can use to improve the quality of their decisions and the effectiveness of their actions. From the perspective of the public health system, surveillance systems support all three essential functions of public health—assurance, assessment, and policy development (39). For a specific surveillance system, however, the objectives should be defined more precisely (Table 1) (see also Supplemental Case Studies 1, 2, and 3; follow the Supplemental Material link in the online version of this article or at http://www.annualreviews.org/). These objectives should be tailored to the outcomes under surveillance, the intended uses of the information generated by the system, and the level (e.g., local, regional, national) of the public health system at which the surveillance system is functioning. The processes and components of a public health surveillance system should in turn be aligned with the objectives of the surveillance system to ensure that valid information can be derived and applied to practice, to promote operational efficiency, and to ensure that the activities are within the legal or regulatory mandate of the public health system. Explicitly documented objectives for the surveillance system are also important for planning the system, evaluating system performance, and enabling continuous improvement of data and system quality. Surveillance objectives and budget should determine the number and type of data variables to be collected (e.g., demographic or behavioral data variables), including the required level of resolution of the data, the population under surveillance, the required timeliness of information for effective action or response, the frequency of data analysis and interpretation, and the resources required to support the surveillance system. Similarly, surveillance objectives should influence decisions about data collection, management, analysis, integration, dissemination, security, and privacy. It is best to identify inconsistencies between the objectives (and their implications for system design and performance) and resources at the planning stage so that one or the other can be adjusted accordingly. Ultimately, a public health surveillance system’s objectives indicate how the data are intended to be used for public health action. Over 40 years ago, Dr. William Foege wrote, The reason for collecting, analyzing and disseminating information on a disease is to control that disease. Collection and analysis should not be allowed to consume resources if action does not follow. Appropriate action, therefore, becomes the ultimate goal and the final assessment of the earlier steps of a surveillance system. (28, p. 30) Supplemental Material Surveillance data have been used to guide a range of public health actions. Table 1 presents examples of surveillance data use by surveillance system objective to illustrate the range of public health actions that can be informed by public health surveillance data. The Expanding Use of Surveillance Systems The initial focus of public health surveillance principles and practices was on infectious diseases, but today public health surveillance systems are used to monitor and forecast a broad range of health determinants (e.g., risk behaviors, health care services, and socioeconomic factors) and outcomes relevant to infectious diseases, injuries, chronic diseases, mental health, and occupational and environmental health. The case studies on the Major League Baseball Health and Injury Tracking System (MLB HITS) and active transportation surveillance are examples of surveillance systems focused on noninfectious diseases (see Supplemental Case Studies 1 and 3). In particular, in the case of the MLB HITS, two non–public health entities, an industry (MLB) and its worker association, established a surveillance system to identify and monitor work-related injuries and associated risk factors and to assure the effectiveness of the intervention (64). www.annualreviews.org • Public Health Surveillance Systems 59 Table 1 Surveillance objectives and examples of surveillance system data use by surveillance objective Surveillance system Acute flaccid paralysis (AFP) surveillance (24, 81; see Supplemental Case Study 2) Surveillance method Monitoring of AFP (a clinical syndrome with several potential etiologies, including poliovirus) as reportable condition in countries not reporting confirmed polio cases Etiology of AFP cases is determined via testing of stool specimens Collection of child height and weight data, typically in a school-based setting at least every two years Example of surveillance data use Determine extent and duration of wild poliovirus circulation; identify causes of outbreak or reason for AFP clustering; initiate control measures (immunization, enhanced surveillance) to interrupt transmission, prevent spread, or improve detection capability Surveillance objective Guide immediate action for cases of public health importance (e.g., initiating investigations or interventions) Measure the burden of a disease (or other health-related event) and monitor trends over time, including changes in incidence and the identification of high-risk populations Support early detection and response to outbreaks or new or emerging health concerns State-based child body mass index (BMI) surveillance systems (10) Detect disparities in prevalence of overweight and obesity based on socioeconomic status and race/ethnicity; identify the determinants of local variation in obesity prevalence; evaluate local public health interventions Gonococcal Isolate Surveillance Project (45) Sentinel surveillance system to monitor trends in antimicrobial susceptibilities of Neisseria gonorrhoeae strains among men with gonococcal urethritis attending 1 of 27 sexually transmitted disease clinics in the United States Monitoring of three million people who received H1N1 vaccine by linking data from private health plans and public immunization registries; analysis of data to identify time-invariant confounders (e.g., chronic illness) and rare outcomes (comparison of current and historical data) Guide development of national treatment recommendations for effective therapy and prevention of gonorrhea; set research and prevention priorities; guide the planning and allocation of STD prevention services and resources; inform clinical practice Guide the planning, implementation, and evaluation of programs to prevent and control disease, injury, disability, or exposure to environmental hazards Post-licensure Rapid Immunization Safety Monitoring (PRISM) program, the immunization safety monitoring component of US Food and Drug Administration’s Mini-Sentinel project (5, 72) Identify adverse health events potentially associated with H1N1 vaccination to increase our understanding of vaccine safety and to inform immunization policy (Continued ) Supplemental Material Groseclose · Buckeridge Table 1 (Continued ) Surveillance system Syndromic surveillance system monitoring over-the-counter (OTC) pharmacy sales (61) Surveillance method Monitoring of OTC antidiarrheal medication sales to identify unusual sales patterns; monitoring of pharmacy sales promotions for these products to aid interpretation of sales data Example of surveillance data use Investigate citywide syndromic surveillance signals identified in sales of OTC antidiarrheal medications using multiple communicable disease surveillance systems data [emergency department visits with chief complaint of diarrhea, school nurse visits for stomach ache or diarrhea, enteric reportable diseases, clinical laboratory stool specimen submissions, water quality indicators and complaints, and social media mentions of key words (e.g., diarrhea, loperamide, water quality)]. Findings possibly reflected sales promotions but not increased diarrheal illness Guide vaccine development; determine vaccine effectiveness; formulate immunization policy Surveillance objective Provide reassurance during periods of perceived increased risk that incidence of a health condition is not increasing Evaluate public policy Active Bacterial Core Surveillance System, a component of CDC Emerging Infections Program (48, 63) Swedish Strategic Programme Against Antibiotic Resistance (77) Population-based, active surveillance network in ten geographically and racially diverse jurisdictions comprising up to 12% of the US population Repeated national point-prevalence surveys of antimicrobial use in Swedish hospitals [monitored treatments for predefined diagnostic groups, defined daily dose (a measure of antibiotic pressure), and reason for antimicrobial use (i.e., prophylaxis, community-acquired infection, hospital-acquired infection)] Annual national household survey of parents of adolescents (13–17 years of age) and provider verification of vaccination Detect changes in health practices and the effects of these changes Note increasing compliance with treatment guidelines for lower urinary tract infections for women and for community-acquired pneumonia. Compliance with surgical prophylaxis guidelines did not improve over the eight-year period. Main intervention was to increase appropriate antimicrobial use through use of an audit feedback approach using local survey data Prioritize the allocation of health resources National Immunization Survey (NIS)—Teen (6) Estimate measles susceptibility, proportion of children vaccinated at each age, and number of adolescents in each state who failed to receive the first dose of measles vaccine; reemphasize the need for high measles vaccination coverage to support population-level immunity and prevent reestablishment of indigenous measles transmission in the United States (Continued ) www.annualreviews.org • Public Health Surveillance Systems Table 1 (Continued ) Surveillance system Surveillance systems and other data sources to determine the impact of rotavirus vaccination in Belgium (70) Surveillance method Eight surveillance systems or data sources, including lab-based surveillance (numbers of tests performed and positive results), hospital discharge surveillance (admissions for rotavirus and all-cause gastroenteritis), mortality due to rotavirus and all-cause gastroenteritis, health services data (rotavirus vaccination coverage) Voluntary enrollment of baseball players into medical record and injury tracking system to monitor injury, medical, treatment, and prevention data Example of surveillance data use Describe changes in timing and age of rotavirus infection (postvaccination annual incidence peak shifted to spring instead of winter; average age at infection and hospitalization increased); obtain evidence for herd immunity (number of lab-confirmed and hospitalized rotavirus cases decreased in both vaccinated and unvaccinated persons); determine changes in rotavirus testing before and after vaccination introduction; estimate vaccination coverage Surveillance objective Describe the clinical course of disease Provide a basis for epidemiologic research Major League Baseball’s (MLB) Health and Injury Tracking System (HITS) (64; see Supplemental Case Study 1) Identify most frequent types of injury; initiate research projects to investigate injury patterns and identify potential interventions (including rules change) Supplemental Material Beyond the monitoring of individual risk factors and outcomes, surveillance systems have also been developed recently to monitor the presence, emergence, or evolution of infectious agents in the environment. For example, a recent surveillance evaluation sought to determine if bioaerosol sampling methods are suitable for routine surveillance for viruses in environments that facilitate aerosolization (3). Investigators compared bioaerosol sampling in swine farms to concurrent animal, environmental, and human sample testing to detect novel influenza viruses. These data supported the identification of novel influenza viruses and of climatic factors and animal husbandry practices that may increase the risk of human exposure to aerosolized influenza A in agricultural settings. The use of surveillance systems has also expanded in relation to communicable diseases, for example to monitor the impact of vaccination programs on viral evolution in order to inform vaccine design and maintain vaccine effectiveness. One such example is the New Vaccine Surveillance Network, which monitors the etiology of acute gastroenteritis in the United States and has documented the decline in Group A rotavirus–associated gastroenteritis incidence among children following reinitiation of rotavirus immunization in the country (11). The Network continues to monitor the genotypic variation in the rotavirus strains circulating in the United States and has noted a major shift in the predominant genotype prevalence in the years 2008–2013. Data from surveillance systems are also used increasingly to forecast or predict future trends in disease distribution. Early identification of the timing and intensity of the annual seasonal epidemic of influenza is useful to inform public health planning and response. To determine if reasonably accurate forecasts can be made during flu season, the Centers for Disease Control and Prevention Groseclose · Buckeridge (CDC) organized a challenge to predict the 2013–2014 United States influenza season (8). Findings indicated that flu forecasting can yield reasonably accurate estimates of the start, peak, and intensity of the influenza season. Although forecasting may be technically feasible, more work is needed to improve its accuracy so that policy makers can use these predictions to guide prevention and control efforts. Many public health agencies have also chosen to actively monitor health communications and news media—especially during a public health emergency—both to refine the public health response and to inform decisions about the creation, alteration, or refinement of health or risk communication messages (67). Recognition that public health law and policy influence community health has led some organizations to establish surveillance systems to monitor the impact of public health laws and policies on communities, environments, and individuals (18, 23, 66). Longitudinal surveillance data used to examine the impact of laws and policies on community health are distinguished from the legislative tracking systems that are commonly utilized to monitor policy interest and activity. Law and policy tracking over time and across jurisdictions can help policy makers, advocates, and researchers understand what the laws are on a particular topic and can provide them with data to evaluate their impact (66). This increasing breadth of application of public health surveillance systems raises many issues, including the engagement of a broader set of stakeholders in surveillance activities and the need to adapt existing analytical methods to new objectives and data sources. In the following two sections, we consider these two issues. The Importance of Stakeholder Engagement in Surveillance Systems Public health organizations often conduct surveillance under a legal mandate, but to maintain the public’s trust, they must conduct surveillance in a manner that is responsible and of sufficient quality to inform population health improvement. The substantial public and private sector investment in surveillance also demands that the data collected by surveillance systems be used and that stakeholders perceive the system to be useful (21). Stakeholders may contribute data or resources to a surveillance effort, act upon the information generated, or use surveillance information to advocate for prevention and control efforts to improve population health. Typical stakeholders include public health practitioners, health care providers, policy makers, and members of affected communities, academia, professional associations, private industry, and not-for-profit advocacy organizations. Engaging with a variety of system stakeholders during system planning, design, implementation, and evaluation encourages broad-based ownership of the surveillance activity and allows the economic, social, and cultural factors that influence prevention and control to be identified and addressed, increasing the likelihood that the information generated by the system will be useful. For surveillance systems to achieve their greatest impact, it is necessary to identify system stakeholders, understand their roles (e.g., contributing data or advocating for assistance to those affected by the health outcome under surveillance), and engage them throughout the surveillance process (21). Prior to the initiation of surveillance, the stakeholder input, such as the information sought for decision making, may inform the definition of system objectives. When the surveillance system is active, stakeholders can help interpret the reported data based on their knowledge of the health outcome or the environment in which the data are collected. They may also respond to the information generated by the surveillance system and recommend and influence surveillance system evaluation to ensure the system is meeting its objectives. Increasingly, the general public can be considered a partner in the surveillance process, as novel data streams “whose content is initiated directly by the user (patient) themselves” are investigated www.annualreviews.org • Public Health Surveillance Systems 63 and integrated into traditional public health surveillance (2, p. 1). Widespread availability of these novel data streams has led to the investigation of new surveillance methods—e.g., natural language processing and monitoring of Twitter posts to augment traditional influenza surveillance (12) or prediction of dengue incidence using search query surveillance (1)—and new surveillance data sources, such as restaurant reservation and review logs (35, 58) or over-the-counter pharmacy sales (62). These novel methods and data sources require validation (e.g., comparison to existing surveillance data) and prospective evaluation to determine if they add value to established surveillance systems. As Salathe et al. (71, p. 403) noted, “the dynamics of information spread are inherently different from the dynamics of disease spread” and the influence of changing user behavior on participatory surveillance and digital health surveillance must be further evaluated. Analysis and Interpretation of Data in Surveillance Systems The analysis of public health surveillance data is typically performed to detect the presence (or absence) of signals (i.e., unusual patterns in the reported data by person, place, and time compared to historical data) that may prompt a public health investigation or other actions. Surveillance data analysis may also be performed to identify the association (or lack thereof) between reported health outcomes and patient or population characteristics to understand the local epidemiology of the condition under surveillance and to determine effective prevention or control measures. Because they are collected and monitored over time, surveillance data can support the detection of trends in reported data by person, place, and time and the identification of changes in disease incidence compared to historical data. Surveillance data analysis and interpretation should directly support the surveillance system objectives and be performed in alignment with surveillance system processes. The content and structure of a valid surveillance system should be defined and monitored to ensure that quality data are received and available for analysis. For example, surveillance data quality review should be initiated as data arrive and conducted at each data update. Similarly, standard analytic routines should be specified prior to receipt of surveillance data and performed at each major data update. In addition, it is useful to anticipate the types and magnitude of signals or surveillance measure values that may be generated by the surveillance system, both to evaluate the system (e.g., by simulating expected signals and determining the capacity of the system to detect those signals) and to define and implement public health action(s) in response to the population health metrics generated by the analysis of the data (14). Ad hoc analyses may be conducted in response to novel questions or stakeholder requests. The available data, together with the system objectives, help to define the analysis options. Traditionally, researchers have used straightforward methods due to limitations in the data (e.g., quantity, resolution, volume) and to a lack of evidence supporting the use of more sophisticated methods. Over the last three decades, the amount of data available has expanded greatly, computing power and storage capacity have increased exponentially, and surveillance objectives have evolved to address a wider set of considerations and more immediate threats. All these factors have pushed innovation in the development and evaluation of analytical methods in surveillance. In particular, the objective of rapid outbreak detection has prompted the examination of a wide range of detection methods and the comparison of these methods against one another using different types of data and outbreaks. This work has contributed some qualitative insights, including the observation that no single detection approach is likely to be optimal for all surveillance applications, and that it may be possible to identify methods that are better suited to one particular application (14). Another recent trend has been toward the integration of data from multiple sources into a single analysis. Whereas traditional surveillance analyses have tended to 64 Groseclose · Buckeridge focus on a single time series of case counts or rates, the increasing availability of data from clinical systems and other sources has provided the opportunity to consider additional covariates within each record (74, 88) and to integrate data on risk factors and multiple outcomes into a single analysis (56). Another strategy for integrating data from multiple sources has been to develop decision rules or policies based on the pattern of alerts observed across data streams (78). Data integration also poses a challenge in chronic disease surveillance, where, given the multifactorial causation and complex progression of most chronic conditions, surveillance systems tend to follow multiple indicators concurrently. One strategy that is being explored to support the effective use of such information draws on the concept of evidence-based public health (13) and presents indicators in the context of existing evidence to help users appreciate patterns in the data and identify actions that are likely to improve population health (75). MONITORING AND EVALUATION OF SURVEILLANCE SYSTEMS Performance monitoring and the evaluation of surveillance systems are complementary processes conducted by public health practitioners and other surveillance system stakeholders. Surveillance system performance is monitored through the ongoing assessment of surveillance processes, such as by following the completeness and timeliness of data collection and reporting. Evaluation focuses on whether the system is meeting its objectives and making effective use of its resources. Both monitoring and evaluation can identify opportunities for surveillance system performance improvement, and both provide information on performance to aid the interpretation of data. Historically, greater attention has been given to evaluation, but monitoring is of increasing interest given the growth in automation and data-intensive systems. Routine Monitoring of Public Health Surveillance Systems The monitoring of a surveillance system can be defined as the routine process of data collection and the measurement of surveillance program or process changes over time using previously agreed-upon plans, schedules, and indicators (27). Discrepancies between actual and planned system performance are identified via routine monitoring, and corrective actions are taken— akin to quality improvement approaches intended to systematically improve the performance of any system. A routine monitoring perspective recognizes that surveillance system performance indicators are not static: They are often reformulated based on evaluation findings (31). Similarly, surveillance monitoring data might indicate where more comprehensive evaluation is needed. Surveillance system performance indicators are defined to specify some aspect of, or a change in, a process, outcome, or activity and to identify sources of variation that, if not addressed, may cause changes in the quality of the information generated by the system or in the system’s performance. Well-defined surveillance indicators are specific in nature, measureable, relevant for the setting in which the system operates, and obtainable in a timely manner (27). Indicators may be of different types, e.g., they may focus on process, outcome, or output. Indicators should have a defined target and be operationalized using a protocol or algorithm that indicates thresholds for alerting or response. It is useful to define the recommended actions to take in response to the indicator data and accompanying protocols to allow the identification of and response to the causes of missed indicator targets. Routine monitoring of surveillance system indicators yields information to assess the current performance of the surveillance system and provides early warning of potential system deviations. Analysis of performance monitoring data allows public health organizations to set goals for system www.annualreviews.org • Public Health Surveillance Systems 65 Supplemental Material performance improvement. Perhaps the most important use of surveillance monitoring data is to aid the interpretation of the data generated by the surveillance system. Surveillance indicators can also be used to set priorities for the surveillance system [e.g., to identify the attributes (timeliness, representativeness, sensitivity, etc.) of greatest importance for the health outcomes under surveillance and for the objectives of the system] or to identify actions that should be undertaken to enhance quality, performance, or impact. Examples of surveillance system performance indicators are provided in the polio surveillance case study (see Supplemental Case Study 2) and in Table 2. Routine monitoring of surveillance systems’ performance is increasingly recognized as an essential surveillance process, particularly for highly automated systems that receive data in real time. Evaluation of Public Health Surveillance Systems A number of factors may indicate that it is time to formally evaluate a surveillance system using a specific study design to assess key surveillance attributes [e.g., sensitivity, positive predictive value, and representativeness (Table 3)] that influence the relevance, effectiveness, and impact of the surveillance system. For example, routine monitoring of surveillance system performance indicators may identify trends that should be explored more thoroughly to identify improvements that can enable the system to better meet its objectives and achieve its desired performance. Unexpected performance, such as a community gastrointestinal disease outbreak that is missed by a foodborne disease surveillance system, may stimulate an evaluation. Discrepancies in the information generated by different surveillance systems may also cause health department staff to initiate an evaluation to determine the reason for the discordance. Evaluation allows the system’s sponsor and stakeholders to determine if the system is generating data that can be used to address its objectives; to make effective use of fiscal, personnel, and technology resources; to meet the needs of stakeholders; and to maintain the system’s public health relevance given changing health care practices, hazards, disease epidemiology, or information needs. Table 3 summarizes the definition of the attributes used to evaluate a surveillance system and the methods employed to measure them. Some attributes are more important for selected surveillance purposes and health outcomes than others (37). The reason for this may be obvious when one considers concurrently the objectives and attributes of a system. For example, rapid outbreak detection is often a stated objective for systems monitoring communicable diseases: Timely and representative population-based health-related data are required to avoid missing outbreaks of public health importance and to ensure rapid public health action. Therefore, surveillance system evaluation is typically driven by the assessment of specific attributes (Table 3); however, if such an evaluation is pursued in a reductionist manner (i.e., by assessing system attributes independent of one another), key insights into the surveillance system process and information may be missed. In other words, the attributes of public health surveillance systems are related to one another; strengthening one system attribute could adversely affect another attribute of a higher priority. For example, data quality, although critical, need only be good enough to meet the objectives of the system. The Council of State and Territorial Epidemiologists recommends that “because no data are perfect and perfecting data can be costly, matching data quality to its use is imperative” (79, p. 238). Data of poor quality make the system less acceptable and potentially less representative of the population under surveillance: Efforts to improve sensitivity, positive predictive value, representativeness, timeliness, and stability can increase the cost of a surveillance system, although savings in efficiency through use of information 66 Groseclose · Buckeridge Table 2 Examples of performance monitoring indicators used by surveillance systems Surveillance objective Measure prevalence of persons with HIV infection and monitor over time Monitoring indicator Completeness of reporting of persons with HIV infection Indicator threshold ≥85% of the expected number of cases for a diagnosis year reported within 12 months of the diagnosis year Method of calculation Jurisdictions use a three-source capture-recapture model to estimate completeness of reporting for persons newly diagnosed with HIV; data sources include health care provider, laboratory, and a third source such as viral hepatitis surveillance data. Jurisdictions link HIV infection case reports with one or more of the following: Vital Statistics Data, Social Security Death Master File, and National Death Index. Intrastate: calculated monthly. Selected case report variables are matched with jurisdiction’s database, followed by manual review and resolution of potential matches. Interstate: calculated semiannually. Selected case report variables are matched across jurisdictions at the national level, followed by manual review and resolution of potential matches. Number of rubella cases reported to NNDSS with complete information for selected variables is compared with total number of rubella cases reported to NNDSS. Indicator type Outcome Surveillance system National Human Immunodeficiency Virus (HIV) Surveillance System, United States (32, 33, 42) Monitor HIV infection case fatality rate and trends Death ascertainment Linkage of HIV case reports to mortality data sources at least annually Process Accurately monitor HIV infection case counts at jurisdictional and national levels Intrastate and interstate duplicate cases ≤5% duplicate cases assessed at 12 months after the close of the report year Process Vaccine-preventable diseases (e.g., rubella) reported to the National Notifiable Diseases Surveillance System (NNDSS), United States (69) Accurately monitor burden of rubella infection and case characteristics Rubella case report data quality Proportion of rubella cases reported to NNDSS with complete information for selected variables (e.g., clinical and epidemiologic case characteristics, lab test results, vaccination history) Outcome (Continued ) www.annualreviews.org • Public Health Surveillance Systems 67 Table 2 (Continued ) Surveillance objective Detect imported rubella cases Monitoring indicator Surveillance system sensitivity to detect rubella cases (because endemic transmission of rubella has been eliminated in the United States) Number of submissions of laboratory requests for rubella diagnostic testing Indicator threshold Proportion of rubella cases having an imported source Method of calculation Number of rubella cases reported to NNDSS that have an imported source is compared with total number of rubella cases reported to NNDSS. Indicator type Outcome Surveillance system Monitor rubella diagnostic efforts Number of submissions of laboratory requests for IgM antibody tests for rubella Electronic lab reports of orders for rubella IgM antibody testing are analyzed per time period Rationale: “If testing occurs, the diagnosis is being considered, so the absence of reported cases is more likely to reflect the absence of disease. Without an external standard, how much testing is ‘enough’ is still open to question” (69). Outcome technology (e.g., electronic reporting) might offset some of these costs. As sensitivity and positive predictive value approach 100%, a system is more likely to be representative of the population with the event under surveillance. However, as sensitivity increases, predictive value positive might decrease. (31, p. 182) A recent systematic review of 99 articles examining animal and human health surveillance system evaluations identified 23 different attributes of surveillance systems (26). Almost one-half of the evaluations assessed only one or two attributes; the attributes assessed most frequently were sensitivity, timeliness, and data quality. The investigators noted with some concern that surveillance objectives for the system being evaluated were often not stated, and therefore the rationale for assessing the selected attributes was not always evident. Advances in Surveillance System Evaluation Investigators continue to pursue novel methods to evaluate and improve the ability of surveillance systems to support new uses of data. Historically, cancer surveillance systems did not focus on improving timeliness because immediate uses of the data for the affected patient or community were limited. However, with improvements in cancer diagnosis and interventions, there is renewed interest in increasing the timeliness of cancer surveillance systems so the data can be used to expedite the enrollment of eligible patients in clinical trials, to identify the inception of individual treatments or population-based interventions, and to monitor the effectiveness of the 68 Groseclose · Buckeridge Table 3 Attribute Public health surveillance system attributes and their measurementa Definition Willingness of persons and organizations to participate in the surveillance system. Measurement considerations May be indicated by the participation rate of data providers, completeness of case report data, timeliness or frequency of data reporting, or responsiveness to requests for supplemental information on reported health events. Acceptability can be influenced by the time and effort required to complete and submit surveillance reports or by perceptions of the benefits derived from participation in the surveillance system. Assessment of surveillance resources typically focuses on direct costs. Because of the complexity of surveillance and response processes, it is usually difficult to define indirect costs. For some infectious disease surveillance systems, investigators have modeled the expected future costs of strategies for continued vaccination, surveillance, and other public health activities (82, 83). Can be measured as the proportion of data intended to be collected that was actually collected (completeness) and the proportion of data entries that correctly reflect the true value of the data collected (validity). Includes proportion of unknown, invalid, and missing values for reported data elements. Validity may be estimated by the proportion of errors in surveillance system data compared to analogous data from one of the system’s data sources. Is best evaluated retrospectively by observing how the system has responded to new requirements or changes in surveillance processes or environment (e.g., new health-related events, case definition changes, modification of policies affecting the patient population eligible to receive care from participating reporting sources, or introduction of new information technologies). Cases reported to the system must be investigated to determine if they represent true or confirmed cases of the health event under surveillance. Low PVP may be addressed by case definition revision, request of additional data, or staff training to increase reporting accuracy. For event-based surveillance, criteria denoting a real outbreak should be defined and the outcomes of investigations of potential outbreaks identified by the system should be monitored and characterized per outbreak definition criteria. (Continued ) Acceptability Cost-effectiveness Relationship between the expected outcomes and the costs of surveillance to achieve these outcomes. Surveillance system costs include direct costs (personnel and material resources), indirect costs (resulting from preparedness and response to surveillance findings), and prevention benefits or costs from a societal perspective (e.g., effects of the information generated on decision making and population health). Completeness and validity of data in the surveillance system. Data quality Flexibility Ability of the surveillance system or its processes to adapt with little additional time or resources to changing epidemiologies, information needs, technologies, or clinical practices. Predictive value positive (PVP) Proportion of cases reported to the system that actually have the health condition under surveillance. For event-based surveillance, PVP represents the probability that a detected outbreak is of public health significance and requires response. www.annualreviews.org • Public Health Surveillance Systems Table 3 Attribute (Continued ) Definition Ability of the system to accurately describe the occurrence of a health condition under surveillance over time and its distribution in the population by place and person. Measurement considerations Representativeness is assessed by comparing the reported cases or events to all actual cases/events. Information on all cases/events is seldom available, but surveillance data representativeness can be described in terms of geographic coverage, demographic distribution, and clinical manifestation of the health conditions under surveillance that were reported to the surveillance system. For example, is there uneven distribution of reported cases/events based on our understanding of the epidemiology of the health condition in the population under surveillance? Surveillance system security policies and practices should be reviewed to ensure that security levels and procedures for surveillance system data or system access are defined and enforced; data use and release policy and protocol is available; and access to the surveillance system software application is controlled. Case detection sensitivity can be measured by the number of cases reported to the surveillance system divided by the number of cases in the population under surveillance. Outbreak detection sensitivity may be estimated retrospectively (e.g., were outbreak cases reported to the system? Was the temporal or spatial association of the cases noted prior to outbreak recognition from some other data source?) or prospectively (e.g., is an observed increase in case reports indicative of an outbreak requiring response?). Simulated or authentic data may be used to determine the characteristics of outbreaks that can be identified by the system (e.g., some absolute number of cases in time and space) or statistically derived thresholds (e.g., based on standard deviations). Sensitivity of the case definition depends on (a) the health condition under surveillance; (b) our knowledge of the epidemiology of the condition; (c) our ability to describe the condition based on clinical signs and symptoms, laboratory criteria, or epidemiological criteria; and (d ) the availability of relevant data elements in the surveillance system’s data sources. May be characterized in terms of (a) availability, amount, and type of data elements needed to characterize health events under surveillance; (b) number and type of organizations providing and using the data monitored; (c) number and type of data interchanges and transformations within the system; (d ) data provider and system operator training requirements; and (e) type of information technologies used by the system. (Continued ) Representativeness Security Processes and methodologies to keep surveillance data and information confidential, available, and accurate. Sensitivity Sensitivity of a surveillance system to (a) case detection (proportion of individuals who have the condition of interest), (b) outbreak detection (probability that the surveillance system will detect a significant increase in a health condition in time or space), and (c) case definition (ability of the case definition criteria to accurately represent the health condition of interest and classify the cases to which it is applied). Simplicity Structure and ease of operation of the system across the surveillance process cycle from data collection to use. Systems should be as simple as possible while still meeting their objectives. Groseclose · Buckeridge Table 3 Attribute Stability (Continued ) Definition Ability to collect, manage, and provide data without failure (reliability) and to be operational when needed (availability). Measurement considerations Measures for determining stability can include the number or duration of unscheduled outages of the information system(s) supporting the surveillance system; the comparison of the desired and actual amount of time or resources required for the system to collect, manage, analyze, interpret, and release data from the system; or the presence or absence of continuity of operations procedures intended to maintain system performance. Determine whether the system uses data standards (e.g., ICD, LOINC, or SNOMED)b as valid values for appropriate data variables or has the ability to translate its variable values to data standard concepts; determine whether the system uses standard data interchange protocols (e.g., Health Level 7) to exchange data with other information systems. For systems aiming for early detection of events of public health concern, timeliness assessment should focus on the detection of the hazard or agent causing the health condition or the identification of symptomatic individuals when they first seek care. For other systems, timeliness measures may indicate time to initiate interventions based on information derived from the system or time to accumulate sufficient information on which to develop risk communications or clinical guidance. Timeliness is influenced by surveillance methods and data source(s). Standards use Use of data exchange, messaging, or other information technology standards by a surveillance system that enhances the ability of the system and its software applications to communicate, exchange data, and use the information that has been exchanged. Time between any two steps in the surveillance process. Steps in the surveillance process will vary by system. The relative importance of timeliness of surveillance process intervals varies by surveillance objective and health event under surveillance. Timeliness Table adapted from References 21, 26, 27, 31, and 37. The International Classification of Disease (ICD) is the diagnostic classification standard for all clinical and research purposes and is maintained by the World Health Organization. The Logical Observations Identifiers Names and Codes (LOINC), produced by the Regenstrief Institute, is a clinical terminology for laboratory test orders and results. The Systematized Nomenclature of Medicine (SNOMED) is a systematic, computer-processable collection of medical terms in human and veterinary medicine that provides codes, terms, synonyms, and definitions covering anatomy, diseases, findings, procedures, microorganisms, and substances. b a interventions (60, 68). Investigators interested in early recognition and management of chronic diseases have improved diabetes case detection timeliness and sensitivity by developing and validating an automated, real-time diabetes case-finding algorithm using data extracted from a comprehensive electronic health record (53). Similar algorithms are being developed and used for electronic case reporting of notifiable infectious diseases in some jurisdictions to increase surveillance timeliness and the sensitivity of case detection (9, 49). Syndromic surveillance systems originally developed to support early detection of public health events are designed to alert at very low thresholds, which often results in high false-positive rates (i.e., low positive predictive value). Historically, practitioners have tried to improve the accuracy of syndromic surveillance systems by modifying statistical outbreak detection algorithms or by comparing results from different surveillance systems. Using methods borrowed from chronic disease surveillance, researchers have identified patient, physician, encounter, and billing characteristics (e.g., patient age, workload, and treating www.annualreviews.org • Public Health Surveillance Systems 71 physician’s billing history) that can be used to adjust analyses of syndromic surveillance data to reduce false positive alerts (19). Assessments of surveillance data quality have typically reported the proportion of selected data variables with missing or unknown data. Less frequently, investigators have attempted to validate the accuracy of the values reported for selected data variables. Two studies looking at the prevalence of cancers and sexually transmitted diseases among American Indians linked surveillance data to Indian Health Service registries to determine the accuracy of race and ethnicity data and the degree of misclassification (40, 86). In both studies, American Indian race was misclassified and underreported, resulting in underestimation of the burden of these health conditions among American Indians and limiting the usefulness of these data for monitoring progress towards addressing health disparities. An assessment of the quality of address data in reportable campylobacteriosis notifications in Montreal identified errors in 10% of reported addresses and missing addresses in 5% of the notifications (90). Address correction changed the case geolocation by a median of 1.1 km—which is significant if interventions are implemented based on neighborhood risk. Further evaluation of the impact of errors in public health data is warranted, focusing in particular on how these errors affect routine public health practice (e.g., geocoding for spatial monitoring or cluster detection or use of surveillance to direct and evaluate interventions or allocate resources). Everything done to influence surveillance processes or activities should support the objectives of the surveillance system being reviewed or evaluated. Use of new methods or tools should be pursued if they are likely to improve surveillance data quality or system performance (with regard to the system’s objectives). Investments in improvements to the performance of surveillance systems should connect to the “so what?” of public health; that is, new technologies or modified surveillance processes should support the generation and application of information for public health decision making and policy formulation. OPPORTUNITIES TO ENHANCE THE EFFECTIVENESS OF SURVEILLANCE SYSTEMS To make surveillance more scientific, one must encourage systematic observation and measurement, data collection, more precise analysis, and more effective data dissemination. In addition, both good science and good management require rigorous evaluation of current surveillance systems and alternative approaches to data collection, analysis, and dissemination. (85, p. 188) This statement is just as applicable today as it was at the time it was written in 1989. Opportunities to improve public health surveillance systems must take a principled approach that will generate evidence of the highest possible quality. Moreover, this evidence must be translated into practice to reduce unnecessary variation in the conduct of surveillance and to increase the efficiency and effectiveness of surveillance and of the public health system more generally. Doing so can be challenging for many reasons, including the increasingly multisectoral nature of surveillance as public health agencies engage with stakeholders to ensure public security and promote health. Despite these challenges, opportunities for enhancing surveillance practice exist in terms of increased scientific rigor, outcomes-focused research, and health informatics. Scientific Rigor and Outcomes in Research on Surveillance Systems Although surveillance can benefit from research in many different fields, research on surveillance is ultimately an applied science. It can be challenging to conduct applied research, because practical demands and resource constraints will frequently conflict with research activities. The 72 Groseclose · Buckeridge field of implementation science, however, can provide helpful guidance in this regard (51). For example, innovations in surveillance systems could be introduced in a staged manner across public health departments using a step-wedged cluster-randomized design (36) to minimize bias in the assessment of the effect of system changes. Given the many components of a surveillance system, it is perhaps not surprising that much of the research in this field has focused on evaluating or improving specific components of the surveillance process as opposed to evaluating the overall impact of a system. For example, the last decade has seen a considerable amount of research on the statistical detection of aberrations in surveillance data (15, 50), but most of these studies have focused on the statistical properties of detection algorithms. With few exceptions, researchers have tended not to examine the effect of detection algorithms on outcomes, such as the time to initiating a response (17) or the number of infections averted (46). A greater focus on outcomes, however, could provide a clearer picture of which enhancements to surveillance systems are likely to make a difference in population health and should be implemented. Measurement of costs and cost-effectiveness is another opportunity to enhance surveillance systems. Although a cost-effectiveness perspective is readily accepted in the evaluation of new clinical interventions, few evaluations of surveillance system modifications have taken this approach (83). For example, public health agencies are frequently faced with the question of whether they should use newly available data streams or adopt new analytical methods. Although it is important to conduct research that characterizes a novel data source or method, the insights from such evaluations do not fully address the question of whether the new procedure should be adopted in practice. Quantifying the costs and the effectiveness of innovations in public health surveillance systems would provide decision makers with highly relevant evidence to guide the allocation of resources. The Role of Informatics in Surveillance Systems Public health informatics has grown rapidly as a discipline in the past two decades (30, 89), and surveillance in particular has been the subject of many studies in the informatics literature (47). Informatics research has helped to advance surveillance systems in many ways, including by facilitating access to new data streams from clinical and other sources (16, 29, 43, 52); automating surveillance processes, such as case detection from free text (22) and aberration detection (38); and enabling the rapid dissemination of information produced by surveillance to a wide range of stakeholders (20). Building on these advances, many opportunities exist for health informatics methods to enhance surveillance systems. One opportunity is the development of controlled terminologies and ontologies for public health surveillance and public health practice in general. Scientific advancement in any field requires clarity and precision in measurement and communication. The establishment of common terminologies can facilitate comparative research and the sharing of software, including the sharing of surveillance software across public health agencies. An Australian group of researchers has taken a first step in this direction by developing a taxonomy of public health actions including surveillance (41), and other researchers have built upon this model and used it as the core of a chronic disease surveillance system (59). Similar work has developed shareable models to guide the conduct of reportable disease surveillance (25). This type of work provides a scalable and open foundation for ongoing improvements in public health surveillance, as it can support the sharing, reuse, and refinement of software systems and public health knowledge. Informatics also offers the opportunity to enhance public health surveillance by enabling new ways to engage the public. For example, some researchers have demonstrated how members of the general public can be recruited to receive messages by email or text routinely asking them to www.annualreviews.org • Public Health Surveillance Systems 73 report the onset of symptoms, such as influenza-like illness (80, 87). Surveillance systems based on participatory cohorts provide information similar to that obtained through sentinel surveillance (4), and they address some of the limitations of surveillance methods based on telephone survey, which may not adequately sample populations without telephone lines (44, 55). Participatory cohorts, however, are subject to sampling bias, and loss of cohort members to attrition can pose a challenge. Another novel use of informatics to enhance surveillance is through social media. Much of the research on this approach has focused on passive monitoring of social media data, such as Twitter, to measure disease activity (76), health behaviors (57), or adverse events (65). In general, evaluations suggest that this approach can produce information that can add value to other surveillance strategies (73); however, the data may not be representative or may lack demographic and geographical details that are important for some surveillance objectives. These limitations are offset to some extent by the ability to use social media not just to gather data, but also to engage individuals, for example by detecting comments about food poisoning and encouraging people to contact a public health agency to file a report (34). CONCLUSION Surveillance is critical for public health, and surveillance systems are needed to conduct surveillance efficiently and effectively. The main output of surveillance systems is information that drives action; consequently, systems require data of sufficient quality and with a resolution and timeliness that match public health objectives. In many ways, surveillance presents a working model of what has been called a “learning health system” (7). The repeated cycle of observation, data analysis to identify opportunities for improvement, and implementation of changes provides a means for rapidly converting data into actionable information to improve population health. Surveillance and the systems that enable surveillance have been the focus of applied research in public health for decades; in recent years, however, there has been an increasing interest in surveillance methods in the context of changing public health objectives and emerging opportunities for data access and computing. Surveillance systems are now applied broadly across many areas beyond infectious diseases, and this increasing breadth of application, along with the associated methodological demands (e.g., rapid signal detection, data integration), is driving innovation and contributing to the scientific basis of public health surveillance. Scholars have emphasized the importance of evaluating key attributes of surveillance systems for decades; today, they increasingly recognize the routine monitoring of key system processes and outcomes as needed to aid data interpretation, ensure quality, and guide focused evaluation activities. Future improvements to surveillance systems must be driven by public health objectives and should be evaluated rigorously to determine their effects and, ideally, costs. We encourage public health surveillance practitioners to focus on opportunities to apply the data in new ways (as reflected in the system’s objectives); to embrace the possibilities that informatics offers to enhance data quality and system efficiency and effectiveness; to incorporate routine monitoring of system performance; and to apply sound study designs to evaluate the effect of innovations on the outcomes of interest. This renewed approach to surveillance systems will be instrumental in improving public health practice and enhancing community health. DISCLOSURE STATEMENT The authors are not aware of any affiliations, memberships, funding, or financial holdings that might be perceived as affecting the objectivity of this review. The findings and conclusions in this 74 Groseclose · Buckeridge review are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention. LITERATURE CITED 1. Althouse BM, Ng YY, Cummings DAT. 2011. Prediction of dengue incidence using search query surveillance. PLOS Negl. Trop. Dis. 5:e1258 2. Althouse BM, Scarpino SV, Meyers LA, Ayers JW, Bargsten M, et al. 2015. Enhancing disease surveillance with novel data streams: challenges and opportunities. EPJ Data Sci. 4:17. https://doi.org/ 10.1140/epjds/s13688-015-0054-0 3. Anderson BD, Ma M, Xia Y, Wang T, Shu B, et al. 2016. Bioaerosol sampling in modern agriculture: a novel approach for emerging pathogen surveillance? J. Infect. Dis. 214:537–45. https://doi.org/ 10.1093/infdis/jiw180 4. Bajardi P, Vespignani A, Funk S, Eames KT, Edmunds WJ, et al. 2014. Determinants of follow-up participation in the Internet-based European influenza surveillance platform Influenzanet. J. Med. Internet Res. 16:e78. https://doi.org/10.2196/jmir.3010 5. Baker MA, Nguyen M, Cole DV, Lee GM, Lieu TA. 2013. Post-licensure rapid immunization safety monitoring program (PRISM) data characterization. Vaccine 31(Suppl.):K98–112 6. Bednarczyk RA, Orenstein WA, Omer SB. 2016. Estimating the number of measles-susceptible children and adolescents in the United States using data from the National Immunization Survey-Teen (NISTeen). Am. J. Epidemiol. 184:148–56. https://doi.org/10.1093/aje/kwv320 7. Bernstein JA, Friedman C, Jacobson P, Rubin JC. 2015. Ensuring public health’s future in a national-scale learning health system. Am. J. Prev. Med. 48: 480–87 8. Biggerstaff M, Alper D, Dredze M, Fox S, Fung IC, et al. 2016. Results from the Centers for Disease Control and Prevention’s Predict the 2013–2014 Influenza Season Challenge. BMC Infect. Dis. 16:357. https://doi.org/10.1186/s12879-016-1669-x 9. Birkhead GS, Klompas M, Shah NR. 2015. Uses of electronic health records for public health surveillance to advance public health. Annu. Rev. Public Health 36: 345–59 10. Blondin KJ, Giles CM, Cradock AL, Gortmaker SL, Long MW. 2016. US state childhood obesity surveillance practices and recommendations for improving them, 2014–2015. Prev. Chronic Dis. 13:160060. https://doi.org/10.5888/pcd13.160060 11. Bowen MD, Mijatovic-Rustempasic S, Esona MD, Teel EN, Gautam R, et al. 2016. Rotavirus strain trends during the postlicensure vaccine era: United States, 2008–2013. J. Infect. Dis. 214:732–38. https://doi.org/10.1093/infdis/jiw233 12. Broniatowski DA, Paul MJ, Dredze M. 2013. National and local influenza surveillance through Twitter: an analysis of the 2012–2013 influenza epidemic. PLOS ONE 8:e83672. https://doi.org/ 10.1371/journal.pone.0083672 13. Brownson RC, Fielding JE, Maylahn CM. 2009. Evidence-based public health: a fundamental concept for public health practice. Annu. Rev. Public Health 30:175–201 14. Buckeridge DL. 2007. Outbreak detection through automated surveillance: a review of the determinants of detection. J. Biomed. Inform. 40:370–79 15. Buckeridge DL, Burkom H, Campbell M, Hogan WR, Moore AW. 2005. Algorithms for rapid outbreak detection: a research synthesis. J. Biomed. Inform. 38:99–113 16. Buckeridge DL, Charland K, Labban A, Ma Y. 2014. A method for neighborhood-level surveillance of food purchasing. Ann. N. Y. Acad. Sci. 1331:270–77. https://doi.org/10.1111/nyas.12332 17. Buckeridge DL, Owens DK, Switzer P, Frank J, Musen MA. 2006. Evaluating detection of an inhalational anthrax outbreak. Emerg. Infect. Dis. 12:1942–49 18. Burris S, Hitchcock L, Ibrahim J, Penn M, Ramanathan T. 2016. Policy surveillance: a vital public health practice comes of age. J. Health Polit. Policy Law. In press. https://doi.org/10.1215/03616878–3665931 19. Cadieux G, Buckeridge DL, Jacques A, Libman M, Dendukuri N, Tamblyn R. 2012. Patient, physician, encounter, and billing characteristics predict the accuracy of syndromic surveillance case definitions. BMC Public Health 12:166 www.annualreviews.org • Public Health Surveillance Systems 75 20. Carroll LN, Au AP, Detwiler LT, Fu T, Painter IS, Abernethy NF. 2014. Visualization and analytics tools for infectious disease epidemiology: a systemic review. J. Biomed. Inform. 51:287–98 21. CDC (Cent. Dis. Control Prev.). 2001. Updated guidelines for evaluating public health surveillance systems: recommendations from the guidelines working group. Morb. Mortal. Wkly. Rep. Recomm. Rep. 50(RR-13):1–35 22. Chapman WW, Dowling JN, Wagner MM. 2005. Classification of emergency department chief complaints into 7 syndromes: a retrospective analysis of 527,228 patients. Ann. Emerg. Med. 46:445–55 23. Chriqui JF, O’Connor JC, Chaloupka FJ. 2011. What gets measured, gets changed: evaluating law and policy for maximum impact. J. Law Med. Ethics 39(Suppl. 1):21–26 24. Condell O, Midgley S, Christiansen CB, Chen M, Chen Nielsen X, et al. 2016. Evaluation of the enterovirus laboratory surveillance system in Denmark, 2010 to 2013. Euro Surveill. 21(18):30218. https://doi.org/10.2807/1560–7917.ES.2016.21.18.30218 25. Doyle TJ, Ma H, Groseclose SL, Hopkins RS. 2005. PHSkb: a knowledgebase to support notifiable disease surveillance. BMC Med. Inform. Decis. Mak. 5:27 26. Drewe JA, Hoinville LJ, Cook AJC, Floyd T, Stark KDC. 2012. Evaluation of animal and public health surveillance systems: a systematic review. Epidemiol. Infect. 140:575–90 27. Eur. Cent. Dis. Prev. Control. 2014. Data Quality Monitoring and Surveillance System Evaluation: A Handbook of Methods and Applications. Stockholm, Swed.: Eur. Cent. Dis. Prev. Control. http://ecdc.europa. eu/en/publications/publications/data-quality-monitoring-surveillance-system-evaluation-sept2014.pdf 28. Foege WH, Hogan RC, Newton LH. 1976. Surveillance projects for selected diseases. Int. J. Epidemiol. 5(1):29–37 29. Freifeld CC, Mandl KD, Reis BY, Brownstein JS. 2008. HealthMap: global infectious disease monitoring through automated classification and visualization of Internet media reports. J. Am. Med. Inform. Assoc. 15:150–57 30. Friede A, Blum HL, McDonald M. 1995. Public health informatics: how information-age technology can strengthen public health. Annu. Rev. Public Health 16:239–52 31. Groseclose SL, German RR, Nsubuga P. 2010. Evaluating public health surveillance. In Principles and Practice of Public Health Surveillance, ed. LM Lee, SM Teutsch, ME St. Louis, SB Thacker, pp. 166–97. New York: Oxford Univ. Press. 3rd ed. 32. Hall HI, Mokotoff ED. 2007. Setting standards and an evaluation framework for human immunodeficiency virus/acquired immunodeficiency syndrome surveillance. J. Public Health Manag. Pract. 13:519–23 33. Hall HI, Song R, Gerstle JE III, Lee LM. 2006. Assessing the completeness of reporting of human immunodeficiency virus diagnoses in 2002–2003: capture-recapture methods. Am. J. Epidemiol. 164:391– 97 34. Harris JK, Mansour R, Choucair B, Olson J, Nissen C, Bhatt J. 2014. Health department use of social media to identify foodborne illness—Chicago, Illinois, 2013–2014. Morb. Mortal. Wkly. Rep. 63:681–85 35. Harrison C, Jorder M, Stern H, Stavinsky F, Reddy V, et al. 2014. Using online reviews by restaurant patrons to identify unreported cases of foodborne illness—New York City, 2012–2013. Morb. Mortal. Wkly. Rep. 63:441–45 36. Hemming K, Lilford R, Girling AJ. 2015. Stepped-wedge cluster randomised controlled trials: a generic framework including parallel and multiple-level designs. Stat. Med. 34:181–96. https://doi.org/ 10.1002/sim.6325 37. Hopkins RS. 2005. Design and operation of state and local infectious disease surveillance systems. J. Public Health Manag. Pract. 11:184–90 38. Hulth A, Andrews N, Ethelberg S, Dreesman J, Faensen D, et al. 2010. Practical usage of computersupported outbreak detection in five European countries. Euro Surveill. 15(36):19658 39. Inst. Med. 1988. The Future of Public Health. Washington, DC: Natl. Acad. Press. doi:10.17226/1091 40. Jim MA, Arias E, Seneca DS, Hoopes MJ, Jim CC, et al. 2014. Racial misclassification of American Indians and Alaska Natives by Indian Health Service contract health service delivery area. Am. J. Public Health 104:S295–302. https://doi.org/10.2105/AJPH.2014.301933 41. Jorm L, Gruszin S, Churches T. 2009. A multidimensional classification of public health activity in Australia. Aust. N. Z. Health Policy 6:9. https://doi.org/10.1186/1743-8462-6-9 76 Groseclose · Buckeridge 42. Karch DL, Chen M, Tang T. 2014. Evaluation of the National Human Immunodeficiency Virus Surveillance System for the 2011 diagnosis year. J. Public Health Manag. Pract. 20:598–607 43. Kass-Hout TA, Xu Z, Mohebbi M, Nelsen H, Baker A, et al. 2016. OpenFDA: an innovative platform providing access to a wealth of FDA’s publicly available data. J. Am. Med. Inform. Assoc. 23:596–600. https://doi.org/10.1093/jamia/ocv153 44. Kempf AM, Remington PL. 2007. New challenges for telephone survey research in the twenty-first century. Annu. Rev. Public Health 28:113–26 45. Kirkcaldy RD, Harvey A, Papp JR, del Rio C, Soge OO, et al. 2016. Neisseria gonorrhoeae antimicrobial susceptibility surveillance—the Gonococcal Isolate Surveillance Project, 27 sites, United States, 2014. Morb. Mortal. Wkly. Rep. Surveill. Summ. 65(7):1–19 46. Kleinman KP, Abrams AM. 2008. Assessing the utility of public health surveillance using specificity, sensitivity, and lives saved. Stat. Med. 27:4057–68 47. Kukafka R, Yasnoff WA. 2007. Public health informatics. J. Biomed. Inform. 40:365–69 48. Langley G, Schaffner W, Farley MM, Lynfield R, Bennett NM, et al. 2015. Twenty years of Active Bacterial Core surveillance. Emerg. Infect. Dis. 21:1520–28 49. Lazarus R, Klompas M, Campion FX, NcNabb SJN, Hou X, et al. 2009. Electronic support for public health: validated case finding and reporting for notifiable diseases using electronic medical data. J. Am. Med. Inform. Assoc. 16:18–24. https://doi.org/10.1197/jamia.M2848.a 50. Lewis B, Eubank S, Abrams AM, Kleinman K. 2013. In silico surveillance: evaluating outbreak detection with simulation models. BMC Med. Inform. Decis. Mak. 13:12 51. Lobb R, Colditz GA. 2013. Implementation science and its application to population health. Annu. Rev. Public Health 34:235–51 52. Ma H, Rolka H, Mandl K, Buckeridge D, Fleischauer A, Pavlin J. 2005. Implementation of laboratory order data in BioSense Early Event Detection and Situation Awareness System. Morb. Mortal. Wkly. Rep. 54(Suppl.):27–30 53. Makam AN, Nguyen OK, Moore B, Ma Y, Amarasingham R. 2013. Identifying patients with diabetes and the earliest data of diagnosis in real time: an electronic health record case-finding algorithm. BMC Med. Inform. Decis. Mak. 13:81 54. McNabb SJN, Surdo AM, Redmond A, Cobb J, Wiley J, et al. 2004. Applying a new conceptual framework to evaluate tuberculosis surveillance and action performance and measure the costs, Hillsborough County, Florida, 2002. Ann. Epidemiol. 14:640–45 55. Mokdad AH. 2009. The Behavioral Risk Factor Surveillance System: past, present, and future. Annu. Rev. Public Health 30:43–54 56. Morrison KT, Shaddick G, Henderson SB, Buckeridge DL. 2016. A latent process model for forecasting multiple time series in environmental public health surveillance. Stat. Med. 35:3085–100 57. Mysl´n M, Zhu SH, Chapman W, Conway M. 2013. Using Twitter to examine smoking behavior and ı perceptions of emerging tobacco products. J. Med. Internet Res. 15(8):e174 58. Nsoesie EO, Buckeridge DL, Brownstein JS. 2014. Guess who’s not coming to dinner? Evaluating online restaurant reservations for disease surveillance. J. Med. Internet Res. 16(1):e22 59. Okhmatovskaia A, Shaban-Nejad A, Lavigne M, Buckeridge DL. 2014. Addressing the challenge of encoding causal epidemiological knowledge in formal ontologies: a practical perspective. Stud. Health Technol. Inform. 205:1125–29 60. Osborne JD, Wyatt M, Westfall AO, Willig J, Bethard S, Gordon G. 2016. Efficient identification of nationally mandated reportable cancer cases using natural language processing and machine learning. J. Am. Med. Inform. Assoc. In press. https://doi.org/10.1093/jamia/ocw006 61. Parton HB, Mathes R, Abdelnabi J, Alleyne L, Econome A, et al. 2016. Investigating a syndromic surveillance signal with complementary data systems. Online J. Public Health Inform. 8(1):e152 62. Patwardhan A, Bilkovski R. 2012. Comparison: flu prescription sales data from a retail pharmacy in the US with Google flu trends and US ILINet (CDC) data as flu activity indicator. PLOS ONE 7(8):e43611 63. Pinner RW, Lynfield R, Hadler JL, Schaffner W, Farley MM, et al. 2015. Cultivation of an adaptive domestic network for surveillance and evaluation of emerging infections. Emerg. Infect. Dis. 21:1499–509 64. Pollack KM, D’Angelo J, Green G, Conte S, Fealy S, et al. 2016. Developing and implementing Major League Baseball’s health and injury tracking system. Am. J. Epidemiol. 183:490–96 www.annualreviews.org • Public Health Surveillance Systems 77 65. Powell GE, Seifert HA, Reblin T, Burstein PJ, Blowers J, et al. 2016. Social media listening for routine post-marketing safety surveillance. Drug Saf. 39:443–54 66. Presley D, Reinstein T, Webb-Barr D, Burris S. 2015. Creating legal data for public health monitoring and evaluation: Delphi standards for policy surveillance. J. Law Med. Ethics 43:27–31 67. Prue CE, Lackey C, Swenarski L, Gantt JM. 2003. Communication monitoring: shaping CDC’s emergency risk communication efforts. J. Health Commun. 8(Suppl. 1):35–49 68. Puckett M, Neri A, Rohan E, Clerkin C, Underwood JM, et al. 2016. Evaluating early case capture of pediatric cancers in seven central cancer registries in the United States, 2013. Public Health Rep. 131:126–36 69. Roush SW. 2011. Surveillance indicators. In Manual for the Surveillance of Vaccine-Preventable Diseases, ed. Cent. Dis. Control Prev. Atlanta, GA: Cent. Dis. Control Prev. 5th ed. http://www.cdc.gov/ vaccines/pubs/surv-manual/chpt18-surv-indicators.pdf 70. Sabbe M, Berger N, Blommaert A, Ogunjimi B, Grammens T, et al. 2016. Sustained low rotavirus activity and hospitalization rates in the post-vaccination era in Belgium, 2007 to 2014. Euro Surveill. 21(27):30273 71. Salathe M, Friefeld CC, Mekaru SR, Tomasulo AF, Brownstein JS. 2013. Influenza A (H7N9) and the importance of digital epidemiology. N. Engl. J. Med. 369:401–4 72. Salmon D, Yih WK, Lee G, Rosofsky R, Brown J, et al. 2012. Success of program linking data sources to monitor H1N1 vaccine safety points to potential for even broader safety surveillance. Health Aff. 31:2518–27 73. Santillana M, Nguyen AT, Dredze M, Paul MJ, Nsoesie EO, Brownstein JS. 2015. Combining search, social media, and traditional data sources to improve influenza surveillance. PLOS Comput. Biol. 11:e1004513 74. Savard N, B´ dard L, Allard R, Buckeridge DL. 2015. Using age, triage score, and disposition data from e emergency department electronic records to improve Influenza-like illness surveillance. J. Am. Med. Inform. Assoc. 22:688–96 75. Shaban-Nejad A, Lavigne M, Okhmatovskaia A, Buckeridge DL. 2016. PopHR: a knowledge-based platform to support integration, analysis and visualization of population health data. Ann. N. Y. Acad. Sci. In press 76. Signorini A, Segre AM, Polgreen PM. 2011. The use of Twitter to track levels of disease activity and public concern in the U.S. during the Influenza A H1N1 pandemic. PLOS ONE 6:e19467 77. Skoog G, Struwe J, Cars O, Hangerger H, Odenholt I, et al. 2016. Repeated nationwide point-prevalence surveys for antimicrobial use in Swedish hospitals: data for actions 2003–2010. Euro Surveill. 21:30264 78. Smith GE, Elliot AJ, Ibbotson S, Morbey R, Edeghere O, et al. 2016. Novel public health risk assessment process developed to support syndromic surveillance for the 2012 Olympic and Paralympic Games. J. Public Health. In press. https://doi.org/10.1093/pubmed/fdw054 79. Smith PF, Hadler JL, Stanbury M, Rolfs RT, Hopkins RS. 2013. “Blueprint Version 2.0”: updating public health surveillance for the 21st century. J. Public Health Manag. Pract. 19:231–39 80. Smolinski MS, Crawley AW, Baltrusaitis K, Chunara R, Olsen JM, et al. 2015. Flu Near You: crowdsourced symptom reporting spanning 2 influenza seasons. Am. J. Public Health 105:2124–30 81. Snider CJ, Diop OM, Tangermann RH, Wassilek SGF. 2016. Surveillance systems to track progress toward polio eradication—worldwide, 2014–2015. Morb. Mortal. Wkly. Rep. 65: 346–51 82. Somda ZC, Perry HN, Messonnier NR, Djingarey MH, Ki SO, Meltzer MI. 2010. Modeling the costeffectiveness of the Integrated Disease Surveillance and Response (ISDR) system: meningitis in Burkina Faso. PLOS ONE 5(9):e13044 83. Tebbens RJD, Sangrujee N, Thompson KM. 2006. The costs of future polio risk management policies. Risk Anal. 26:1507–31 84. Thacker SB, Berkelman RL. 1988. Public health surveillance in the United States. Epidemiol. Rev. 10:164– 90 85. Thacker SB, Berkelman RL, Stroup DF. 1989. The science of public health surveillance. J. Public Health Policy 10:187–203 86. Thoroughman DA, Frederickson D, Cameron HD, Shelby LK, Cheek JE. 2002. Racial misclassification of American Indians in Oklahoma state surveillance data for sexually transmitted diseases. Am. J. Epidemiol. 155:1137–41 78 Groseclose · Buckeridge 87. van Noort SP, Codeco CT, Koppeschaar CE, van Ranst M, Paolotti D, Gomes MG. 2015. Ten-year performance of Influenzanet: ILI time series, risks, vaccine effects, and care-seeking behaviour. Epidemics 13:28–36 88. Wong WK, Moore A, Cooper G, Wagner MM. 2003. WSARE: What’s Strange About Recent Events? J. Urban Health 80(Suppl. 1):i66–75 89. Yasnoff WA, O’Carroll PW, Koo D, Linkins RW, Kilbourne EM. 2000. Public health informatics: improving and transforming public health in the information age. J. Public Health Manag. Pract. 6:67–75 90. Zinszer K, Jauvin C, Verma A, Bedard L, Allard R, et al. 2010. Residential address errors in public health surveillance data: a description and analysis of the impact on geocoding. Spat. Spatiotemporal Epidemiol. 1(2–3):163–68 www.annualreviews.org • Public Health Surveillance Systems http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Annual Review of Public Health Annual Reviews

Public Health Surveillance Systems: Recent Advances in Their Use and Evaluation

Download PDF
23 pages

Loading next page...
 
/lp/annual-reviews/public-health-surveillance-systems-recent-advances-in-their-use-and-OadYfUPcu6
Publisher
Annual Reviews
Copyright
Copyright © 2017 Annual Reviews.
ISSN
0163-7525
eISSN
1545-2093
DOI
10.1146/annurev-publhealth-031816-044348
pmid
27992726
Publisher site
See Article on Publisher Site

Abstract

Surveillance is critical for improving population health. Public health surveillance systems generate information that drives action, and the data must be of sufficient quality and with a resolution and timeliness that matches objectives. In the context of scientific advances in public health surveillance, changing health care and public health environments, and rapidly evolving technologies, the aim of this article is to review public health surveillance systems. We consider their current use to increase the efficiency and effectiveness of the public health system, the role of system stakeholders, the analysis and interpretation of surveillance data, approaches to system monitoring and evaluation, and opportunities for future advances in terms of increased scientific rigor, outcomes-focused research, and health informatics. INTRODUCTION Public health surveillance, and by extension the systems used to enable surveillance, is central to the practice of modern public health. Public health surveillance contributes data and information to assess and characterize the burden and distribution of adverse health events, prioritize public health actions, monitor the impact of control measures, and identify emerging health conditions that may have a significant impact upon population health. The core role of surveillance systems within public health practice, and their concomitant capacity to greatly influence the efficiency and effectiveness of the public health system, has stimulated research to strengthen the scientific basis of public health surveillance. In 1970, only 7% of PubMed articles about surveillance (20/277) focused on methods, but that proportion rose to 60% by 2015 (7,400/12,400). In the context of scientific advances in public health surveillance, changing health care and public health environments, and rapidly evolving technologies, the aim of this article is to review public health surveillance systems, including their current role, recent advances, and opportunities for future advances. This review is divided into three sections. In the first two sections, we review the contemporary use and evaluation of surveillance systems in public health practice and highlight some notable recent advances in their use and evaluation through case studies and other examples. In the third section, we discuss some promising opportunities for advancing surveillance systems in the future and highlight notable research activities. THE USE OF SURVEILLANCE SYSTEMS The Concept of a Surveillance System Surveillance, a core function of public health practice, is defined as “the ongoing, systematic collection, analysis, and interpretation of health data essential to the planning, implementation, and evaluation of public health practice, closely integrated with the timely dissemination of [this information] to those who need to know” and act upon that information (84, p. 164). A surveillance system, in turn, is a collection of processes and components that enable public health practitioners to conduct surveillance. Surveillance processes include data collection, data quality monitoring, data management, data analysis, interpretation of analytical results, information dissemination, and application of the information to public health programs. The enabling components of surveillance systems may include laboratory diagnostics to detect or confirm health conditions; information technologies to support the surveillance processes of data collection, analysis, and dissemination; clinician consultation and reporting; clinician, public health, and laboratory worker education and training; legislation, regulations, and policies that support the conduct of surveillance; systems and directories for disseminating alerts, bulletins, clinical guidelines, and prevention recommendations; program administration and management; and human factors (e.g., multisector communications and relationships) (54). Ultimately, public health surveillance systems should produce information to guide public health decisions in many areas, including disease prevention, prevention program planning and management, health promotion, quality improvement, and resource allocation. Using Surveillance Systems to Meet Public Health Objectives From a societal perspective, public health surveillance systems should increase the efficiency and effectiveness of the public health system, which is a primary determinant of population health. A surveillance system affects population health by capturing data and generating information that 58 Groseclose · Buckeridge public health practitioners and stakeholders can use to improve the quality of their decisions and the effectiveness of their actions. From the perspective of the public health system, surveillance systems support all three essential functions of public health—assurance, assessment, and policy development (39). For a specific surveillance system, however, the objectives should be defined more precisely (Table 1) (see also Supplemental Case Studies 1, 2, and 3; follow the Supplemental Material link in the online version of this article or at http://www.annualreviews.org/). These objectives should be tailored to the outcomes under surveillance, the intended uses of the information generated by the system, and the level (e.g., local, regional, national) of the public health system at which the surveillance system is functioning. The processes and components of a public health surveillance system should in turn be aligned with the objectives of the surveillance system to ensure that valid information can be derived and applied to practice, to promote operational efficiency, and to ensure that the activities are within the legal or regulatory mandate of the public health system. Explicitly documented objectives for the surveillance system are also important for planning the system, evaluating system performance, and enabling continuous improvement of data and system quality. Surveillance objectives and budget should determine the number and type of data variables to be collected (e.g., demographic or behavioral data variables), including the required level of resolution of the data, the population under surveillance, the required timeliness of information for effective action or response, the frequency of data analysis and interpretation, and the resources required to support the surveillance system. Similarly, surveillance objectives should influence decisions about data collection, management, analysis, integration, dissemination, security, and privacy. It is best to identify inconsistencies between the objectives (and their implications for system design and performance) and resources at the planning stage so that one or the other can be adjusted accordingly. Ultimately, a public health surveillance system’s objectives indicate how the data are intended to be used for public health action. Over 40 years ago, Dr. William Foege wrote, The reason for collecting, analyzing and disseminating information on a disease is to control that disease. Collection and analysis should not be allowed to consume resources if action does not follow. Appropriate action, therefore, becomes the ultimate goal and the final assessment of the earlier steps of a surveillance system. (28, p. 30) Supplemental Material Surveillance data have been used to guide a range of public health actions. Table 1 presents examples of surveillance data use by surveillance system objective to illustrate the range of public health actions that can be informed by public health surveillance data. The Expanding Use of Surveillance Systems The initial focus of public health surveillance principles and practices was on infectious diseases, but today public health surveillance systems are used to monitor and forecast a broad range of health determinants (e.g., risk behaviors, health care services, and socioeconomic factors) and outcomes relevant to infectious diseases, injuries, chronic diseases, mental health, and occupational and environmental health. The case studies on the Major League Baseball Health and Injury Tracking System (MLB HITS) and active transportation surveillance are examples of surveillance systems focused on noninfectious diseases (see Supplemental Case Studies 1 and 3). In particular, in the case of the MLB HITS, two non–public health entities, an industry (MLB) and its worker association, established a surveillance system to identify and monitor work-related injuries and associated risk factors and to assure the effectiveness of the intervention (64). www.annualreviews.org • Public Health Surveillance Systems 59 Table 1 Surveillance objectives and examples of surveillance system data use by surveillance objective Surveillance system Acute flaccid paralysis (AFP) surveillance (24, 81; see Supplemental Case Study 2) Surveillance method Monitoring of AFP (a clinical syndrome with several potential etiologies, including poliovirus) as reportable condition in countries not reporting confirmed polio cases Etiology of AFP cases is determined via testing of stool specimens Collection of child height and weight data, typically in a school-based setting at least every two years Example of surveillance data use Determine extent and duration of wild poliovirus circulation; identify causes of outbreak or reason for AFP clustering; initiate control measures (immunization, enhanced surveillance) to interrupt transmission, prevent spread, or improve detection capability Surveillance objective Guide immediate action for cases of public health importance (e.g., initiating investigations or interventions) Measure the burden of a disease (or other health-related event) and monitor trends over time, including changes in incidence and the identification of high-risk populations Support early detection and response to outbreaks or new or emerging health concerns State-based child body mass index (BMI) surveillance systems (10) Detect disparities in prevalence of overweight and obesity based on socioeconomic status and race/ethnicity; identify the determinants of local variation in obesity prevalence; evaluate local public health interventions Gonococcal Isolate Surveillance Project (45) Sentinel surveillance system to monitor trends in antimicrobial susceptibilities of Neisseria gonorrhoeae strains among men with gonococcal urethritis attending 1 of 27 sexually transmitted disease clinics in the United States Monitoring of three million people who received H1N1 vaccine by linking data from private health plans and public immunization registries; analysis of data to identify time-invariant confounders (e.g., chronic illness) and rare outcomes (comparison of current and historical data) Guide development of national treatment recommendations for effective therapy and prevention of gonorrhea; set research and prevention priorities; guide the planning and allocation of STD prevention services and resources; inform clinical practice Guide the planning, implementation, and evaluation of programs to prevent and control disease, injury, disability, or exposure to environmental hazards Post-licensure Rapid Immunization Safety Monitoring (PRISM) program, the immunization safety monitoring component of US Food and Drug Administration’s Mini-Sentinel project (5, 72) Identify adverse health events potentially associated with H1N1 vaccination to increase our understanding of vaccine safety and to inform immunization policy (Continued ) Supplemental Material Groseclose · Buckeridge Table 1 (Continued ) Surveillance system Syndromic surveillance system monitoring over-the-counter (OTC) pharmacy sales (61) Surveillance method Monitoring of OTC antidiarrheal medication sales to identify unusual sales patterns; monitoring of pharmacy sales promotions for these products to aid interpretation of sales data Example of surveillance data use Investigate citywide syndromic surveillance signals identified in sales of OTC antidiarrheal medications using multiple communicable disease surveillance systems data [emergency department visits with chief complaint of diarrhea, school nurse visits for stomach ache or diarrhea, enteric reportable diseases, clinical laboratory stool specimen submissions, water quality indicators and complaints, and social media mentions of key words (e.g., diarrhea, loperamide, water quality)]. Findings possibly reflected sales promotions but not increased diarrheal illness Guide vaccine development; determine vaccine effectiveness; formulate immunization policy Surveillance objective Provide reassurance during periods of perceived increased risk that incidence of a health condition is not increasing Evaluate public policy Active Bacterial Core Surveillance System, a component of CDC Emerging Infections Program (48, 63) Swedish Strategic Programme Against Antibiotic Resistance (77) Population-based, active surveillance network in ten geographically and racially diverse jurisdictions comprising up to 12% of the US population Repeated national point-prevalence surveys of antimicrobial use in Swedish hospitals [monitored treatments for predefined diagnostic groups, defined daily dose (a measure of antibiotic pressure), and reason for antimicrobial use (i.e., prophylaxis, community-acquired infection, hospital-acquired infection)] Annual national household survey of parents of adolescents (13–17 years of age) and provider verification of vaccination Detect changes in health practices and the effects of these changes Note increasing compliance with treatment guidelines for lower urinary tract infections for women and for community-acquired pneumonia. Compliance with surgical prophylaxis guidelines did not improve over the eight-year period. Main intervention was to increase appropriate antimicrobial use through use of an audit feedback approach using local survey data Prioritize the allocation of health resources National Immunization Survey (NIS)—Teen (6) Estimate measles susceptibility, proportion of children vaccinated at each age, and number of adolescents in each state who failed to receive the first dose of measles vaccine; reemphasize the need for high measles vaccination coverage to support population-level immunity and prevent reestablishment of indigenous measles transmission in the United States (Continued ) www.annualreviews.org • Public Health Surveillance Systems Table 1 (Continued ) Surveillance system Surveillance systems and other data sources to determine the impact of rotavirus vaccination in Belgium (70) Surveillance method Eight surveillance systems or data sources, including lab-based surveillance (numbers of tests performed and positive results), hospital discharge surveillance (admissions for rotavirus and all-cause gastroenteritis), mortality due to rotavirus and all-cause gastroenteritis, health services data (rotavirus vaccination coverage) Voluntary enrollment of baseball players into medical record and injury tracking system to monitor injury, medical, treatment, and prevention data Example of surveillance data use Describe changes in timing and age of rotavirus infection (postvaccination annual incidence peak shifted to spring instead of winter; average age at infection and hospitalization increased); obtain evidence for herd immunity (number of lab-confirmed and hospitalized rotavirus cases decreased in both vaccinated and unvaccinated persons); determine changes in rotavirus testing before and after vaccination introduction; estimate vaccination coverage Surveillance objective Describe the clinical course of disease Provide a basis for epidemiologic research Major League Baseball’s (MLB) Health and Injury Tracking System (HITS) (64; see Supplemental Case Study 1) Identify most frequent types of injury; initiate research projects to investigate injury patterns and identify potential interventions (including rules change) Supplemental Material Beyond the monitoring of individual risk factors and outcomes, surveillance systems have also been developed recently to monitor the presence, emergence, or evolution of infectious agents in the environment. For example, a recent surveillance evaluation sought to determine if bioaerosol sampling methods are suitable for routine surveillance for viruses in environments that facilitate aerosolization (3). Investigators compared bioaerosol sampling in swine farms to concurrent animal, environmental, and human sample testing to detect novel influenza viruses. These data supported the identification of novel influenza viruses and of climatic factors and animal husbandry practices that may increase the risk of human exposure to aerosolized influenza A in agricultural settings. The use of surveillance systems has also expanded in relation to communicable diseases, for example to monitor the impact of vaccination programs on viral evolution in order to inform vaccine design and maintain vaccine effectiveness. One such example is the New Vaccine Surveillance Network, which monitors the etiology of acute gastroenteritis in the United States and has documented the decline in Group A rotavirus–associated gastroenteritis incidence among children following reinitiation of rotavirus immunization in the country (11). The Network continues to monitor the genotypic variation in the rotavirus strains circulating in the United States and has noted a major shift in the predominant genotype prevalence in the years 2008–2013. Data from surveillance systems are also used increasingly to forecast or predict future trends in disease distribution. Early identification of the timing and intensity of the annual seasonal epidemic of influenza is useful to inform public health planning and response. To determine if reasonably accurate forecasts can be made during flu season, the Centers for Disease Control and Prevention Groseclose · Buckeridge (CDC) organized a challenge to predict the 2013–2014 United States influenza season (8). Findings indicated that flu forecasting can yield reasonably accurate estimates of the start, peak, and intensity of the influenza season. Although forecasting may be technically feasible, more work is needed to improve its accuracy so that policy makers can use these predictions to guide prevention and control efforts. Many public health agencies have also chosen to actively monitor health communications and news media—especially during a public health emergency—both to refine the public health response and to inform decisions about the creation, alteration, or refinement of health or risk communication messages (67). Recognition that public health law and policy influence community health has led some organizations to establish surveillance systems to monitor the impact of public health laws and policies on communities, environments, and individuals (18, 23, 66). Longitudinal surveillance data used to examine the impact of laws and policies on community health are distinguished from the legislative tracking systems that are commonly utilized to monitor policy interest and activity. Law and policy tracking over time and across jurisdictions can help policy makers, advocates, and researchers understand what the laws are on a particular topic and can provide them with data to evaluate their impact (66). This increasing breadth of application of public health surveillance systems raises many issues, including the engagement of a broader set of stakeholders in surveillance activities and the need to adapt existing analytical methods to new objectives and data sources. In the following two sections, we consider these two issues. The Importance of Stakeholder Engagement in Surveillance Systems Public health organizations often conduct surveillance under a legal mandate, but to maintain the public’s trust, they must conduct surveillance in a manner that is responsible and of sufficient quality to inform population health improvement. The substantial public and private sector investment in surveillance also demands that the data collected by surveillance systems be used and that stakeholders perceive the system to be useful (21). Stakeholders may contribute data or resources to a surveillance effort, act upon the information generated, or use surveillance information to advocate for prevention and control efforts to improve population health. Typical stakeholders include public health practitioners, health care providers, policy makers, and members of affected communities, academia, professional associations, private industry, and not-for-profit advocacy organizations. Engaging with a variety of system stakeholders during system planning, design, implementation, and evaluation encourages broad-based ownership of the surveillance activity and allows the economic, social, and cultural factors that influence prevention and control to be identified and addressed, increasing the likelihood that the information generated by the system will be useful. For surveillance systems to achieve their greatest impact, it is necessary to identify system stakeholders, understand their roles (e.g., contributing data or advocating for assistance to those affected by the health outcome under surveillance), and engage them throughout the surveillance process (21). Prior to the initiation of surveillance, the stakeholder input, such as the information sought for decision making, may inform the definition of system objectives. When the surveillance system is active, stakeholders can help interpret the reported data based on their knowledge of the health outcome or the environment in which the data are collected. They may also respond to the information generated by the surveillance system and recommend and influence surveillance system evaluation to ensure the system is meeting its objectives. Increasingly, the general public can be considered a partner in the surveillance process, as novel data streams “whose content is initiated directly by the user (patient) themselves” are investigated www.annualreviews.org • Public Health Surveillance Systems 63 and integrated into traditional public health surveillance (2, p. 1). Widespread availability of these novel data streams has led to the investigation of new surveillance methods—e.g., natural language processing and monitoring of Twitter posts to augment traditional influenza surveillance (12) or prediction of dengue incidence using search query surveillance (1)—and new surveillance data sources, such as restaurant reservation and review logs (35, 58) or over-the-counter pharmacy sales (62). These novel methods and data sources require validation (e.g., comparison to existing surveillance data) and prospective evaluation to determine if they add value to established surveillance systems. As Salathe et al. (71, p. 403) noted, “the dynamics of information spread are inherently different from the dynamics of disease spread” and the influence of changing user behavior on participatory surveillance and digital health surveillance must be further evaluated. Analysis and Interpretation of Data in Surveillance Systems The analysis of public health surveillance data is typically performed to detect the presence (or absence) of signals (i.e., unusual patterns in the reported data by person, place, and time compared to historical data) that may prompt a public health investigation or other actions. Surveillance data analysis may also be performed to identify the association (or lack thereof) between reported health outcomes and patient or population characteristics to understand the local epidemiology of the condition under surveillance and to determine effective prevention or control measures. Because they are collected and monitored over time, surveillance data can support the detection of trends in reported data by person, place, and time and the identification of changes in disease incidence compared to historical data. Surveillance data analysis and interpretation should directly support the surveillance system objectives and be performed in alignment with surveillance system processes. The content and structure of a valid surveillance system should be defined and monitored to ensure that quality data are received and available for analysis. For example, surveillance data quality review should be initiated as data arrive and conducted at each data update. Similarly, standard analytic routines should be specified prior to receipt of surveillance data and performed at each major data update. In addition, it is useful to anticipate the types and magnitude of signals or surveillance measure values that may be generated by the surveillance system, both to evaluate the system (e.g., by simulating expected signals and determining the capacity of the system to detect those signals) and to define and implement public health action(s) in response to the population health metrics generated by the analysis of the data (14). Ad hoc analyses may be conducted in response to novel questions or stakeholder requests. The available data, together with the system objectives, help to define the analysis options. Traditionally, researchers have used straightforward methods due to limitations in the data (e.g., quantity, resolution, volume) and to a lack of evidence supporting the use of more sophisticated methods. Over the last three decades, the amount of data available has expanded greatly, computing power and storage capacity have increased exponentially, and surveillance objectives have evolved to address a wider set of considerations and more immediate threats. All these factors have pushed innovation in the development and evaluation of analytical methods in surveillance. In particular, the objective of rapid outbreak detection has prompted the examination of a wide range of detection methods and the comparison of these methods against one another using different types of data and outbreaks. This work has contributed some qualitative insights, including the observation that no single detection approach is likely to be optimal for all surveillance applications, and that it may be possible to identify methods that are better suited to one particular application (14). Another recent trend has been toward the integration of data from multiple sources into a single analysis. Whereas traditional surveillance analyses have tended to 64 Groseclose · Buckeridge focus on a single time series of case counts or rates, the increasing availability of data from clinical systems and other sources has provided the opportunity to consider additional covariates within each record (74, 88) and to integrate data on risk factors and multiple outcomes into a single analysis (56). Another strategy for integrating data from multiple sources has been to develop decision rules or policies based on the pattern of alerts observed across data streams (78). Data integration also poses a challenge in chronic disease surveillance, where, given the multifactorial causation and complex progression of most chronic conditions, surveillance systems tend to follow multiple indicators concurrently. One strategy that is being explored to support the effective use of such information draws on the concept of evidence-based public health (13) and presents indicators in the context of existing evidence to help users appreciate patterns in the data and identify actions that are likely to improve population health (75). MONITORING AND EVALUATION OF SURVEILLANCE SYSTEMS Performance monitoring and the evaluation of surveillance systems are complementary processes conducted by public health practitioners and other surveillance system stakeholders. Surveillance system performance is monitored through the ongoing assessment of surveillance processes, such as by following the completeness and timeliness of data collection and reporting. Evaluation focuses on whether the system is meeting its objectives and making effective use of its resources. Both monitoring and evaluation can identify opportunities for surveillance system performance improvement, and both provide information on performance to aid the interpretation of data. Historically, greater attention has been given to evaluation, but monitoring is of increasing interest given the growth in automation and data-intensive systems. Routine Monitoring of Public Health Surveillance Systems The monitoring of a surveillance system can be defined as the routine process of data collection and the measurement of surveillance program or process changes over time using previously agreed-upon plans, schedules, and indicators (27). Discrepancies between actual and planned system performance are identified via routine monitoring, and corrective actions are taken— akin to quality improvement approaches intended to systematically improve the performance of any system. A routine monitoring perspective recognizes that surveillance system performance indicators are not static: They are often reformulated based on evaluation findings (31). Similarly, surveillance monitoring data might indicate where more comprehensive evaluation is needed. Surveillance system performance indicators are defined to specify some aspect of, or a change in, a process, outcome, or activity and to identify sources of variation that, if not addressed, may cause changes in the quality of the information generated by the system or in the system’s performance. Well-defined surveillance indicators are specific in nature, measureable, relevant for the setting in which the system operates, and obtainable in a timely manner (27). Indicators may be of different types, e.g., they may focus on process, outcome, or output. Indicators should have a defined target and be operationalized using a protocol or algorithm that indicates thresholds for alerting or response. It is useful to define the recommended actions to take in response to the indicator data and accompanying protocols to allow the identification of and response to the causes of missed indicator targets. Routine monitoring of surveillance system indicators yields information to assess the current performance of the surveillance system and provides early warning of potential system deviations. Analysis of performance monitoring data allows public health organizations to set goals for system www.annualreviews.org • Public Health Surveillance Systems 65 Supplemental Material performance improvement. Perhaps the most important use of surveillance monitoring data is to aid the interpretation of the data generated by the surveillance system. Surveillance indicators can also be used to set priorities for the surveillance system [e.g., to identify the attributes (timeliness, representativeness, sensitivity, etc.) of greatest importance for the health outcomes under surveillance and for the objectives of the system] or to identify actions that should be undertaken to enhance quality, performance, or impact. Examples of surveillance system performance indicators are provided in the polio surveillance case study (see Supplemental Case Study 2) and in Table 2. Routine monitoring of surveillance systems’ performance is increasingly recognized as an essential surveillance process, particularly for highly automated systems that receive data in real time. Evaluation of Public Health Surveillance Systems A number of factors may indicate that it is time to formally evaluate a surveillance system using a specific study design to assess key surveillance attributes [e.g., sensitivity, positive predictive value, and representativeness (Table 3)] that influence the relevance, effectiveness, and impact of the surveillance system. For example, routine monitoring of surveillance system performance indicators may identify trends that should be explored more thoroughly to identify improvements that can enable the system to better meet its objectives and achieve its desired performance. Unexpected performance, such as a community gastrointestinal disease outbreak that is missed by a foodborne disease surveillance system, may stimulate an evaluation. Discrepancies in the information generated by different surveillance systems may also cause health department staff to initiate an evaluation to determine the reason for the discordance. Evaluation allows the system’s sponsor and stakeholders to determine if the system is generating data that can be used to address its objectives; to make effective use of fiscal, personnel, and technology resources; to meet the needs of stakeholders; and to maintain the system’s public health relevance given changing health care practices, hazards, disease epidemiology, or information needs. Table 3 summarizes the definition of the attributes used to evaluate a surveillance system and the methods employed to measure them. Some attributes are more important for selected surveillance purposes and health outcomes than others (37). The reason for this may be obvious when one considers concurrently the objectives and attributes of a system. For example, rapid outbreak detection is often a stated objective for systems monitoring communicable diseases: Timely and representative population-based health-related data are required to avoid missing outbreaks of public health importance and to ensure rapid public health action. Therefore, surveillance system evaluation is typically driven by the assessment of specific attributes (Table 3); however, if such an evaluation is pursued in a reductionist manner (i.e., by assessing system attributes independent of one another), key insights into the surveillance system process and information may be missed. In other words, the attributes of public health surveillance systems are related to one another; strengthening one system attribute could adversely affect another attribute of a higher priority. For example, data quality, although critical, need only be good enough to meet the objectives of the system. The Council of State and Territorial Epidemiologists recommends that “because no data are perfect and perfecting data can be costly, matching data quality to its use is imperative” (79, p. 238). Data of poor quality make the system less acceptable and potentially less representative of the population under surveillance: Efforts to improve sensitivity, positive predictive value, representativeness, timeliness, and stability can increase the cost of a surveillance system, although savings in efficiency through use of information 66 Groseclose · Buckeridge Table 2 Examples of performance monitoring indicators used by surveillance systems Surveillance objective Measure prevalence of persons with HIV infection and monitor over time Monitoring indicator Completeness of reporting of persons with HIV infection Indicator threshold ≥85% of the expected number of cases for a diagnosis year reported within 12 months of the diagnosis year Method of calculation Jurisdictions use a three-source capture-recapture model to estimate completeness of reporting for persons newly diagnosed with HIV; data sources include health care provider, laboratory, and a third source such as viral hepatitis surveillance data. Jurisdictions link HIV infection case reports with one or more of the following: Vital Statistics Data, Social Security Death Master File, and National Death Index. Intrastate: calculated monthly. Selected case report variables are matched with jurisdiction’s database, followed by manual review and resolution of potential matches. Interstate: calculated semiannually. Selected case report variables are matched across jurisdictions at the national level, followed by manual review and resolution of potential matches. Number of rubella cases reported to NNDSS with complete information for selected variables is compared with total number of rubella cases reported to NNDSS. Indicator type Outcome Surveillance system National Human Immunodeficiency Virus (HIV) Surveillance System, United States (32, 33, 42) Monitor HIV infection case fatality rate and trends Death ascertainment Linkage of HIV case reports to mortality data sources at least annually Process Accurately monitor HIV infection case counts at jurisdictional and national levels Intrastate and interstate duplicate cases ≤5% duplicate cases assessed at 12 months after the close of the report year Process Vaccine-preventable diseases (e.g., rubella) reported to the National Notifiable Diseases Surveillance System (NNDSS), United States (69) Accurately monitor burden of rubella infection and case characteristics Rubella case report data quality Proportion of rubella cases reported to NNDSS with complete information for selected variables (e.g., clinical and epidemiologic case characteristics, lab test results, vaccination history) Outcome (Continued ) www.annualreviews.org • Public Health Surveillance Systems 67 Table 2 (Continued ) Surveillance objective Detect imported rubella cases Monitoring indicator Surveillance system sensitivity to detect rubella cases (because endemic transmission of rubella has been eliminated in the United States) Number of submissions of laboratory requests for rubella diagnostic testing Indicator threshold Proportion of rubella cases having an imported source Method of calculation Number of rubella cases reported to NNDSS that have an imported source is compared with total number of rubella cases reported to NNDSS. Indicator type Outcome Surveillance system Monitor rubella diagnostic efforts Number of submissions of laboratory requests for IgM antibody tests for rubella Electronic lab reports of orders for rubella IgM antibody testing are analyzed per time period Rationale: “If testing occurs, the diagnosis is being considered, so the absence of reported cases is more likely to reflect the absence of disease. Without an external standard, how much testing is ‘enough’ is still open to question” (69). Outcome technology (e.g., electronic reporting) might offset some of these costs. As sensitivity and positive predictive value approach 100%, a system is more likely to be representative of the population with the event under surveillance. However, as sensitivity increases, predictive value positive might decrease. (31, p. 182) A recent systematic review of 99 articles examining animal and human health surveillance system evaluations identified 23 different attributes of surveillance systems (26). Almost one-half of the evaluations assessed only one or two attributes; the attributes assessed most frequently were sensitivity, timeliness, and data quality. The investigators noted with some concern that surveillance objectives for the system being evaluated were often not stated, and therefore the rationale for assessing the selected attributes was not always evident. Advances in Surveillance System Evaluation Investigators continue to pursue novel methods to evaluate and improve the ability of surveillance systems to support new uses of data. Historically, cancer surveillance systems did not focus on improving timeliness because immediate uses of the data for the affected patient or community were limited. However, with improvements in cancer diagnosis and interventions, there is renewed interest in increasing the timeliness of cancer surveillance systems so the data can be used to expedite the enrollment of eligible patients in clinical trials, to identify the inception of individual treatments or population-based interventions, and to monitor the effectiveness of the 68 Groseclose · Buckeridge Table 3 Attribute Public health surveillance system attributes and their measurementa Definition Willingness of persons and organizations to participate in the surveillance system. Measurement considerations May be indicated by the participation rate of data providers, completeness of case report data, timeliness or frequency of data reporting, or responsiveness to requests for supplemental information on reported health events. Acceptability can be influenced by the time and effort required to complete and submit surveillance reports or by perceptions of the benefits derived from participation in the surveillance system. Assessment of surveillance resources typically focuses on direct costs. Because of the complexity of surveillance and response processes, it is usually difficult to define indirect costs. For some infectious disease surveillance systems, investigators have modeled the expected future costs of strategies for continued vaccination, surveillance, and other public health activities (82, 83). Can be measured as the proportion of data intended to be collected that was actually collected (completeness) and the proportion of data entries that correctly reflect the true value of the data collected (validity). Includes proportion of unknown, invalid, and missing values for reported data elements. Validity may be estimated by the proportion of errors in surveillance system data compared to analogous data from one of the system’s data sources. Is best evaluated retrospectively by observing how the system has responded to new requirements or changes in surveillance processes or environment (e.g., new health-related events, case definition changes, modification of policies affecting the patient population eligible to receive care from participating reporting sources, or introduction of new information technologies). Cases reported to the system must be investigated to determine if they represent true or confirmed cases of the health event under surveillance. Low PVP may be addressed by case definition revision, request of additional data, or staff training to increase reporting accuracy. For event-based surveillance, criteria denoting a real outbreak should be defined and the outcomes of investigations of potential outbreaks identified by the system should be monitored and characterized per outbreak definition criteria. (Continued ) Acceptability Cost-effectiveness Relationship between the expected outcomes and the costs of surveillance to achieve these outcomes. Surveillance system costs include direct costs (personnel and material resources), indirect costs (resulting from preparedness and response to surveillance findings), and prevention benefits or costs from a societal perspective (e.g., effects of the information generated on decision making and population health). Completeness and validity of data in the surveillance system. Data quality Flexibility Ability of the surveillance system or its processes to adapt with little additional time or resources to changing epidemiologies, information needs, technologies, or clinical practices. Predictive value positive (PVP) Proportion of cases reported to the system that actually have the health condition under surveillance. For event-based surveillance, PVP represents the probability that a detected outbreak is of public health significance and requires response. www.annualreviews.org • Public Health Surveillance Systems Table 3 Attribute (Continued ) Definition Ability of the system to accurately describe the occurrence of a health condition under surveillance over time and its distribution in the population by place and person. Measurement considerations Representativeness is assessed by comparing the reported cases or events to all actual cases/events. Information on all cases/events is seldom available, but surveillance data representativeness can be described in terms of geographic coverage, demographic distribution, and clinical manifestation of the health conditions under surveillance that were reported to the surveillance system. For example, is there uneven distribution of reported cases/events based on our understanding of the epidemiology of the health condition in the population under surveillance? Surveillance system security policies and practices should be reviewed to ensure that security levels and procedures for surveillance system data or system access are defined and enforced; data use and release policy and protocol is available; and access to the surveillance system software application is controlled. Case detection sensitivity can be measured by the number of cases reported to the surveillance system divided by the number of cases in the population under surveillance. Outbreak detection sensitivity may be estimated retrospectively (e.g., were outbreak cases reported to the system? Was the temporal or spatial association of the cases noted prior to outbreak recognition from some other data source?) or prospectively (e.g., is an observed increase in case reports indicative of an outbreak requiring response?). Simulated or authentic data may be used to determine the characteristics of outbreaks that can be identified by the system (e.g., some absolute number of cases in time and space) or statistically derived thresholds (e.g., based on standard deviations). Sensitivity of the case definition depends on (a) the health condition under surveillance; (b) our knowledge of the epidemiology of the condition; (c) our ability to describe the condition based on clinical signs and symptoms, laboratory criteria, or epidemiological criteria; and (d ) the availability of relevant data elements in the surveillance system’s data sources. May be characterized in terms of (a) availability, amount, and type of data elements needed to characterize health events under surveillance; (b) number and type of organizations providing and using the data monitored; (c) number and type of data interchanges and transformations within the system; (d ) data provider and system operator training requirements; and (e) type of information technologies used by the system. (Continued ) Representativeness Security Processes and methodologies to keep surveillance data and information confidential, available, and accurate. Sensitivity Sensitivity of a surveillance system to (a) case detection (proportion of individuals who have the condition of interest), (b) outbreak detection (probability that the surveillance system will detect a significant increase in a health condition in time or space), and (c) case definition (ability of the case definition criteria to accurately represent the health condition of interest and classify the cases to which it is applied). Simplicity Structure and ease of operation of the system across the surveillance process cycle from data collection to use. Systems should be as simple as possible while still meeting their objectives. Groseclose · Buckeridge Table 3 Attribute Stability (Continued ) Definition Ability to collect, manage, and provide data without failure (reliability) and to be operational when needed (availability). Measurement considerations Measures for determining stability can include the number or duration of unscheduled outages of the information system(s) supporting the surveillance system; the comparison of the desired and actual amount of time or resources required for the system to collect, manage, analyze, interpret, and release data from the system; or the presence or absence of continuity of operations procedures intended to maintain system performance. Determine whether the system uses data standards (e.g., ICD, LOINC, or SNOMED)b as valid values for appropriate data variables or has the ability to translate its variable values to data standard concepts; determine whether the system uses standard data interchange protocols (e.g., Health Level 7) to exchange data with other information systems. For systems aiming for early detection of events of public health concern, timeliness assessment should focus on the detection of the hazard or agent causing the health condition or the identification of symptomatic individuals when they first seek care. For other systems, timeliness measures may indicate time to initiate interventions based on information derived from the system or time to accumulate sufficient information on which to develop risk communications or clinical guidance. Timeliness is influenced by surveillance methods and data source(s). Standards use Use of data exchange, messaging, or other information technology standards by a surveillance system that enhances the ability of the system and its software applications to communicate, exchange data, and use the information that has been exchanged. Time between any two steps in the surveillance process. Steps in the surveillance process will vary by system. The relative importance of timeliness of surveillance process intervals varies by surveillance objective and health event under surveillance. Timeliness Table adapted from References 21, 26, 27, 31, and 37. The International Classification of Disease (ICD) is the diagnostic classification standard for all clinical and research purposes and is maintained by the World Health Organization. The Logical Observations Identifiers Names and Codes (LOINC), produced by the Regenstrief Institute, is a clinical terminology for laboratory test orders and results. The Systematized Nomenclature of Medicine (SNOMED) is a systematic, computer-processable collection of medical terms in human and veterinary medicine that provides codes, terms, synonyms, and definitions covering anatomy, diseases, findings, procedures, microorganisms, and substances. b a interventions (60, 68). Investigators interested in early recognition and management of chronic diseases have improved diabetes case detection timeliness and sensitivity by developing and validating an automated, real-time diabetes case-finding algorithm using data extracted from a comprehensive electronic health record (53). Similar algorithms are being developed and used for electronic case reporting of notifiable infectious diseases in some jurisdictions to increase surveillance timeliness and the sensitivity of case detection (9, 49). Syndromic surveillance systems originally developed to support early detection of public health events are designed to alert at very low thresholds, which often results in high false-positive rates (i.e., low positive predictive value). Historically, practitioners have tried to improve the accuracy of syndromic surveillance systems by modifying statistical outbreak detection algorithms or by comparing results from different surveillance systems. Using methods borrowed from chronic disease surveillance, researchers have identified patient, physician, encounter, and billing characteristics (e.g., patient age, workload, and treating www.annualreviews.org • Public Health Surveillance Systems 71 physician’s billing history) that can be used to adjust analyses of syndromic surveillance data to reduce false positive alerts (19). Assessments of surveillance data quality have typically reported the proportion of selected data variables with missing or unknown data. Less frequently, investigators have attempted to validate the accuracy of the values reported for selected data variables. Two studies looking at the prevalence of cancers and sexually transmitted diseases among American Indians linked surveillance data to Indian Health Service registries to determine the accuracy of race and ethnicity data and the degree of misclassification (40, 86). In both studies, American Indian race was misclassified and underreported, resulting in underestimation of the burden of these health conditions among American Indians and limiting the usefulness of these data for monitoring progress towards addressing health disparities. An assessment of the quality of address data in reportable campylobacteriosis notifications in Montreal identified errors in 10% of reported addresses and missing addresses in 5% of the notifications (90). Address correction changed the case geolocation by a median of 1.1 km—which is significant if interventions are implemented based on neighborhood risk. Further evaluation of the impact of errors in public health data is warranted, focusing in particular on how these errors affect routine public health practice (e.g., geocoding for spatial monitoring or cluster detection or use of surveillance to direct and evaluate interventions or allocate resources). Everything done to influence surveillance processes or activities should support the objectives of the surveillance system being reviewed or evaluated. Use of new methods or tools should be pursued if they are likely to improve surveillance data quality or system performance (with regard to the system’s objectives). Investments in improvements to the performance of surveillance systems should connect to the “so what?” of public health; that is, new technologies or modified surveillance processes should support the generation and application of information for public health decision making and policy formulation. OPPORTUNITIES TO ENHANCE THE EFFECTIVENESS OF SURVEILLANCE SYSTEMS To make surveillance more scientific, one must encourage systematic observation and measurement, data collection, more precise analysis, and more effective data dissemination. In addition, both good science and good management require rigorous evaluation of current surveillance systems and alternative approaches to data collection, analysis, and dissemination. (85, p. 188) This statement is just as applicable today as it was at the time it was written in 1989. Opportunities to improve public health surveillance systems must take a principled approach that will generate evidence of the highest possible quality. Moreover, this evidence must be translated into practice to reduce unnecessary variation in the conduct of surveillance and to increase the efficiency and effectiveness of surveillance and of the public health system more generally. Doing so can be challenging for many reasons, including the increasingly multisectoral nature of surveillance as public health agencies engage with stakeholders to ensure public security and promote health. Despite these challenges, opportunities for enhancing surveillance practice exist in terms of increased scientific rigor, outcomes-focused research, and health informatics. Scientific Rigor and Outcomes in Research on Surveillance Systems Although surveillance can benefit from research in many different fields, research on surveillance is ultimately an applied science. It can be challenging to conduct applied research, because practical demands and resource constraints will frequently conflict with research activities. The 72 Groseclose · Buckeridge field of implementation science, however, can provide helpful guidance in this regard (51). For example, innovations in surveillance systems could be introduced in a staged manner across public health departments using a step-wedged cluster-randomized design (36) to minimize bias in the assessment of the effect of system changes. Given the many components of a surveillance system, it is perhaps not surprising that much of the research in this field has focused on evaluating or improving specific components of the surveillance process as opposed to evaluating the overall impact of a system. For example, the last decade has seen a considerable amount of research on the statistical detection of aberrations in surveillance data (15, 50), but most of these studies have focused on the statistical properties of detection algorithms. With few exceptions, researchers have tended not to examine the effect of detection algorithms on outcomes, such as the time to initiating a response (17) or the number of infections averted (46). A greater focus on outcomes, however, could provide a clearer picture of which enhancements to surveillance systems are likely to make a difference in population health and should be implemented. Measurement of costs and cost-effectiveness is another opportunity to enhance surveillance systems. Although a cost-effectiveness perspective is readily accepted in the evaluation of new clinical interventions, few evaluations of surveillance system modifications have taken this approach (83). For example, public health agencies are frequently faced with the question of whether they should use newly available data streams or adopt new analytical methods. Although it is important to conduct research that characterizes a novel data source or method, the insights from such evaluations do not fully address the question of whether the new procedure should be adopted in practice. Quantifying the costs and the effectiveness of innovations in public health surveillance systems would provide decision makers with highly relevant evidence to guide the allocation of resources. The Role of Informatics in Surveillance Systems Public health informatics has grown rapidly as a discipline in the past two decades (30, 89), and surveillance in particular has been the subject of many studies in the informatics literature (47). Informatics research has helped to advance surveillance systems in many ways, including by facilitating access to new data streams from clinical and other sources (16, 29, 43, 52); automating surveillance processes, such as case detection from free text (22) and aberration detection (38); and enabling the rapid dissemination of information produced by surveillance to a wide range of stakeholders (20). Building on these advances, many opportunities exist for health informatics methods to enhance surveillance systems. One opportunity is the development of controlled terminologies and ontologies for public health surveillance and public health practice in general. Scientific advancement in any field requires clarity and precision in measurement and communication. The establishment of common terminologies can facilitate comparative research and the sharing of software, including the sharing of surveillance software across public health agencies. An Australian group of researchers has taken a first step in this direction by developing a taxonomy of public health actions including surveillance (41), and other researchers have built upon this model and used it as the core of a chronic disease surveillance system (59). Similar work has developed shareable models to guide the conduct of reportable disease surveillance (25). This type of work provides a scalable and open foundation for ongoing improvements in public health surveillance, as it can support the sharing, reuse, and refinement of software systems and public health knowledge. Informatics also offers the opportunity to enhance public health surveillance by enabling new ways to engage the public. For example, some researchers have demonstrated how members of the general public can be recruited to receive messages by email or text routinely asking them to www.annualreviews.org • Public Health Surveillance Systems 73 report the onset of symptoms, such as influenza-like illness (80, 87). Surveillance systems based on participatory cohorts provide information similar to that obtained through sentinel surveillance (4), and they address some of the limitations of surveillance methods based on telephone survey, which may not adequately sample populations without telephone lines (44, 55). Participatory cohorts, however, are subject to sampling bias, and loss of cohort members to attrition can pose a challenge. Another novel use of informatics to enhance surveillance is through social media. Much of the research on this approach has focused on passive monitoring of social media data, such as Twitter, to measure disease activity (76), health behaviors (57), or adverse events (65). In general, evaluations suggest that this approach can produce information that can add value to other surveillance strategies (73); however, the data may not be representative or may lack demographic and geographical details that are important for some surveillance objectives. These limitations are offset to some extent by the ability to use social media not just to gather data, but also to engage individuals, for example by detecting comments about food poisoning and encouraging people to contact a public health agency to file a report (34). CONCLUSION Surveillance is critical for public health, and surveillance systems are needed to conduct surveillance efficiently and effectively. The main output of surveillance systems is information that drives action; consequently, systems require data of sufficient quality and with a resolution and timeliness that match public health objectives. In many ways, surveillance presents a working model of what has been called a “learning health system” (7). The repeated cycle of observation, data analysis to identify opportunities for improvement, and implementation of changes provides a means for rapidly converting data into actionable information to improve population health. Surveillance and the systems that enable surveillance have been the focus of applied research in public health for decades; in recent years, however, there has been an increasing interest in surveillance methods in the context of changing public health objectives and emerging opportunities for data access and computing. Surveillance systems are now applied broadly across many areas beyond infectious diseases, and this increasing breadth of application, along with the associated methodological demands (e.g., rapid signal detection, data integration), is driving innovation and contributing to the scientific basis of public health surveillance. Scholars have emphasized the importance of evaluating key attributes of surveillance systems for decades; today, they increasingly recognize the routine monitoring of key system processes and outcomes as needed to aid data interpretation, ensure quality, and guide focused evaluation activities. Future improvements to surveillance systems must be driven by public health objectives and should be evaluated rigorously to determine their effects and, ideally, costs. We encourage public health surveillance practitioners to focus on opportunities to apply the data in new ways (as reflected in the system’s objectives); to embrace the possibilities that informatics offers to enhance data quality and system efficiency and effectiveness; to incorporate routine monitoring of system performance; and to apply sound study designs to evaluate the effect of innovations on the outcomes of interest. This renewed approach to surveillance systems will be instrumental in improving public health practice and enhancing community health. DISCLOSURE STATEMENT The authors are not aware of any affiliations, memberships, funding, or financial holdings that might be perceived as affecting the objectivity of this review. The findings and conclusions in this 74 Groseclose · Buckeridge review are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention. LITERATURE CITED 1. Althouse BM, Ng YY, Cummings DAT. 2011. Prediction of dengue incidence using search query surveillance. PLOS Negl. Trop. Dis. 5:e1258 2. Althouse BM, Scarpino SV, Meyers LA, Ayers JW, Bargsten M, et al. 2015. Enhancing disease surveillance with novel data streams: challenges and opportunities. EPJ Data Sci. 4:17. https://doi.org/ 10.1140/epjds/s13688-015-0054-0 3. Anderson BD, Ma M, Xia Y, Wang T, Shu B, et al. 2016. Bioaerosol sampling in modern agriculture: a novel approach for emerging pathogen surveillance? J. Infect. Dis. 214:537–45. https://doi.org/ 10.1093/infdis/jiw180 4. Bajardi P, Vespignani A, Funk S, Eames KT, Edmunds WJ, et al. 2014. Determinants of follow-up participation in the Internet-based European influenza surveillance platform Influenzanet. J. Med. Internet Res. 16:e78. https://doi.org/10.2196/jmir.3010 5. Baker MA, Nguyen M, Cole DV, Lee GM, Lieu TA. 2013. Post-licensure rapid immunization safety monitoring program (PRISM) data characterization. Vaccine 31(Suppl.):K98–112 6. Bednarczyk RA, Orenstein WA, Omer SB. 2016. Estimating the number of measles-susceptible children and adolescents in the United States using data from the National Immunization Survey-Teen (NISTeen). Am. J. Epidemiol. 184:148–56. https://doi.org/10.1093/aje/kwv320 7. Bernstein JA, Friedman C, Jacobson P, Rubin JC. 2015. Ensuring public health’s future in a national-scale learning health system. Am. J. Prev. Med. 48: 480–87 8. Biggerstaff M, Alper D, Dredze M, Fox S, Fung IC, et al. 2016. Results from the Centers for Disease Control and Prevention’s Predict the 2013–2014 Influenza Season Challenge. BMC Infect. Dis. 16:357. https://doi.org/10.1186/s12879-016-1669-x 9. Birkhead GS, Klompas M, Shah NR. 2015. Uses of electronic health records for public health surveillance to advance public health. Annu. Rev. Public Health 36: 345–59 10. Blondin KJ, Giles CM, Cradock AL, Gortmaker SL, Long MW. 2016. US state childhood obesity surveillance practices and recommendations for improving them, 2014–2015. Prev. Chronic Dis. 13:160060. https://doi.org/10.5888/pcd13.160060 11. Bowen MD, Mijatovic-Rustempasic S, Esona MD, Teel EN, Gautam R, et al. 2016. Rotavirus strain trends during the postlicensure vaccine era: United States, 2008–2013. J. Infect. Dis. 214:732–38. https://doi.org/10.1093/infdis/jiw233 12. Broniatowski DA, Paul MJ, Dredze M. 2013. National and local influenza surveillance through Twitter: an analysis of the 2012–2013 influenza epidemic. PLOS ONE 8:e83672. https://doi.org/ 10.1371/journal.pone.0083672 13. Brownson RC, Fielding JE, Maylahn CM. 2009. Evidence-based public health: a fundamental concept for public health practice. Annu. Rev. Public Health 30:175–201 14. Buckeridge DL. 2007. Outbreak detection through automated surveillance: a review of the determinants of detection. J. Biomed. Inform. 40:370–79 15. Buckeridge DL, Burkom H, Campbell M, Hogan WR, Moore AW. 2005. Algorithms for rapid outbreak detection: a research synthesis. J. Biomed. Inform. 38:99–113 16. Buckeridge DL, Charland K, Labban A, Ma Y. 2014. A method for neighborhood-level surveillance of food purchasing. Ann. N. Y. Acad. Sci. 1331:270–77. https://doi.org/10.1111/nyas.12332 17. Buckeridge DL, Owens DK, Switzer P, Frank J, Musen MA. 2006. Evaluating detection of an inhalational anthrax outbreak. Emerg. Infect. Dis. 12:1942–49 18. Burris S, Hitchcock L, Ibrahim J, Penn M, Ramanathan T. 2016. Policy surveillance: a vital public health practice comes of age. J. Health Polit. Policy Law. In press. https://doi.org/10.1215/03616878–3665931 19. Cadieux G, Buckeridge DL, Jacques A, Libman M, Dendukuri N, Tamblyn R. 2012. Patient, physician, encounter, and billing characteristics predict the accuracy of syndromic surveillance case definitions. BMC Public Health 12:166 www.annualreviews.org • Public Health Surveillance Systems 75 20. Carroll LN, Au AP, Detwiler LT, Fu T, Painter IS, Abernethy NF. 2014. Visualization and analytics tools for infectious disease epidemiology: a systemic review. J. Biomed. Inform. 51:287–98 21. CDC (Cent. Dis. Control Prev.). 2001. Updated guidelines for evaluating public health surveillance systems: recommendations from the guidelines working group. Morb. Mortal. Wkly. Rep. Recomm. Rep. 50(RR-13):1–35 22. Chapman WW, Dowling JN, Wagner MM. 2005. Classification of emergency department chief complaints into 7 syndromes: a retrospective analysis of 527,228 patients. Ann. Emerg. Med. 46:445–55 23. Chriqui JF, O’Connor JC, Chaloupka FJ. 2011. What gets measured, gets changed: evaluating law and policy for maximum impact. J. Law Med. Ethics 39(Suppl. 1):21–26 24. Condell O, Midgley S, Christiansen CB, Chen M, Chen Nielsen X, et al. 2016. Evaluation of the enterovirus laboratory surveillance system in Denmark, 2010 to 2013. Euro Surveill. 21(18):30218. https://doi.org/10.2807/1560–7917.ES.2016.21.18.30218 25. Doyle TJ, Ma H, Groseclose SL, Hopkins RS. 2005. PHSkb: a knowledgebase to support notifiable disease surveillance. BMC Med. Inform. Decis. Mak. 5:27 26. Drewe JA, Hoinville LJ, Cook AJC, Floyd T, Stark KDC. 2012. Evaluation of animal and public health surveillance systems: a systematic review. Epidemiol. Infect. 140:575–90 27. Eur. Cent. Dis. Prev. Control. 2014. Data Quality Monitoring and Surveillance System Evaluation: A Handbook of Methods and Applications. Stockholm, Swed.: Eur. Cent. Dis. Prev. Control. http://ecdc.europa. eu/en/publications/publications/data-quality-monitoring-surveillance-system-evaluation-sept2014.pdf 28. Foege WH, Hogan RC, Newton LH. 1976. Surveillance projects for selected diseases. Int. J. Epidemiol. 5(1):29–37 29. Freifeld CC, Mandl KD, Reis BY, Brownstein JS. 2008. HealthMap: global infectious disease monitoring through automated classification and visualization of Internet media reports. J. Am. Med. Inform. Assoc. 15:150–57 30. Friede A, Blum HL, McDonald M. 1995. Public health informatics: how information-age technology can strengthen public health. Annu. Rev. Public Health 16:239–52 31. Groseclose SL, German RR, Nsubuga P. 2010. Evaluating public health surveillance. In Principles and Practice of Public Health Surveillance, ed. LM Lee, SM Teutsch, ME St. Louis, SB Thacker, pp. 166–97. New York: Oxford Univ. Press. 3rd ed. 32. Hall HI, Mokotoff ED. 2007. Setting standards and an evaluation framework for human immunodeficiency virus/acquired immunodeficiency syndrome surveillance. J. Public Health Manag. Pract. 13:519–23 33. Hall HI, Song R, Gerstle JE III, Lee LM. 2006. Assessing the completeness of reporting of human immunodeficiency virus diagnoses in 2002–2003: capture-recapture methods. Am. J. Epidemiol. 164:391– 97 34. Harris JK, Mansour R, Choucair B, Olson J, Nissen C, Bhatt J. 2014. Health department use of social media to identify foodborne illness—Chicago, Illinois, 2013–2014. Morb. Mortal. Wkly. Rep. 63:681–85 35. Harrison C, Jorder M, Stern H, Stavinsky F, Reddy V, et al. 2014. Using online reviews by restaurant patrons to identify unreported cases of foodborne illness—New York City, 2012–2013. Morb. Mortal. Wkly. Rep. 63:441–45 36. Hemming K, Lilford R, Girling AJ. 2015. Stepped-wedge cluster randomised controlled trials: a generic framework including parallel and multiple-level designs. Stat. Med. 34:181–96. https://doi.org/ 10.1002/sim.6325 37. Hopkins RS. 2005. Design and operation of state and local infectious disease surveillance systems. J. Public Health Manag. Pract. 11:184–90 38. Hulth A, Andrews N, Ethelberg S, Dreesman J, Faensen D, et al. 2010. Practical usage of computersupported outbreak detection in five European countries. Euro Surveill. 15(36):19658 39. Inst. Med. 1988. The Future of Public Health. Washington, DC: Natl. Acad. Press. doi:10.17226/1091 40. Jim MA, Arias E, Seneca DS, Hoopes MJ, Jim CC, et al. 2014. Racial misclassification of American Indians and Alaska Natives by Indian Health Service contract health service delivery area. Am. J. Public Health 104:S295–302. https://doi.org/10.2105/AJPH.2014.301933 41. Jorm L, Gruszin S, Churches T. 2009. A multidimensional classification of public health activity in Australia. Aust. N. Z. Health Policy 6:9. https://doi.org/10.1186/1743-8462-6-9 76 Groseclose · Buckeridge 42. Karch DL, Chen M, Tang T. 2014. Evaluation of the National Human Immunodeficiency Virus Surveillance System for the 2011 diagnosis year. J. Public Health Manag. Pract. 20:598–607 43. Kass-Hout TA, Xu Z, Mohebbi M, Nelsen H, Baker A, et al. 2016. OpenFDA: an innovative platform providing access to a wealth of FDA’s publicly available data. J. Am. Med. Inform. Assoc. 23:596–600. https://doi.org/10.1093/jamia/ocv153 44. Kempf AM, Remington PL. 2007. New challenges for telephone survey research in the twenty-first century. Annu. Rev. Public Health 28:113–26 45. Kirkcaldy RD, Harvey A, Papp JR, del Rio C, Soge OO, et al. 2016. Neisseria gonorrhoeae antimicrobial susceptibility surveillance—the Gonococcal Isolate Surveillance Project, 27 sites, United States, 2014. Morb. Mortal. Wkly. Rep. Surveill. Summ. 65(7):1–19 46. Kleinman KP, Abrams AM. 2008. Assessing the utility of public health surveillance using specificity, sensitivity, and lives saved. Stat. Med. 27:4057–68 47. Kukafka R, Yasnoff WA. 2007. Public health informatics. J. Biomed. Inform. 40:365–69 48. Langley G, Schaffner W, Farley MM, Lynfield R, Bennett NM, et al. 2015. Twenty years of Active Bacterial Core surveillance. Emerg. Infect. Dis. 21:1520–28 49. Lazarus R, Klompas M, Campion FX, NcNabb SJN, Hou X, et al. 2009. Electronic support for public health: validated case finding and reporting for notifiable diseases using electronic medical data. J. Am. Med. Inform. Assoc. 16:18–24. https://doi.org/10.1197/jamia.M2848.a 50. Lewis B, Eubank S, Abrams AM, Kleinman K. 2013. In silico surveillance: evaluating outbreak detection with simulation models. BMC Med. Inform. Decis. Mak. 13:12 51. Lobb R, Colditz GA. 2013. Implementation science and its application to population health. Annu. Rev. Public Health 34:235–51 52. Ma H, Rolka H, Mandl K, Buckeridge D, Fleischauer A, Pavlin J. 2005. Implementation of laboratory order data in BioSense Early Event Detection and Situation Awareness System. Morb. Mortal. Wkly. Rep. 54(Suppl.):27–30 53. Makam AN, Nguyen OK, Moore B, Ma Y, Amarasingham R. 2013. Identifying patients with diabetes and the earliest data of diagnosis in real time: an electronic health record case-finding algorithm. BMC Med. Inform. Decis. Mak. 13:81 54. McNabb SJN, Surdo AM, Redmond A, Cobb J, Wiley J, et al. 2004. Applying a new conceptual framework to evaluate tuberculosis surveillance and action performance and measure the costs, Hillsborough County, Florida, 2002. Ann. Epidemiol. 14:640–45 55. Mokdad AH. 2009. The Behavioral Risk Factor Surveillance System: past, present, and future. Annu. Rev. Public Health 30:43–54 56. Morrison KT, Shaddick G, Henderson SB, Buckeridge DL. 2016. A latent process model for forecasting multiple time series in environmental public health surveillance. Stat. Med. 35:3085–100 57. Mysl´n M, Zhu SH, Chapman W, Conway M. 2013. Using Twitter to examine smoking behavior and ı perceptions of emerging tobacco products. J. Med. Internet Res. 15(8):e174 58. Nsoesie EO, Buckeridge DL, Brownstein JS. 2014. Guess who’s not coming to dinner? Evaluating online restaurant reservations for disease surveillance. J. Med. Internet Res. 16(1):e22 59. Okhmatovskaia A, Shaban-Nejad A, Lavigne M, Buckeridge DL. 2014. Addressing the challenge of encoding causal epidemiological knowledge in formal ontologies: a practical perspective. Stud. Health Technol. Inform. 205:1125–29 60. Osborne JD, Wyatt M, Westfall AO, Willig J, Bethard S, Gordon G. 2016. Efficient identification of nationally mandated reportable cancer cases using natural language processing and machine learning. J. Am. Med. Inform. Assoc. In press. https://doi.org/10.1093/jamia/ocw006 61. Parton HB, Mathes R, Abdelnabi J, Alleyne L, Econome A, et al. 2016. Investigating a syndromic surveillance signal with complementary data systems. Online J. Public Health Inform. 8(1):e152 62. Patwardhan A, Bilkovski R. 2012. Comparison: flu prescription sales data from a retail pharmacy in the US with Google flu trends and US ILINet (CDC) data as flu activity indicator. PLOS ONE 7(8):e43611 63. Pinner RW, Lynfield R, Hadler JL, Schaffner W, Farley MM, et al. 2015. Cultivation of an adaptive domestic network for surveillance and evaluation of emerging infections. Emerg. Infect. Dis. 21:1499–509 64. Pollack KM, D’Angelo J, Green G, Conte S, Fealy S, et al. 2016. Developing and implementing Major League Baseball’s health and injury tracking system. Am. J. Epidemiol. 183:490–96 www.annualreviews.org • Public Health Surveillance Systems 77 65. Powell GE, Seifert HA, Reblin T, Burstein PJ, Blowers J, et al. 2016. Social media listening for routine post-marketing safety surveillance. Drug Saf. 39:443–54 66. Presley D, Reinstein T, Webb-Barr D, Burris S. 2015. Creating legal data for public health monitoring and evaluation: Delphi standards for policy surveillance. J. Law Med. Ethics 43:27–31 67. Prue CE, Lackey C, Swenarski L, Gantt JM. 2003. Communication monitoring: shaping CDC’s emergency risk communication efforts. J. Health Commun. 8(Suppl. 1):35–49 68. Puckett M, Neri A, Rohan E, Clerkin C, Underwood JM, et al. 2016. Evaluating early case capture of pediatric cancers in seven central cancer registries in the United States, 2013. Public Health Rep. 131:126–36 69. Roush SW. 2011. Surveillance indicators. In Manual for the Surveillance of Vaccine-Preventable Diseases, ed. Cent. Dis. Control Prev. Atlanta, GA: Cent. Dis. Control Prev. 5th ed. http://www.cdc.gov/ vaccines/pubs/surv-manual/chpt18-surv-indicators.pdf 70. Sabbe M, Berger N, Blommaert A, Ogunjimi B, Grammens T, et al. 2016. Sustained low rotavirus activity and hospitalization rates in the post-vaccination era in Belgium, 2007 to 2014. Euro Surveill. 21(27):30273 71. Salathe M, Friefeld CC, Mekaru SR, Tomasulo AF, Brownstein JS. 2013. Influenza A (H7N9) and the importance of digital epidemiology. N. Engl. J. Med. 369:401–4 72. Salmon D, Yih WK, Lee G, Rosofsky R, Brown J, et al. 2012. Success of program linking data sources to monitor H1N1 vaccine safety points to potential for even broader safety surveillance. Health Aff. 31:2518–27 73. Santillana M, Nguyen AT, Dredze M, Paul MJ, Nsoesie EO, Brownstein JS. 2015. Combining search, social media, and traditional data sources to improve influenza surveillance. PLOS Comput. Biol. 11:e1004513 74. Savard N, B´ dard L, Allard R, Buckeridge DL. 2015. Using age, triage score, and disposition data from e emergency department electronic records to improve Influenza-like illness surveillance. J. Am. Med. Inform. Assoc. 22:688–96 75. Shaban-Nejad A, Lavigne M, Okhmatovskaia A, Buckeridge DL. 2016. PopHR: a knowledge-based platform to support integration, analysis and visualization of population health data. Ann. N. Y. Acad. Sci. In press 76. Signorini A, Segre AM, Polgreen PM. 2011. The use of Twitter to track levels of disease activity and public concern in the U.S. during the Influenza A H1N1 pandemic. PLOS ONE 6:e19467 77. Skoog G, Struwe J, Cars O, Hangerger H, Odenholt I, et al. 2016. Repeated nationwide point-prevalence surveys for antimicrobial use in Swedish hospitals: data for actions 2003–2010. Euro Surveill. 21:30264 78. Smith GE, Elliot AJ, Ibbotson S, Morbey R, Edeghere O, et al. 2016. Novel public health risk assessment process developed to support syndromic surveillance for the 2012 Olympic and Paralympic Games. J. Public Health. In press. https://doi.org/10.1093/pubmed/fdw054 79. Smith PF, Hadler JL, Stanbury M, Rolfs RT, Hopkins RS. 2013. “Blueprint Version 2.0”: updating public health surveillance for the 21st century. J. Public Health Manag. Pract. 19:231–39 80. Smolinski MS, Crawley AW, Baltrusaitis K, Chunara R, Olsen JM, et al. 2015. Flu Near You: crowdsourced symptom reporting spanning 2 influenza seasons. Am. J. Public Health 105:2124–30 81. Snider CJ, Diop OM, Tangermann RH, Wassilek SGF. 2016. Surveillance systems to track progress toward polio eradication—worldwide, 2014–2015. Morb. Mortal. Wkly. Rep. 65: 346–51 82. Somda ZC, Perry HN, Messonnier NR, Djingarey MH, Ki SO, Meltzer MI. 2010. Modeling the costeffectiveness of the Integrated Disease Surveillance and Response (ISDR) system: meningitis in Burkina Faso. PLOS ONE 5(9):e13044 83. Tebbens RJD, Sangrujee N, Thompson KM. 2006. The costs of future polio risk management policies. Risk Anal. 26:1507–31 84. Thacker SB, Berkelman RL. 1988. Public health surveillance in the United States. Epidemiol. Rev. 10:164– 90 85. Thacker SB, Berkelman RL, Stroup DF. 1989. The science of public health surveillance. J. Public Health Policy 10:187–203 86. Thoroughman DA, Frederickson D, Cameron HD, Shelby LK, Cheek JE. 2002. Racial misclassification of American Indians in Oklahoma state surveillance data for sexually transmitted diseases. Am. J. Epidemiol. 155:1137–41 78 Groseclose · Buckeridge 87. van Noort SP, Codeco CT, Koppeschaar CE, van Ranst M, Paolotti D, Gomes MG. 2015. Ten-year performance of Influenzanet: ILI time series, risks, vaccine effects, and care-seeking behaviour. Epidemics 13:28–36 88. Wong WK, Moore A, Cooper G, Wagner MM. 2003. WSARE: What’s Strange About Recent Events? J. Urban Health 80(Suppl. 1):i66–75 89. Yasnoff WA, O’Carroll PW, Koo D, Linkins RW, Kilbourne EM. 2000. Public health informatics: improving and transforming public health in the information age. J. Public Health Manag. Pract. 6:67–75 90. Zinszer K, Jauvin C, Verma A, Bedard L, Allard R, et al. 2010. Residential address errors in public health surveillance data: a description and analysis of the impact on geocoding. Spat. Spatiotemporal Epidemiol. 1(2–3):163–68 www.annualreviews.org • Public Health Surveillance Systems

Journal

Annual Review of Public Health – Annual Reviews

Published: Mar 20, 2017

There are no references for this article.