Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

The PRISMA Statement for Reporting Systematic Reviews and Meta-Analyses of Studies That Evaluate Health Care Interventions: Explanation and Elaboration

The PRISMA Statement for Reporting Systematic Reviews and Meta-Analyses of Studies That Evaluate... Guidelines and Guidance The PRISMA Statement for Reporting Systematic Reviews and Meta-Analyses of Studies That Evaluate Health Care Interventions: Explanation and Elaboration 1,2 3 4 5 6 Alessandro Liberati *, Douglas G. Altman , Jennifer Tetzlaff , Cynthia Mulrow , Peter C. Gøtzsche , 7 8,9 10 11,12 4,13 John P. A. Ioannidis , Mike Clarke , P. J. Devereaux , Jos Kleijnen , David Moher 1 Universita` di Modena e Reggio Emilia, Modena, Italy, 2 Centro Cochrane Italiano, Istituto Ricerche Farmacologiche Mario Negri, Milan, Italy, 3 Centre for Statistics in Medicine, University of Oxford, Oxford, United Kingdom, 4 Ottawa Methods Centre, Ottawa Hospital Research Institute, Ottawa, Ontario, Canada, 5 Annals of Internal Medicine, Philadelphia, Pennsylvania, United States of America, 6 The Nordic Cochrane Centre, Copenhagen, Denmark, 7 Department of Hygiene and Epidemiology, University of Ioannina School of Medicine, Ioannina, Greece, 8 UK Cochrane Centre, Oxford, United Kingdom, 9 School of Nursing and Midwifery, Trinity College, Dublin, Ireland, 10 Departments of Medicine, Clinical Epidemiology and Biostatistics, McMaster University, Hamilton, Ontario, Canada, 11 Kleijnen Systematic Reviews Ltd, York, United Kingdom, 12 School for Public Health and Primary Care (CAPHRI), University of Maastricht, Maastricht, The Netherlands, 13 Department of Epidemiology and Community Medicine, Faculty of Medicine, Ottawa, Ontario, Canada Recent data suggest that at least 2,500 new systematic reviews Abstract: Systematic reviews and meta-analyses are reported in English are indexed in MEDLINE annually [3]. essential to summarize evidence relating to efficacy and Unfortunately, there is considerable evidence that key informa- safety of health care interventions accurately and reliably. tion is often poorly reported in systematic reviews, thus The clarity and transparency of these reports, however, is diminishing their potential usefulness [3,4,5,6]. As is true for all not optimal. Poor reporting of systematic reviews research, systematic reviews should be reported fully and diminishes their value to clinicians, policy makers, and transparently to allow readers to assess the strengths and other users. Since the development of the QUOROM weaknesses of the investigation [7]. That rationale led to the (QUality Of Reporting Of Meta-analysis) Statement—a development of the QUOROM (QUality Of Reporting Of Meta- reporting guideline published in 1999—there have been analyses) Statement; those detailed reporting recommendations several conceptual, methodological, and practical advanc- were published in 1999 [8]. In this paper we describe the updating es regarding the conduct and reporting of systematic reviews and meta-analyses. Also, reviews of published systematic reviews have found that key information about these studies is often poorly reported. Realizing these Citation: Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gøtzsche PC, et al. (2009) The issues, an international group that included experienced PRISMA Statement for Reporting Systematic Reviews and Meta-Analyses of authors and methodologists developed PRISMA (Preferred Studies That Evaluate Health Care Interventions: Explanation and Elaboration. PLoS Med 6(7): e1000100. doi:10.1371/journal.pmed.1000100 Reporting Items for Systematic reviews and Meta-Analy- ses) as an evolution of the original QUOROM guideline for Published July 21, 2009 systematic reviews and meta-analyses of evaluations of Copyright:  2009 Liberati et al. This is an open-access article distributed health care interventions. The PRISMA Statement con- under the terms of the Creative Commons Attribution License, which permits sists of a 27-item checklist and a four-phase flow diagram. unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. The checklist includes items deemed essential for transparent reporting of a systematic review. In this Funding: PRISMA was funded by the Canadian Institutes of Health Research; Universita di Modena e Reggio Emilia, Italy; Cancer Research UK; Clinical Evidence Explanation and Elaboration document, we explain the BMJ Knowledge; The Cochrane Collaboration; and GlaxoSmithKline, Canada. AL is meaning and rationale for each checklist item. For each funded, in part, through grants of the Italian Ministry of University (COFIN - PRIN item, we include an example of good reporting and, 2002 prot. 2002061749 and COFIN - PRIN 2006 prot. 2006062298). DGA is funded where possible, references to relevant empirical studies by Cancer Research UK. DM is funded by a University of Ottawa Research Chair. None of the sponsors had any involvement in the planning, execution, or write-up and methodological literature. The PRISMA Statement, of the PRISMA documents. Additionally, no funder played a role in drafting the this document, and the associated Web site (http://www. manuscript. prisma-statement.org/) should be helpful resources to Competing Interests: MC’s employment is as Director of the UK Cochrane improve reporting of systematic reviews and meta- Centre. He is employed by the Oxford Radcliffe Hospitals Trust on behalf of the analyses. Department of Health and the National Institute for Health Research in England. This is a fixed term contract, the renewal of which is dependent upon the value placed upon his work, that of the UK Cochrane Centre, and of The Cochrane Collaboration more widely by the Department of Health. His work involves the conduct of systematic reviews and the support of the conduct and use of Introduction systematic reviews. Therefore, work–such as this manuscript–relating to systematic reviews might have an impact on his employment. Systematic reviews and meta-analyses are essential tools for summarizing evidence accurately and reliably. They help Abbreviations: PICOS, participants, interventions, comparators, outcomes, and study design; PRISMA, Preferred Reporting Items for Systematic reviews and clinicians keep up-to-date; provide evidence for policy makers to Meta-Analyses; QUOROM, QUality Of Reporting Of Meta-analyses. judge risks, benefits, and harms of health care behaviors and * E-mail: alesslib@mailbase.it interventions; gather together and summarize related research for Provenance: Not commissioned; externally peer reviewed. In order to patients and their carers; provide a starting point for clinical encourage dissemination of the PRISMA explanatory paper, this article is freely practice guideline developers; provide summaries of previous accessible on the PLoS Medicine, Annals of Internal Medicine,and BMJ Web sites. research for funders wishing to support new research [1]; and help The authors jointly hold the copyright of this article. For details on further use see editors judge the merits of publishing reports of new studies [2]. the PRISMA Web site (http://www.prisma-statement.org/). PLoS Medicine | www.plosmedicine.org 1 July 2009 | Volume 6 | Issue 7 | e1000100 of that guidance. Our aim is to ensure clear presentation of what Box 1. Terminology was planned, done, and found in a systematic review. The terminology used to describe systematic reviews and Terminology used to describe systematic reviews and meta- meta-analyses has evolved over time and varies between analyses has evolved over time and varies across different groups of fields. Different terms have been used by different groups, researchers and authors (see Box 1). In this document we adopt the such as educators and psychologists. The conduct of a definitions used by the Cochrane Collaboration [9]. A systematic systematic review comprises several explicit and repro- review attempts to collate all empirical evidence that fits pre- ducible steps, such as identifying all likely relevant records, specified eligibility criteria to answer a specific research question. selecting eligible studies, assessing the risk of bias, It uses explicit, systematic methods that are selected to minimize extracting data, qualitative synthesis of the included bias, thus providing reliable findings from which conclusions can studies, and possibly meta-analyses. be drawn and decisions made. Meta-analysis is the use of statistical Initially this entire process was termed a meta-analysis methods to summarize and combine the results of independent and was so defined in the QUOROM Statement [8]. More studies. Many systematic reviews contain meta-analyses, but not recently, especially in health care research, there has been a all. trend towards preferring the term systematic review. If quantitative synthesis is performed, this last stage alone is The QUOROM Statement and Its Evolution into referred to as a meta-analysis. The Cochrane Collaboration uses this terminology [9], under which a meta-analysis, if PRISMA performed, is a component of a systematic review. The QUOROM Statement, developed in 1996 and published Regardless of the question addressed and the complexities in 1999 [8], was conceived as a reporting guidance for authors involved, it is always possible to complete a systematic reporting a meta-analysis of randomized trials. Since then, much review of existing data, but not always possible, or has happened. First, knowledge about the conduct and reporting desirable, to quantitatively synthesize results, due to clinical, methodological, or statistical differences across the includ- of systematic reviews has expanded considerably. For example, ed studies. Conversely, with prospective accumulation of The Cochrane Library’s Methodology Register (which includes studies and datasets where the plan is eventually to reports of studies relevant to the methods for systematic reviews) combine them, the term ‘‘(prospective) meta-analysis’’ now contains more than 11,000 entries (March 2009). Second, may make more sense than ‘‘systematic review.’’ there have been many conceptual advances, such as ‘‘outcome- For retrospective efforts, one possibility is to use the level’’ assessments of the risk of bias [10,11], that apply to term systematic review for the whole process up to the systematic reviews. Third, authors have increasingly used point when one decides whether to perform a quantitative systematic reviews to summarize evidence other than that synthesis. If a quantitative synthesis is performed, some provided by randomized trials. researchers refer to this as a meta-analysis. This definition However, despite advances, the quality of the conduct and is similar to that found in the current edition of the reporting of systematic reviews remains well short of ideal Dictionary of Epidemiology [183]. [3,4,5,6]. All of these issues prompted the need for an update While we recognize that the use of these terms is and expansion of the QUOROM Statement. Of note, recognizing inconsistent and there is residual disagreement among the that the updated statement now addresses the above conceptual members of the panel working on PRISMA, we have adopted the definitions used by the Cochrane Collaboration [9]. and methodological issues and may also have broader applicability Systematic review: A systematic review attempts to than the original QUOROM Statement, we changed the name of collate all empirical evidence that fits pre-specified the reporting guidance to PRISMA (Preferred Reporting Items for eligibility criteria to answer a specific research question. Systematic reviews and Meta-Analyses). It uses explicit, systematic methods that are selected with a view to minimizing bias, thus providing reliable findings Development of PRISMA from which conclusions can be drawn and decisions made [184,185]. The key characteristics of a systematic review The PRISMA Statement was developed by a group of 29 review are: (a) a clearly stated set of objectives with an explicit, authors, methodologists, clinicians, medical editors, and consum- reproducible methodology; (b) a systematic search that ers [12]. They attended a three-day meeting in 2005 and attempts to identify all studies that would meet the participated in extensive post-meeting electronic correspondence. eligibility criteria; (c) an assessment of the validity of the A consensus process that was informed by evidence, whenever findings of the included studies, for example through the possible, was used to develop a 27-item checklist (Table 1; see also assessment of risk of bias; and (d) systematic presentation, Text S1 for a downloadable template checklist for researchers to and synthesis, of the characteristics and findings of the re-use) and a four-phase flow diagram (Figure 1; see Figure S1 for included studies. a downloadable template document for researchers to re-use). Meta-analysis: Meta-analysis is the use of statistical Items deemed essential for transparent reporting of a systematic techniques to integrate and summarize the results of review were included in the checklist. The flow diagram originally included studies. Many systematic reviews contain meta- proposed by QUOROM was also modified to show numbers of analyses, but not all. By combining information from all identified records, excluded articles, and included studies. After 11 relevant studies, meta-analyses can provide more precise estimates of the effects of health care than those derived revisions the group approved the checklist, flow diagram, and this from the individual studies included within a review. explanatory paper. The PRISMA Statement itself provides further details regarding its background and development [12]. This accom- panying Explanation and Elaboration document explains the meaning and rationale for each checklist item. A few PRISMA on several occasions to further refine the document, which was Group participants volunteered to help draft specific items for circulated and ultimately approved by the larger PRISMA this document, and four of these (DGA, AL, DM, and JT) met Group. PLoS Medicine | www.plosmedicine.org 2 July 2009 | Volume 6 | Issue 7 | e1000100 Table 1. Checklist of items to include when reporting a systematic review (with or without meta-analysis). Section/Topic # Checklist Item Reported on Page # TITLE Title 1 Identify the report as a systematic review, meta-analysis, or both. ABSTRACT Structured summary 2 Provide a structured summary including, as applicable: background; objectives; data sources; study eligibility criteria, participants, and interventions; study appraisal and synthesis methods; results; limitations; conclusions and implications of key findings; systematic review registration number. INTRODUCTION Rationale 3 Describe the rationale for the review in the context of what is already known. Objectives 4 Provide an explicit statement of questions being addressed with reference to participants, interventions, comparisons, outcomes, and study design (PICOS). METHODS Protocol and registration 5 Indicate if a review protocol exists, if and where it can be accessed (e.g., Web address), and, if available, provide registration information including registration number. Eligibility criteria 6 Specify study characteristics (e.g., PICOS, length of follow-up) and report characteristics (e.g., years considered, language, publication status) used as criteria for eligibility, giving rationale. Information sources 7 Describe all information sources (e.g., databases with dates of coverage, contact with study authors to identify additional studies) in the search and date last searched. Search 8 Present full electronic search strategy for at least one database, including any limits used, such that it could be repeated. Study selection 9 State the process for selecting studies (i.e., screening, eligibility, included in systematic review, and, if applicable, included in the meta-analysis). Data collection process 10 Describe method of data extraction from reports (e.g., piloted forms, independently, in duplicate) and any processes for obtaining and confirming data from investigators. Data items 11 List and define all variables for which data were sought (e.g., PICOS, funding sources) and any assumptions and simplifications made. Risk of bias in individual 12 Describe methods used for assessing risk of bias of individual studies (including specification of whether this was studies done at the study or outcome level), and how this information is to be used in any data synthesis. Summary measures 13 State the principal summary measures (e.g., risk ratio, difference in means). Synthesis of results 14 Describe the methods of handling data and combining results of studies, if done, including measures of consistency (e.g., I ) for each meta-analysis. Risk of bias across studies 15 Specify any assessment of risk of bias that may affect the cumulative evidence (e.g., publication bias, selective reporting within studies). Additional analyses 16 Describe methods of additional analyses (e.g., sensitivity or subgroup analyses, meta-regression), if done, indicating which were pre-specified. RESULTS Study selection 17 Give numbers of studies screened, assessed for eligibility, and included in the review, with reasons for exclusions at each stage, ideally with a flow diagram. Study characteristics 18 For each study, present characteristics for which data were extracted (e.g., study size, PICOS, follow-up period) and provide the citations. Risk of bias within studies 19 Present data on risk of bias of each study and, if available, any outcome-level assessment (see Item 12). Results of individual 20 For all outcomes considered (benefits or harms), present, for each study: (a) simple summary data for each studies intervention group and (b) effect estimates and confidence intervals, ideally with a forest plot. Synthesis of results 21 Present results of each meta-analysis done, including confidence intervals and measures of consistency. Risk of bias across studies 22 Present results of any assessment of risk of bias across studies (see Item 15). Additional analysis 23 Give results of additional analyses, if done (e.g., sensitivity or subgroup analyses, meta-regression [see Item 16]). DISCUSSION Summary of evidence 24 Summarize the main findings including the strength of evidence for each main outcome; consider their relevance to key groups (e.g., health care providers, users, and policy makers). Limitations 25 Discuss limitations at study and outcome level (e.g., risk of bias), and at review level (e.g., incomplete retrieval of identified research, reporting bias). Conclusions 26 Provide a general interpretation of the results in the context of other evidence, and implications for future research. FUNDING Funding 27 Describe sources of funding for the systematic review and other support (e.g., supply of data); role of funders for the systematic review. doi:10.1371/journal.pmed.1000100.t001 PLoS Medicine | www.plosmedicine.org 3 July 2009 | Volume 6 | Issue 7 | e1000100 Figure 1. Flow of information through the different phases of a systematic review. doi:10.1371/journal.pmed.1000100.g001 examples by removing citations or Web addresses, or by spelling Scope of PRISMA out abbreviations.) We then explain the pertinent issue, the PRISMA focuses on ways in which authors can ensure the rationale for including the item, and relevant evidence from the transparent and complete reporting of systematic reviews and literature, whenever possible. No systematic search was carried out meta-analyses. It does not address directly or in a detailed manner to identify exemplars and evidence. We also include seven Boxes the conduct of systematic reviews, for which other guides are that provide a more comprehensive explanation of certain available [13,14,15,16]. thematic aspects of the methodology and conduct of systematic We developed the PRISMA Statement and this explanatory reviews. document to help authors report a wide array of systematic Although we focus on a minimal list of items to consider when reviews to assess the benefits and harms of a health care reporting a systematic review, we indicate places where additional intervention. We consider most of the checklist items relevant information is desirable to improve transparency of the review when reporting systematic reviews of non-randomized studies process. We present the items numerically from 1 to 27; however, assessing the benefits and harms of interventions. However, we authors need not address items in this particular order in their recognize that authors who address questions relating to reports. Rather, what is important is that the information for each etiology, diagnosis, or prognosis, for example, and who review item is given somewhere within the report. epidemiological or diagnostic accuracy studies may need to modify or incorporate additional items for their systematic The PRISMA Checklist reviews. TITLE and ABSTRACT How To Use This Paper Item 1: TITLE. Identify the report as a systematic review, meta-analysis, or both. We modeled this Explanation and Elaboration document after those prepared for other reporting guidelines [17,18,19]. To Examples. ‘‘Recurrence rates of video-assisted thoraco- maximize the benefit of this document, we encourage people to read it in conjunction with the PRISMA Statement [11]. scopic versus open surgery in the prevention of recurrent We present each checklist item and follow it with a published pneumothoraces: a systematic review of randomised and exemplar of good reporting for that item. (We edited some non-randomised trials’’ [20] PLoS Medicine | www.plosmedicine.org 4 July 2009 | Volume 6 | Issue 7 | e1000100 ‘‘Mortality in randomized trials of antioxidant supplements Box 2. Helping To Develop the Research for primary and secondary prevention: systematic review Question(s): The PICOS Approach and meta-analysis’’ [21] Formulating relevant and precise questions that can be answered in a systematic review can be complex and time Explanation. Authors should identify their report as a consuming. A structured approach for framing questions systematic review or meta-analysis. Terms such as ‘‘review’’ or that uses five components may help facilitate the process. ‘‘overview’’ do not describe for readers whether the review was This approach is commonly known by the acronym ‘‘PICOS’’ systematic or whether a meta-analysis was performed. A recent where each letter refers to a component: the patient survey found that 50% of 300 authors did not mention the terms population or the disease being addressed (P), the ‘‘systematic review’’ or ‘‘meta-analysis’’ in the title or abstract of interventions or exposure (I), the comparator group (C), their systematic review [3]. Although sensitive search strategies the outcome or endpoint (O), and the study design chosen have been developed to identify systematic reviews [22], inclusion (S) [186]. Issues relating to PICOS impact several PRISMA of the terms systematic review or meta-analysis in the title may items (i.e., Items 6, 8, 9, 10, 11, and 18). improve indexing and identification. Providing information about the population requires a We advise authors to use informative titles that make key precise definition of a group of participants (often patients), such as men over the age of 65 years, their defining information easily accessible to readers. Ideally, a title reflecting the characteristics of interest (often disease), and possibly the PICOS approach (participants, interventions, comparators, out- setting of care considered, such as an acute care hospital. comes, and study design) (see Item 11 and Box 2) may help readers as The interventions (exposures) under consideration in it provides key information about the scope of the review. Specifying the systematic review need to be transparently reported. the design(s) of the studies included, as shown in the examples, may For example, if the reviewers answer a question regarding also help some readers and those searching databases. the association between a woman’s prenatal exposure to Some journals recommend ‘‘indicative titles’’ that indicate the folic acid and subsequent offspring’s neural tube defects, topic matter of the review, while others require declarative titles reporting the dose, frequency, and duration of folic acid that give the review’s main conclusion. Busy practitioners may used in different studies is likely to be important for prefer to see the conclusion of the review in the title, but readers to interpret the review’s results and conclusions. declarative titles can oversimplify or exaggerate findings. Thus, Other interventions (exposures) might include diagnostic, many journals and methodologists prefer indicative titles as used in preventative, or therapeutic treatments, arrangements of the examples above. specific processes of care, lifestyle changes, psychosocial or educational interventions, or risk factors. Item 2: STRUCTURED SUMMARY. Provide a structured Clearly reporting the comparator (control) group summary including, as applicable: background; objectives; data intervention(s), such as usual care, drug, or placebo, is sources; study eligibility criteria, participants, and interventions; essential for readers to fully understand the selection criteria study appraisal and synthesis methods; results; limitations; of primary studies included in systematic reviews, and conclusions and implications of key findings; funding for the might be a source of heterogeneity investigators have to deal with. Comparators are often very poorly described. systematic review; and systematic review registration number. Clearly reporting what the intervention is compared with is very important and may sometimes have implications for Example. ‘‘Context: The role and dose of oral vitamin D the inclusion of studies in a review—many reviews compare supplementation in nonvertebral fracture prevention have with ‘‘standard care,’’ which is otherwise undefined; this not been well established. should be properly addressed by authors. Objective: To estimate the effectiveness of vitamin D The outcomes of the intervention being assessed, such as mortality, morbidity, symptoms, or quality of life improve- supplementation in preventing hip and nonvertebral frac- ments, should be clearly specified as they are required to tures in older persons. interpret the validity and generalizability of the systematic Data Sources: A systematic review of English and non-English review’s results. articles using MEDLINE and the Cochrane Controlled Finally, the type of study design(s) included in the Trials Register (1960–2005), and EMBASE (1991–2005). review should be reported. Some reviews only include Additional studies were identified by contacting clinical reports of randomized trials whereas others have broader experts and searching bibliographies and abstracts presented design criteria and include randomized trials and certain at the American Society for Bone and Mineral Research types of observational studies. Still other reviews, such as (1995–2004). Search terms included randomized controlled those specifically answering questions related to harms, may include a wide variety of designs ranging from cohort trial (RCT), controlled clinical trial, random allocation, studies to case reports. Whatever study designs are double-blind method, cholecalciferol, ergocalciferol, 25- included in the review, these should be reported. hydroxyvitamin D, fractures, humans, elderly, falls, and Independently from how difficult it is to identify the bone density. components of the research question, the important point is Study Selection: Only double-blind RCTs of oral vitamin D that a structured approach is preferable, and this extends supplementation (cholecalciferol, ergocalciferol) with or beyond systematic reviews of effectiveness. Ideally the without calcium supplementation vs calcium supplementa- PICOS criteria should be formulated a priori, in the tion or placebo in older persons (.60 years) that examined systematic review’s protocol, although some revisions might be required due to the iterative nature of the review process. hip or nonvertebral fractures were included. Authors are encouraged to report their PICOS criteria and Data Extraction: Independent extraction of articles by 2 whether any modifications were made during the review authors using predefined data fields, including study quality process. A useful example in this realm is the Appendix of indicators. the ‘‘Systematic Reviews of Water Fluoridation’’ undertaken Data Synthesis: All pooled analyses were based on random- by the Centre for Reviews and Dissemination [187]. effects models. Five RCTs for hip fracture (n = 9294) and 7 PLoS Medicine | www.plosmedicine.org 5 July 2009 | Volume 6 | Issue 7 | e1000100 heading, authors might describe the most important weaknesses RCTs for nonvertebral fracture risk (n = 9820) met our of included studies as well as limitations of the review process. inclusion criteria. All trials used cholecalciferol. Heteroge- Then authors should provide clear and balanced Conclusions that neity among studies for both hip and nonvertebral fracture are closely linked to the objective and findings of the review. prevention was observed, which disappeared after pooling Additionally, it would be helpful if authors included some RCTs with low-dose (400 IU/d) and higher-dose vitamin D information about funding for the review. Finally, although (700–800 IU/d), separately. A vitamin D dose of 700 to protocol registration for systematic reviews is still not common 800 IU/d reduced the relative risk (RR) of hip fracture by practice, if authors have registered their review or received a 26% (3 RCTs with 5572 persons; pooled RR, 0.74; 95% registration number, we recommend providing the registration confidence interval [CI], 0.61–0.88) and any nonvertebral information at the end of the abstract. fracture by 23% (5 RCTs with 6098 persons; pooled RR, Taking all the above considerations into account, the intrinsic 0.77; 95% CI, 0.68–0.87) vs calcium or placebo. No tension between the goal of completeness of the abstract and its significant benefit was observed for RCTs with 400 IU/d keeping into the space limit often set by journal editors is vitamin D (2 RCTs with 3722 persons; pooled RR for hip recognized as a major challenge. fracture, 1.15; 95% CI, 0.88–1.50; and pooled RR for any nonvertebral fracture, 1.03; 95% CI, 0.86–1.24). INTRODUCTION Conclusions: Oral vitamin D supplementation between 700 to Item 3: RATIONALE. Describe the rationale for the review 800 IU/d appears to reduce the risk of hip and any in the context of what is already known. nonvertebral fractures in ambulatory or institutionalized elderly persons. An oral vitamin D dose of 400 IU/d is not sufficient for fracture prevention.’’ [23] Example. ‘‘Reversing the trend of increasing weight for height in children has proven difficult. It is widely accepted Explanation. Abstracts provide key information that enables that increasing energy expenditure and reducing energy readers to understand the scope, processes, and findings of a intake form the theoretical basis for management. There- review and to decide whether to read the full report. The abstract fore, interventions aiming to increase physical activity and may be all that is readily available to a reader, for example, in a improve diet are the foundation of efforts to prevent and bibliographic database. The abstract should present a balanced treat childhood obesity. Such lifestyle interventions have and realistic assessment of the review’s findings that mirrors, albeit been supported by recent systematic reviews, as well as by briefly, the main text of the report. the Canadian Paediatric Society, the Royal College of We agree with others that the quality of reporting in abstracts Paediatrics and Child Health, and the American Academy presented at conferences and in journal publications needs of Pediatrics. However, these interventions are fraught with improvement [24,25]. While we do not uniformly favor a specific poor adherence. Thus, school-based interventions are format over another, we generally recommend structured abstracts. theoretically appealing because adherence with interven- Structured abstracts provide readers with a series of headings tions can be improved. Consequently, many local govern- pertaining to the purpose, conduct, findings, and conclusions of the ments have enacted or are considering policies that mandate systematic review being reported [26,27]. They give readers more complete information and facilitate finding information more easily increased physical activity in schools, although the effect of than unstructured abstracts [28,29,30,31,32]. such interventions on body composition has not been A highly structured abstract of a systematic review could include assessed.’’ [33] the following headings: Context (or Background); Objective (or Explanation. Readers need to understand the rationale Purpose); Data Sources; Study Selection (or Eligibility Criteria); behind the study and what the systematic review may add to Study Appraisal and Synthesis Methods (or Data Extraction and what is already known. Authors should tell readers whether their Data Synthesis); Results; Limitations; and Conclusions (or report is a new systematic review or an update of an existing one. Implications). Alternatively, a simpler structure could cover but If the review is an update, authors should state reasons for the collapse some of the above headings (e.g., label Study Selection update, including what has been added to the evidence base since and Study Appraisal as Review Methods) or omit some headings the previous version of the review. such as Background and Limitations. An ideal background or introduction that sets context for In the highly structured abstract mentioned above, authors use readers might include the following. First, authors might define the the Background heading to set the context for readers and explain importance of the review question from different perspectives (e.g., the importance of the review question. Under the Objectives public health, individual patient, or health policy). Second, authors heading, they ideally use elements of PICOS (see Box 2) to state might briefly mention the current state of knowledge and its the primary objective of the review. Under a Data Sources heading, they summarize sources that were searched, any limitations. As in the above example, information about the effects of several different interventions may be available that helps language or publication type restrictions, and the start and end dates of searches. Study Selection statements then ideally describe readers understand why potential relative benefits or harms of particular interventions need review. Third, authors might whet who selected studies using what inclusion criteria. Data Extraction Methods statements describe appraisal methods during data readers’ appetites by clearly stating what the review aims to add. abstraction and the methods used to integrate or summarize They also could discuss the extent to which the limitations of the the data. The Data Synthesis section is where the main results of existing evidence base may be overcome by the review. the review are reported. If the review includes meta-analyses, Item 4: OBJECTIVES. Provide an explicit statement of authors should provide numerical results with confidence questions being addressed with reference to participants, intervals for the most important outcomes. Ideally, they should interventions, comparisons, outcomes, and study design specify the amount of evidence in these analyses (numbers of (PICOS). studies and numbers of participants). Under a Limitations PLoS Medicine | www.plosmedicine.org 6 July 2009 | Volume 6 | Issue 7 | e1000100 Cochrane reviews revealed indirect evidence for possible selective Example. ‘‘To examine whether topical or intraluminal reporting bias for systematic reviews. Almost all (n = 43) contained antibiotics reduce catheter-related bloodstream infection, we a major change, such as the addition or deletion of outcomes, reviewed randomized, controlled trials that assessed the between the protocol and the full publication [44]. Whether (or to efficacy of these antibiotics for primary prophylaxis against what extent) the changes reflected bias, however, was not clear. catheter-related bloodstream infection and mortality com- For example, it has been rather common not to describe outcomes pared with no antibiotic therapy in adults undergoing that were not presented in any of the included studies. hemodialysis.’’ [34] Registration of a systematic review, typically with a protocol and registration number, is not yet common, but some opportunities Explanation. The questions being addressed, and the exist [45,46]. Registration may possibly reduce the risk of multiple rationale for them, are one of the most critical parts of a reviews addressing the same question [45,46,47,48], reduce systematic review. They should be stated precisely and explicitly publication bias, and provide greater transparency when updating so that readers can understand quickly the review’s scope and the systematic reviews. Of note, a survey of systematic reviews indexed potential applicability of the review to their interests [35]. in MEDLINE in November 2004 found that reports of protocol Framing questions so that they include the following five use had increased to about 46% [3] from 8% noted in previous ‘‘PICOS’’ components may improve the explicitness of review surveys [49]. The improvement was due mostly to Cochrane questions: (1) the patient population or disease being addressed reviews, which, by requirement, have a published protocol [3]. (P), (2) the interventions or exposure of interest (I), (3) the comparators (C), (4) the main outcome or endpoint of interest Item 6: ELIGIBILITY CRITERIA. Specify study charac- (O), and (5) the study designs chosen (S). For more detail teristics (e.g., PICOS, length of follow-up) and report regarding PICOS, see Box 2. characteristics (e.g., years considered, language, publication Good review questions may be narrowly focused or broad, status) used as criteria for eligibility, giving rationale. depending on the overall objectives of the review. Sometimes broad questions might increase the applicability of the results and facilitate detection of bias, exploratory analyses, and sensitivity Examples. Types of studies: ‘‘Randomised clinical trials analyses [35,36]. Whether narrowly focused or broad, precisely studying the administration of hepatitis B vaccine to CRF stated review objectives are critical as they help define other [chronic renal failure] patients, with or without dialysis. No components of the review process such as the eligibility criteria language, publication date, or publication status restrictions (Item 6) and the search for relevant literature (Items 7 and 8). were imposed…’’ Types of participants: ‘‘Participants of any age with CRF or METHODS receiving dialysis (haemodialysis or peritoneal dialysis) were Item 5: PROTOCOL AND REGISTRATION. Indicate if a considered. CRF was defined as serum creatinine greater review protocol exists, if and where it can be accessed (e.g., Web than 200 mmol/L for a period of more than six months or address) and, if available, provide registration information individuals receiving dialysis (haemodialysis or peritoneal including the registration number. dialysis)…Renal transplant patients were excluded from this review as these individuals are immunosuppressed and are Example. ‘‘Methods of the analysis and inclusion criteria receiving immunosuppressant agents to prevent rejection of were specified in advance and documented in a protocol.’’ [37] their transplanted organs, and they have essentially normal renal function…’’ Explanation. A protocol is important because it pre-specifies Types of intervention: ‘‘Trials comparing the beneficial and the objectives and methods of the systematic review. For instance, harmful effects of hepatitis B vaccines with adjuvant or a protocol specifies outcomes of primary interest, how reviewers will extract information about those outcomes, and methods that cytokine co-interventions [and] trials comparing the bene- reviewers might use to quantitatively summarize the outcome data ficial and harmful effects of immunoglobulin prophylaxis. (see Item 13). Having a protocol can help restrict the likelihood of This review was limited to studies looking at active biased post hoc decisions in review methods, such as selective immunization. Hepatitis B vaccines (plasma or recombinant outcome reporting. Several sources provide guidance about (yeast) derived) of all types, dose, and regimens versus elements to include in the protocol for a systematic review placebo, control vaccine, or no vaccine…’’ [16,38,39]. For meta-analyses of individual patient-level data, we Types of outcome measures: ‘‘Primary outcome measures: advise authors to describe whether a protocol was explicitly Seroconversion, ie, proportion of patients with adequate designed and whether, when, and how participating collaborators anti-HBs response (.10 IU/L or Sample Ratio Units). endorsed it [40,41]. Hepatitis B infections (as measured by hepatitis B core Authors may modify protocols during the research, and readers antigen (HBcAg) positivity or persistent HBsAg positivity), should not automatically consider such modifications inappropri- both acute and chronic. Acute (primary) HBV [hepatitis B ate. For example, legitimate modifications may extend the period virus] infections were defined as seroconversion to HBsAg of searches to include older or newer studies, broaden eligibility positivity or development of IgM anti-HBc. Chronic HBV criteria that proved too narrow, or add analyses if the primary infections were defined as the persistence of HBsAg for more analyses suggest that additional ones are warranted. Authors than six months or HBsAg positivity and liver biopsy should, however, describe the modifications and explain their compatible with a diagnosis or chronic hepatitis B. rationale. Secondary outcome measures: Adverse events of hepatitis Although worthwhile protocol amendments are common, one B vaccinations…[and]…mortality.’’ [50] must consider the effects that protocol modifications may have on the results of a systematic review, especially if the primary outcome is changed. Bias from selective outcome reporting in randomized Explanation. Knowledge of the eligibility criteria is essential trials has been well documented [42,43]. An examination of 47 in appraising the validity, applicability, and comprehensiveness of PLoS Medicine | www.plosmedicine.org 7 July 2009 | Volume 6 | Issue 7 | e1000100 a review. Thus, authors should unambiguously specify eligibility In addition to searching databases, authors should report the use of supplementary approaches to identify studies, such as hand criteria used in the review. Carefully defined eligibility criteria inform various steps of the review methodology. They influence searching of journals, checking reference lists, searching trials registries or regulatory agency Web sites [67], contacting the development of the search strategy and serve to ensure that studies are selected in a systematic and unbiased manner. manufacturers, or contacting authors. Authors should also report if they attempted to acquire any missing information (e.g., on study A study may be described in multiple reports, and one report may methods or results) from investigators or sponsors; it is useful to describe multiple studies. Therefore, we separate eligibility criteria describe briefly who was contacted and what unpublished into the following two components: study characteristics and report information was obtained. characteristics. Both need to be reported. Study eligibility criteria are likely to include the populations, interventions, comparators, Item 8: SEARCH. Present the full electronic search strategy outcomes, and study designs of interest (PICOS; see Box 2), as well for at least one major database, including any limits used, such that as other study-specific elements, such as specifying a minimum it could be repeated. length of follow-up. Authors should state whether studies will be excluded because they do not include (or report) specific outcomes to help readers ascertain whether the systematic review may be Examples. In text: ‘‘We used the following search terms to biased as a consequence of selective reporting [42,43]. search all trials registers and databases: immunoglobulin*; Report eligibility criteria are likely to include language of IVIG; sepsis; septic shock; septicaemia; and septicemia…’’ publication, publication status (e.g., inclusion of unpublished [68] material and abstracts), and year of publication. Inclusion or not of In appendix: ‘‘Search strategy: MEDLINE (OVID) non-English language literature [51,52,53,54,55], unpublished 01. immunoglobulins/ data, or older data can influence the effect estimates in meta- 02. immunoglobulin$.tw. analyses [56,57,58,59]. Caution may need to be exercised in 03. ivig.tw. including all identified studies due to potential differences in the 04. 1 or 2 or 3 risk of bias such as, for example, selective reporting in abstracts 05. sepsis/ [60,61,62]. 06. sepsis.tw. 07. septic shock/ Item 7: INFORMATION SOURCES. Describe all 08. septic shock.tw. information sources in the search (e.g., databases with dates of 09. septicemia/ coverage, contact with study authors to identify additional studies) and date last searched. 10. septicaemia.tw. 11. septicemia.tw. 12. 5 or 6 or 7 or 8 or 9 or 10 or 11 Example. ‘‘Studies were identified by searching electronic 13. 4 and 12 databases, scanning reference lists of articles and consulta- 14. randomized controlled trials/ tion with experts in the field and drug companies…No limits 15. randomized-controlled-trial.pt. were applied for language and foreign papers were 16. controlled-clinical-trial.pt. translated. This search was applied to Medline (1966– 17. random allocation/ Present), CancerLit (1975–Present), and adapted for Embase 18. double-blind method/ (1980–Present), Science Citation Index Expanded (1981– 19. single-blind method/ Present) and Pre-Medline electronic databases. Cochrane 20. 14 or 15 or 16 or 17 or 18 or 19 and DARE (Database of Abstracts of Reviews of Effective- 21. exp clinical trials/ ness) databases were reviewed…The last search was run on 22. clinical-trial.pt. 19 June 2001. In addition, we handsearched contents pages 23. (clin$ adj trial$).ti,ab. of Journal of Clinical Oncology 2001, European Journal of 24. ((singl$ or doubl$ or trebl$ or tripl$) adj (blind$)).ti,ab. Cancer 2001 and Bone 2001, together with abstracts printed 25. placebos/ in these journals 1999–2001. A limited update literature 26. placebo$.ti,ab. search was performed from 19 June 2001 to 31 December 27. random$.ti,ab. 2003.’’ [63] 28. 21 or 22 or 23 or 24 or 25 or 26 or 27 29. research design/ Explanation. The National Library of Medicine’s 30. comparative study/ MEDLINE database is one of the most comprehensive sources 31. exp evaluation studies/ of health care information in the world. Like any database, 32. follow-up studies/ however, its coverage is not complete and varies according to the 33. prospective studies/ field. Retrieval from any single database, even by an experienced 34. (control$ or prospective$ or volunteer$).ti,ab. searcher, may be imperfect, which is why detailed reporting is important within the systematic review. 35. 30 or 31 or 32 or 33 or 34 At a minimum, for each database searched, authors should 36. 20 or 28 or 29 or 35 report the database, platform, or provider (e.g., Ovid, Dialog, 37. 13 and 36’’ [68] PubMed) and the start and end dates for the search of each database. This information lets readers assess the currency of the Explanation. The search strategy is an essential part of the review, which is important because the publication time-lag report of any systematic review. Searches may be complicated and outdates the results of some reviews [64]. This information should iterative, particularly when reviewers search unfamiliar databases also make updating more efficient [65]. Authors should also report or their review is addressing a broad or new topic. Perusing the who developed and conducted the search [66]. search strategy allows interested readers to assess the PLoS Medicine | www.plosmedicine.org 8 July 2009 | Volume 6 | Issue 7 | e1000100 comprehensiveness and completeness of the search, and to efforts were made to resolve disagreements (e.g., by contact with replicate it. Thus, we advise authors to report their full the authors of the original studies). electronic search strategy for at least one major database. As an Item 10: DATA COLLECTION PROCESS. Describe the alternative to presenting search strategies for all databases, authors method of data extraction from reports (e.g., piloted forms, could indicate how the search took into account other databases independently by two reviewers) and any processes for obtaining searched, as index terms vary across databases. If different and confirming data from investigators. searches are used for different parts of a wider question (e.g., questions relating to benefits and questions relating to harms), we recommend authors provide at least one example of a strategy for Example. ‘‘We developed a data extraction sheet (based on each part of the objective [69]. We also encourage authors to state the Cochrane Consumers and Communication Review whether search strategies were peer reviewed as part of the Group’s data extraction template), pilot-tested it on ten systematic review process [70]. randomly-selected included studies, and refined it accord- We realize that journal restrictions vary and that having the ingly. One review author extracted the following data from search strategy in the text of the report is not always feasible. We included studies and the second author checked the strongly encourage all journals, however, to find ways, such as a extracted data…Disagreements were resolved by discussion ‘‘Web extra,’’ appendix, or electronic link to an archive, to make between the two review authors; if no agreement could be search strategies accessible to readers. We also advise all authors to reached, it was planned a third author would decide. We archive their searches so that (1) others may access and review contacted five authors for further information. All responded them (e.g., replicate them or understand why their review of a and one provided numerical data that had only been similar topic did not identify the same reports), and (2) future updates of their review are facilitated. presented graphically in the published paper.’’ [77] Several sources provide guidance on developing search strategies [71,72,73]. Most searches have constraints, for example Explanation. Reviewers extract information from each relating to limited time or financial resources, inaccessible or included study so that they can critique, present, and summarize inadequately indexed reports and databases, unavailability of evidence in a systematic review. They might also contact authors experts with particular language or database searching skills, or of included studies for information that has not been, or is review questions for which pertinent evidence is not easy to find. unclearly, reported. In meta-analysis of individual patient data, Authors should be straightforward in describing their search this phase involves collection and scrutiny of detailed raw constraints. Apart from the keywords used to identify or exclude databases. The authors should describe these methods, including records, they should report any additional limitations relevant to any steps taken to reduce bias and mistakes during data collection the search, such as language and date restrictions (see also and data extraction [78] (Box 3). eligibility criteria, Item 6) [51]. Some systematic reviewers use a data extraction form that could be reported as an appendix or ‘‘Web extra’’ to their report. These Item 9: STUDY SELECTION. State the process for selecting forms could show the reader what information reviewers sought studies (i.e., for screening, for determining eligibility, for inclusion (see Item 11) and how they extracted it. Authors could tell readers in the systematic review, and, if applicable, for inclusion in the if the form was piloted. Regardless, we advise authors to tell meta-analysis). readers who extracted what data, whether any extractions were completed in duplicate, and, if so, whether duplicate abstraction was done independently and how disagreements were resolved. Example. ‘‘Eligibility assessment…[was] performed inde- Published reports of the included studies may not provide all the pendently in an unblinded standardized manner by 2 information required for the review. Reviewers should describe reviewers…Disagreements between reviewers were resolved any actions they took to seek additional information from the by consensus.’’ [74] original researchers (see Item 7). The description might include how they attempted to contact researchers, what they asked for, Explanation. There is no standard process for selecting and their success in obtaining the necessary information. Authors studies to include in a systematic review. Authors usually start with should also tell readers when individual patient data were sought a large number of identified records from their search and from the original researchers [41] (see Item 11) and indicate the sequentially exclude records according to eligibility criteria. We studies for which such data were used in the analyses. The advise authors to report how they screened the retrieved records reviewers ideally should also state whether they confirmed the (typically a title and abstract), how often it was necessary to review accuracy of the information included in their review with the the full text publication, and if any types of record (e.g., letters to original researchers, for example, by sending them a copy of the the editor) were excluded. We also advise using the PRISMA flow draft review [79]. diagram to summarize study selection processes (see Item 17; Box Some studies are published more than once. Duplicate 3). publications may be difficult to ascertain, and their inclusion Efforts to enhance objectivity and avoid mistakes in study may introduce bias [80,81]. We advise authors to describe any selection are important. Thus authors should report whether each steps they used to avoid double counting and piece together data stage was carried out by one or several people, who these people from multiple reports of the same study (e.g., juxtaposing author were, and, whenever multiple independent investigators per- formed the selection, what the process was for resolving names, treatment comparisons, sample sizes, or outcomes). We also advise authors to indicate whether all reports on a study were disagreements. The use of at least two investigators may reduce the possibility of rejecting relevant reports [75]. The benefit may considered, as inconsistencies may reveal important limitations. For example, a review of multiple publications of drug trials be greatest for topics where selection or rejection of an article requires difficult judgments [76]. For these topics, authors should showed that reported study characteristics may differ from report ideally tell readers the level of inter-rater agreement, how to report, including the description of the design, number of commonly arbitration about selection was required, and what patients analyzed, chosen significance level, and outcomes [82]. PLoS Medicine | www.plosmedicine.org 9 July 2009 | Volume 6 | Issue 7 | e1000100 Box 3. Identification of Study Reports and Box 4. Study Quality and Risk of Bias Data Extraction In this paper, and elsewhere [11], we sought to use a new Comprehensive searches usually result in a large number term for many readers, namely, risk of bias, for evaluating of identified records, a much smaller number of studies each included study in a systematic review. Previous included in the systematic review, and even fewer of these papers [89,188] tended to use the term ‘‘quality’’. When studies included in any meta-analyses. Reports of system- carrying out a systematic review we believe it is important atic reviews often provide little detail as to the methods to distinguish between quality and risk of bias and to focus used by the review team in this process. Readers are often on evaluating and reporting the latter. Quality is often the left with what can be described as the ‘‘X-files’’ phenom- best the authors have been able to do. For example, enon, as it is unclear what occurs between the initial set of authors may report the results of surgical trials in which identified records and those finally included in the review. blinding of the outcome assessors was not part of the Sometimes, review authors simply report the number of trial’s conduct. Even though this may have been the best included studies; more often they report the initial number methodology the researchers were able to do, there are of identified records and the number of included studies. still theoretical grounds for believing that the study was Rarely, although this is optimal for readers, do review susceptible to (risk of) bias. authors report the number of identified records, the Assessing the risk of bias should be part of the conduct smaller number of potentially relevant studies, and the and reporting of any systematic review. In all situations, we even smaller number of included studies, by outcome. encourage systematic reviewers to think ahead carefully Review authors also need to differentiate between the about what risks of bias (methodological and clinical) may number of reports and studies. Often there will not be a have a bearing on the results of their systematic reviews. 1:1 ratio of reports to studies and this information needs to For systematic reviewers, understanding the risk of bias be described in the systematic review report. on the results of studies is often difficult, because the Ideally, the identification of study reports should be report is only a surrogate of the actual conduct of the reported as text in combination with use of the PRISMA study. There is some suggestion [189,190] that the report flow diagram. While we recommend use of the flow may not be a reasonable facsimile of the study, although diagram, a small number of reviews might be particularly this view is not shared by all [88,191]. There are three main simple and can be sufficiently described with a few brief ways to assess risk of bias: individual components, sentences of text. More generally, review authors will need checklists, and scales. There are a great many scales to report the process used for each step: screening the available [192], although we caution their use based on identified records; examining the full text of potentially theoretical grounds [193] and emerging empirical evi- relevant studies (and reporting the number that could not dence [194]. Checklists are less frequently used and be obtained); and applying eligibility criteria to select the potentially run the same problems as scales. We advocate included studies. using a component approach and one that is based on Such descriptions should also detail how potentially domains for which there is good empirical evidence and eligible records were promoted to the next stage of the perhaps strong clinical grounds. The new Cochrane risk of review (e.g., full text screening) and to the final stage of bias tool [11] is one such component approach. this process, the included studies. Often review teams have The Cochrane risk of bias tool consists of five items for three response options for excluding records or promoting which there is empirical evidence for their biasing them to the next stage of the winnowing process: ‘‘yes,’’ influence on the estimates of an intervention’s effective- ‘‘no,’’ and ‘‘maybe.’’ ness in randomized trials (sequence generation, allocation Similarly, some detail should be reported on who concealment, blinding, incomplete outcome data, and participated and how such processes were completed. For selective outcome reporting) and a catch-all item called example, a single person may screen the identified records ‘‘other sources of bias’’ [11]. There is also some consensus while a second person independently examines a small that these items can be applied for evaluation of studies sample of them. The entire winnowing process is one of across very diverse clinical areas [93]. Other risk of bias ‘‘good book keeping’’ whereby interested readers should items may be topic or even study specific, i.e., they may be able to work backwards from the included studies to stem from some peculiarity of the research topic or some come up with the same numbers of identified records. special feature of the design of a specific study. These There is often a paucity of information describing the peculiarities need to be investigated on a case-by-case data extraction processes in reports of systematic reviews. basis, based on clinical and methodological acumen, and Authors may simply report that ‘‘relevant’’ data were there can be no general recipe. In all situations, systematic extracted from each included study with little information reviewers need to think ahead carefully about what about the processes used for data extraction. It may be aspects of study quality may have a bearing on the results. useful for readers to know whether a systematic review’s authors developed, a priori or not, a data extraction form, Authors ideally should present any algorithm that they used to whether multiple forms were used, the number of select data from overlapping reports and any efforts they used to questions, whether the form was pilot tested, and who solve logical inconsistencies across reports. completed the extraction. For example, it is important for readers to know whether one or more people extracted Item 11: DATA ITEMS. List and define all variables for data, and if so, whether this was completed independent- which data were sought (e.g., PICOS, funding sources), and any ly, whether ‘‘consensus’’ data were used in the analyses, assumptions and simplifications made. and if the review team completed an informal training exercise or a more formal reliability exercise. Examples. ‘‘Information was extracted from each included trial on: (1) characteristics of trial participants (including age, stage and severity of disease, and method of diagnosis), and PLoS Medicine | www.plosmedicine.org 10 July 2009 | Volume 6 | Issue 7 | e1000100 Explanation. The likelihood that the treatment effect reported the trial’s inclusion and exclusion criteria; (2) type of in a systematic review approximates the truth depends on the validity intervention (including type, dose, duration and frequency of the included studies, as certain methodological characteristics may of the NSAID [non-steroidal anti-inflammatory drug]; be associated with effect sizes [87,88]. For example, trials without versus placebo or versus the type, dose, duration and reported adequate allocation concealment exaggerate treatment frequency of another NSAID; or versus another pain effects on average compared to those with adequate concealment management drug; or versus no treatment); (3) type of [88]. Therefore, it is important for authors to describe any methods outcome measure (including the level of pain reduction, that they used to gauge the risk of bias in the included studies and how improvement in quality of life score (using a validated scale), that information was used [89]. Additionally, authors should provide effect on daily activities, absence from work or school, length a rationale if no assessment of risk of bias was undertaken. The most of follow up, unintended effects of treatment, number of popular term to describe the issues relevant to this item is ‘‘quality,’’ women requiring more invasive treatment).’’ [83] but for the reasons that are elaborated in Box 4 we prefer to name this item as ‘‘assessment of risk of bias.’’ Explanation. It is important for readers to know what Many methods exist to assess the overall risk of bias in included information review authors sought, even if some of this studies, including scales, checklists, and individual components information was not available [84]. If the review is limited to [90,91]. As discussed in Box 4, scales that numerically summarize reporting only those variables that were obtained, rather than multiple components into a single number are misleading and those that were deemed important but could not be obtained, bias unhelpful [92,93]. Rather, authors should specify the methodolog- might be introduced and the reader might be misled. It is therefore ical components that they assessed. Common markers of validity for helpful if authors can refer readers to the protocol (see Item 5), and randomized trials include the following: appropriate generation of archive their extraction forms (see Item 10), including definitions random allocation sequence [94]; concealment of the allocation of variables. The published systematic review should include a sequence [93]; blinding of participants, health care providers, data description of the processes used with, if relevant, specification of collectors, and outcome adjudicators [95,96,97,98]; proportion of how readers can get access to additional materials. patients lost to follow-up [99,100]; stopping of trials early for benefit We encourage authors to report whether some variables were [101]; and whether the analysis followed the intention-to-treat added after the review started. Such variables might include those principle [100,102]. The ultimate decision regarding which found in the studies that the reviewers identified (e.g., important methodological features to evaluate requires consideration of the outcome measures that the reviewers initially overlooked). Authors strength of the empiric data, theoretical rationale, and the unique should describe the reasons for adding any variables to those circumstances of the included studies. already pre-specified in the protocol so that readers can Authors should report how they assessed risk of bias; whether understand the review process. it was in a blind manner; and if assessments were completed by We advise authors to report any assumptions they made about more than one person, and if so, whether they were completed missing or unclear information and to explain those processes. For independently [103,104]. Similarly, we encourage authors to example, in studies of women aged 50 or older it is reasonable to report any calibration exercises among review team members assume that none were pregnant, even if this is not reported. that were done. Finally, authors need to report how their Likewise, review authors might make assumptions about the route assessments of risk of bias are used subsequently in the data of administration of drugs assessed. However, special care should synthesis (see Item 16). Despite the often difficult task of be taken in making assumptions about qualitative information. For assessing the risk of bias in included studies, authors are example, the upper age limit for ‘‘children’’ can vary from 15 years sometimes silent on what they did with the resultant assessments to 21 years, ‘‘intense’’ physiotherapy might mean very different [89]. If authors exclude studies from the review or any things to different researchers at different times and for different subsequent analyses on the basis of the risk of bias, they should patients, and the volume of blood associated with ‘‘heavy’’ blood tell readers which studies they excluded and explain the reasons loss might vary widely depending on the setting. for those exclusions (see Item 6). Authors should also describe any planned sensitivity or subgroup analyses related to bias Item 12: RISK OF BIAS IN INDIVIDUAL STUDIES. assessments (see Item 16). Describe methods used for assessing risk of bias in individual studies (including specification of whether this was done at the Item 13: SUMMARY MEASURES. State the principal study or outcome level, or both), and how this information is to be summary measures (e.g., risk ratio, difference in means). used in any data synthesis. Examples. ‘‘Relative risk of mortality reduction was the Example. ‘‘To ascertain the validity of eligible randomized primary measure of treatment effect.’’ [105] trials, pairs of reviewers working independently and with ‘‘The meta-analyses were performed by computing relative adequate reliability determined the adequacy of randomi- risks (RRs) using random-effects model. Quantitative zation and concealment of allocation, blinding of patients, analyses were performed on an intention-to-treat basis and health care providers, data collectors, and outcome were confined to data derived from the period of follow-up. assessors; and extent of loss to follow-up (i.e. proportion of RR and 95% confidence intervals for each side effect (and patients in whom the investigators were not able to ascertain all side effects) were calculated.’’ [106] outcomes).’’ [85] ‘‘The primary outcome measure was the mean difference in ‘‘To explore variability in study results (heterogeneity) we log HIV-1 viral load comparing zinc supplementation to specified the following hypotheses before conducting the placebo…’’ [107] analysis. We hypothesised that effect size may differ according to the methodological quality of the studies.’’ Explanation. When planning a systematic review, it is [86] generally desirable that authors pre-specify the outcomes of PLoS Medicine | www.plosmedicine.org 11 July 2009 | Volume 6 | Issue 7 | e1000100 primary interest (see Item 5) as well as the intended summary effect depression) are used across studies, the sign of some scores may measure for each outcome. The chosen summary effect measure need to be reversed to ensure that all scales are aligned (e.g., so low may differ from that used in some of the included studies. If values represent good health on all scales). Standard deviations possible the choice of effect measures should be explained, though may have to be reconstructed from other statistics such as p-values it is not always easy to judge in advance which measure is the most and t statistics [115,116], or occasionally they may be imputed appropriate. from the standard deviations observed in other studies [117]. For binary outcomes, the most common summary measures are Time-to-event data also usually need careful conversions to a the risk ratio, odds ratio, and risk difference [108]. Relative effects consistent format [111]. Authors should report details of any such are more consistent across studies than absolute effects [109,110], data processing. although absolute differences are important when interpreting Statistical combination of data from two or more separate findings (see Item 24). studies in a meta-analysis may be neither necessary nor desirable For continuous outcomes, the natural effect measure is the (see Box 5 and Item 21). Regardless of the decision to combine difference in means [108]. Its use is appropriate when outcome individual study results, authors should report how they planned to measurements in all studies are made on the same scale. The evaluate between-study variability (heterogeneity or inconsistency) standardized difference in means is used when the studies do not (Box 6). The consistency of results across trials may influence the yield directly comparable data. Usually this occurs when all studies decision of whether to combine trial results in a meta-analysis. assess the same outcome but measure it in a variety of ways (e.g., When meta-analysis is done, authors should specify the effect different scales to measure depression). measure (e.g., relative risk or mean difference) (see Item 13), the For time-to-event outcomes, the hazard ratio is the most statistical method (e.g., inverse variance), and whether a fixed- or common summary measure. Reviewers need the log hazard ratio random-effects approach, or some other method (e.g., Bayesian) and its standard error for a study to be included in a meta-analysis was used (see Box 6). If possible, authors should explain the [111]. This information may not be given for all studies, but reasons for those choices. methods are available for estimating the desired quantities from Item 15: RISK OF BIAS ACROSS STUDIES. Specify any other reported information [111]. Risk ratio and odds ratio (in assessment of risk of bias that may affect the cumulative evidence relation to events occurring by a fixed time) are not equivalent to (e.g., publication bias, selective reporting within studies). the hazard ratio, and median survival times are not a reliable basis for meta-analysis [112]. If authors have used these measures they should describe their methods in the report. Examples. ‘‘For each trial we plotted the effect by the inverse of its standard error. The symmetry of such ‘funnel Item 14: PLANNED METHODS OF ANALYSIS. Describe plots’ was assessed both visually, and formally with Egger’s the methods of handling data and combining results of studies, if test, to see if the effect decreased with increasing sample done, including measures of consistency (e.g., I ) for each meta- size.’’ [118] analysis. ‘‘We assessed the possibility of publication bias by evaluating a funnel plot of the trial mean differences for asymmetry, Examples. ‘‘We tested for heterogeneity with the Breslow- which can result from the non publication of small trials with Day test, and used the method proposed by Higgins et al. to negative results…Because graphical evaluation can be measure inconsistency (the percentage of total variation subjective, we also conducted an adjusted rank correlation across studies due to heterogeneity) of effects across lipid- test and a regression asymmetry test as formal statistical tests lowering interventions. The advantages of this measure of for publication bias…We acknowledge that other factors, inconsistency (termed I ) are that it does not inherently such as differences in trial quality or true study heteroge- depend on the number of studies and is accompanied by an neity, could produce asymmetry in funnel plots.’’ [119] uncertainty interval.’’ [113] ‘‘In very few instances, estimates of baseline mean or mean Explanation. Reviewers should explore the possibility that QOL [Quality of life] responses were obtained without the available data are biased. They may examine results from the corresponding estimates of variance (standard deviation available studies for clues that suggest there may be missing studies [SD] or standard error). In these instances, an SD was (publication bias) or missing data from the included studies imputed from the mean of the known SDs. In a number of (selective reporting bias) (see Box 7). Authors should report in cases, the response data available were the mean and detail any methods used to investigate possible bias across studies. variance in a pre study condition and after therapy. The It is difficult to assess whether within-study selective reporting is within-patient variance in these cases could not be present in a systematic review. If a protocol of an individual study is calculated directly and was approximated by assuming available, the outcomes in the protocol and the published report can independence.’’ [114] be compared. Even in the absence of a protocol, outcomes listed in the methods section of the published report can be compared with Explanation. The data extracted from the studies in the those for which results are presented [120]. In only half of 196 trial reports describing comparisons of two drugs in arthritis were all the review may need some transformation (processing) before they are suitable for analysis or for presentation in an evidence table. effect variables in the methods and results sections the same [82]. In other cases, knowledge of the clinical area may suggest that it is Although such data handling may facilitate meta-analyses, it is sometimes needed even when meta-analyses are not done. For likely that the outcome was measured even if it was not reported. For example, in a particular disease, if one of two linked outcomes is example, in trials with more than two intervention groups it may be necessary to combine results for two or more groups (e.g., reported but the other is not, then one should question whether the latter has been selectively omitted [121,122]. receiving similar but non-identical interventions), or it may be desirable to include only a subset of the data to match the review’s Only 36% (76 of 212) of therapeutic systematic reviews inclusion criteria. When several different scales (e.g., for published in November 2004 reported that study publication PLoS Medicine | www.plosmedicine.org 12 July 2009 | Volume 6 | Issue 7 | e1000100 Sensitivity analyses are used to explore the degree to which the Box 5. Whether or Not To Combine Data main findings of a systematic review are affected by changes in Deciding whether or not to combine data involves its methods or in the data used from individual studies (e.g., statistical, clinical, and methodological considerations. study inclusion criteria, results of risk of bias assessment). The statistical decisions are perhaps the most technical Subgroup analyses address whether the summary effects vary and evidence-based. These are more thoroughly discussed in relation to specific (usually clinical) characteristics of the in Box 6. The clinical and methodological decisions are included studies or their participants. Meta-regression extends generally based on discussions within the review team and the idea of subgroup analysis to the examination of the may be more subjective. quantitative influence of study characteristics on the effect size Clinical considerations will be influenced by the [126]. Meta-regression also allows authors to examine the question the review is attempting to address. Broad contribution of different variables to the heterogeneity in study questions might provide more ‘‘license’’ to combine more findings. Readers of systematic reviews should be aware that disparate studies, such as whether ‘‘Ritalin is effective in meta-regression has many limitations, including a danger of increasing focused attention in people diagnosed with over-interpretation of findings [127,128]. attention deficit hyperactivity disorder (ADHD).’’ Here Even with limited data, many additional analyses can be authors might elect to combine reports of studies undertaken. The choice of which analysis to undertake will depend involving children and adults. If the clinical question is on the aims of the review. None of these analyses, however, are more focused, such as whether ‘‘Ritalin is effective in exempt from producing potentially misleading results. It is increasing classroom attention in previously undiagnosed important to inform readers whether these analyses were ADHD children who have no comorbid conditions,’’ it is performed, their rationale, and which were pre-specified. likely that different decisions regarding synthesis of studies are taken by authors. In any case authors should describe their clinical decisions in the systematic review report. RESULTS Deciding whether or not to combine data also has a Item 17: STUDY SELECTION. Give numbers of studies methodological component. Reviewers may decide not to screened, assessed for eligibility, and included in the review, with combine studies of low risk of bias with those of high risk reasons for exclusions at each stage, ideally with a flow diagram. of bias (see Items 12 and 19). For example, for subjective outcomes, systematic review authors may not wish to Examples. In text: combine assessments that were completed under blind ‘‘A total of 10 studies involving 13 trials were identified for conditions with those that were not. For any particular question there may not be a ‘‘right’’ inclusion in the review. The search of Medline, PsycInfo and or ‘‘wrong’’ choice concerning synthesis, as such decisions Cinahl databases provided a total of 584 citations. After are likely complex. However, as the choice may be adjusting for duplicates 509 remained. Of these, 479 studies subjective, authors should be transparent as to their key were discarded because after reviewing the abstracts it decisions and describe them for readers. appeared that these papers clearly did not meet the criteria. Three additional studies…were discarded because full text of the study was not available or the paper could not be feasibly translated into English. The full text of the bias was considered, and only a quarter of those intended to remaining 27 citations was examined in more detail. It carry out a formal assessment for that bias [3]. Of 60 meta- appeared that 22 studies did not meet the inclusion criteria analyses in 24 articles published in 2005 in which formal as described. Five studies…met the inclusion criteria and assessments were reported, most were based on fewer than ten were included in the systematic review. An additional five studies; most displayed statistically significant heterogeneity; and studies…that met the criteria for inclusion were identified by many reviewers misinterpreted the results of the tests employed checking the references of located, relevant papers and [123]. A review of trials of antidepressants found that meta- searching for studies that have cited these papers. No analysis of only the published trials gave effect estimates 32% unpublished relevant studies were obtained.’’ [129] larger on average than when all trials sent to the drug agency See flow diagram Figure 2. were analyzed [67]. Item 16: ADDITIONAL ANALYSES. Describe methods of Explanation. Authors should report, ideally with a flow additional analyses (e.g., sensitivity or subgroup analyses, meta- diagram, the total number of records identified from electronic regression), if done, indicating which were pre-specified. bibliographic sources (including specialized database or registry searches), hand searches of various sources, reference lists, citation indices, and experts. It is useful if authors delineate for readers the Example. ‘‘Sensitivity analyses were pre-specified. The number of selected articles that were identified from the different treatment effects were examined according to quality sources so that they can see, for example, whether most articles were components (concealed treatment allocation, blinding of identified through electronic bibliographic sources or from references patients and caregivers, blinded outcome assessment), time or experts. Literature identified primarily from references or experts to initiation of statins, and the type of statin. One post-hoc may be prone to citation or publication bias [131,132]. sensitivity analysis was conducted including unpublished The flow diagram and text should describe clearly the process of data from a trial using cerivastatin.’’ [124] report selection throughout the review. Authors should report: unique records identified in searches; records excluded after Explanation. Authors may perform additional analyses to preliminary screening (e.g., screening of titles and abstracts); help understand whether the results of their review are robust, all reports retrieved for detailed evaluation; potentially eligible reports of which should be reported. Such analyses include sensitivity that were not retrievable; retrieved reports that did not meet analysis, subgroup analysis, and meta-regression [125]. inclusion criteria and the primary reasons for exclusion; and the PLoS Medicine | www.plosmedicine.org 13 July 2009 | Volume 6 | Issue 7 | e1000100 Box 6. Meta-Analysis and Assessment of Consistency (Heterogeneity) geneity (see below), some consider that the use of a fixed- Meta-Analysis: Statistical Combination of the Results effect analysis is counterintuitive because their main of Multiple Studies If it is felt that studies should have assumption is violated. Others argue that it is inappropriate their results combined statistically, other issues must be to conduct any meta-analysis when there is unexplained considered because there are many ways to conduct a meta- variability across trial results. If the reviewers decide not to analysis. Different effect measures can be used for both combine the data quantitatively, a danger is that eventually binary and continuous outcomes (see Item 13). Also, there are two commonly used statistical models for combining they may end up using quasi-quantitative rules of poor data in a meta-analysis [195]. The fixed-effect model assumes validity (e.g., vote counting of how many studies have that there is a common treatment effect for all included nominally significant results) for interpreting the evidence. studies [196]; it is assumed that the observed differences in Statistical methods to combine data exist for almost any results across studies reflect random variation [196]. The complex situation that may arise in a systematic review, but random-effects model assumes that there is no common one has to be aware of their assumptions and limitations to treatment effect for all included studies but rather that the avoid misapplying or misinterpreting these methods. variation of the effects across studies follows a particular Assessment of Consistency (Heterogeneity) We expect distribution [197]. In a random-effects model it is believed some variation (inconsistency) in the results of different that the included studies represent a random sample from a studies due to chance alone. Variability in excess of that due larger population of studies addressing the question of to chance reflects true differences in the results of the trials, interest [198]. and is called ‘‘heterogeneity.’’ The conventional statistical There is no consensus about whether to use fixed- or approach to evaluating heterogeneity is a chi-squared test random-effects models, and both are in wide use. The (Cochran’s Q), but it has low power when there are few following differences have influenced some researchers studies and excessive power when there are many studies regarding their choice between them. The random-effects [202]. By contrast, the I statistic quantifies the amount of model gives more weight to the results of smaller trials than variation in results across studies beyond that expected by does the fixed-effect analysis, which may be undesirable as chance and so is preferable to Q [202,203]. I represents the small trials may be inferior and most prone to publication percentage of the total variation in estimated effects across bias. The fixed-effect model considers only within-study studies that is due to heterogeneity rather than to chance; variability whereas the random-effects model considers both some authors consider an I value less than 25% as low [202]. within- and between-study variability. This is why a fixed- However, I also suffers from large uncertainty in the effect analysis tends to give narrower confidence intervals common situation where only a few studies are available (i.e., provide greater precision) than a random-effects [204], and reporting the uncertainty in I (e.g., as the 95% analysis [110,196,199]. In the absence of any between-study confidence interval) may be helpful [145]. When there are heterogeneity, the fixed- and random-effects estimates will few studies, inferences about heterogeneity should be coincide. cautious. In addition, there are different methods for performing When considerable heterogeneity is observed, it is both types of meta-analysis [200]. Common fixed-effect advisable to consider possible reasons [205]. In particular, approaches are Mantel-Haenszel and inverse variance, the heterogeneity may be due to differences between whereas random-effects analyses usually use the DerSimo- subgroups of studies (see Item 16). Also, data extraction nian and Laird approach, although other methods exist, errors are a common cause of substantial heterogeneity in including Bayesian meta-analysis [201]. results with continuous outcomes [139]. In the presence of demonstrable between-study hetero- studies included in the review. Indeed, the most appropriate layout Examples. In text: may vary for different reviews. ‘‘Characteristics of included studies Authors should also note the presence of duplicate or Methods supplementary reports so that readers understand the number of All four studies finally selected for the review were randomised individual studies compared to the number of reports that were controlled trials published in English. The duration of the included in the review. Authors should be consistent in their use of intervention was 24 months for the RIO-North America and terms, such as whether they are reporting on counts of citations, 12 months for the RIO-Diabetes, RIO-Lipids and RIO- records, publications, or studies. We believe that reporting the Europe study. Although the last two described a period of 24 number of studies is the most important. months during which they were conducted, only the first 12- A flow diagram can be very useful; it should depict all the months results are provided. All trials had a run-in, as a single studies included based upon fulfilling the eligibility criteria, blind period before the randomisation. whether or not data have been combined for statistical analysis. Participants A recent review of 87 systematic reviews found that about half The included studies involved 6625 participants. The main included a QUOROM flow diagram [133]. The authors of this inclusion criteria entailed adults (18 years or older), with a research recommended some important ways that reviewers can body mass index greater than 27 kg/m and less than 5 kg improve the use of a flow diagram when describing the flow of variation in body weight within the three months before information throughout the review process, including a separate study entry. flow diagram for each important outcome reported [133]. Intervention All trials were multicentric. The RIO-North America was Item 18: STUDY CHARACTERISTICS. For each study, present characteristics for which data were extracted (e.g., study conducted in the USA and Canada, RIO-Europe in Europe size, PICOS, follow-up period) and provide the citation. and the USA, RIO-Diabetes in the USA and 10 other PLoS Medicine | www.plosmedicine.org 14 July 2009 | Volume 6 | Issue 7 | e1000100 Box 7. Bias Caused by Selective Publication of Studies or Results within Studies Systematic reviews aim to incorporate information from all multiple ways and the choice of presentation influenced by relevant studies. The absence of information from some the results obtained. In a study of 102 randomized trials, studies may pose a serious threat to the validity of a review. comparison of published reports with trial protocols showed Data may be incomplete because some studies were not that a median of 38% efficacy and 50% safety outcomes per published, or because of incomplete or inadequate reporting trial, respectively, were not available for meta-analysis. within a published article. These problems are often summa- Statistically significant outcomes had a higher odds of being rized as ‘‘publication bias’’ although in fact the bias arises from fully reported in publications when compared with non- non-publication of full studies and selective publication of significant outcomes for both efficacy (pooled odds ratio 2.4; results in relation to their findings. Non-publication of research 95% confidence interval 1.4 to 4.0) and safety (4.7, 1.8 to 12) findings dependent on the actual results is an important risk of data. Several other studies have had similar findings [210,211]. bias to a systematic review and meta-analysis. Detection of Missing Information Missing studies may Missing Studies Several empirical investigations have increasingly be identified from trials registries. Evidence of shown that the findings from clinical trials are more likely missing outcomes may come from comparison with the to be published if the results are statistically significant study protocol, if available, or by careful examination of (p,0.05) than if they are not [125,206,207]. For example, of published articles [11]. Study publication bias and selective 500 oncology trials with more than 200 participants for outcome reporting are difficult to exclude or verify from the which preliminary results were presented at a conference of available results, especially when few studies are available. the American Society of Clinical Oncology, 81% with p,0.05 If the available data are affected by either (or both) of the were published in full within five years compared to only above biases, smaller studies would tend to show larger 68% of those with p.0.05 [208]. estimates of the effects of the intervention. Thus one Also, among published studies, those with statistically possibility is to investigate the relation between effect size significant results are published sooner than those with non- and sample size (or more specifically, precision of the effect significant findings [209]. When some studies are missing for estimate). Graphical methods, especially the funnel plot these reasons, the available results will be biased towards [212], and analytic methods (e.g., Egger’s test) are often used exaggerating the effect of an intervention. [213,214,215], although their interpretation can be problem- atic [216,217]. Strictly speaking, such analyses investigate Missing Outcomes In many systematic reviews only some ‘‘small study bias’’; there may be many reasons why smaller of the eligible studies (often a minority) can be included in a studies have systematically different effect sizes than larger meta-analysis for a specific outcome. For some studies, the studies, of which reporting bias is just one [218]. Several outcome may not be measured or may be measured but not alternative tests for bias have also been proposed, beyond reported. The former will not lead to bias, but the latter the ones testing small study bias [215,219,220], but none can could. be considered a gold standard. Although evidence that Evidence is accumulating that selective reporting bias is smaller studies had larger estimated effects than large ones widespread and of considerable importance [42,43]. In may suggest the possibility that the available evidence is addition, data for a given outcome may be analyzed in biased, misinterpretation of such data is common [123]. Explanation. For readers to gauge the validity and different countries not specified, and RIO-Lipids in eight applicability of a systematic review’s results, they need to know unspecified different countries. something about the included studies. Such information includes The intervention received was placebo, 5 mg of rimonabant PICOS (Box 2) and specific information relevant to the review or 20 mg of rimonabant once daily in addition to a mild question. For example, if the review is examining the long-term hypocaloric diet (600 kcal/day deficit). effects of antidepressants for moderate depressive disorder, authors Outcomes should report the follow-up periods of the included studies. For Primary each included study, authors should provide a citation for the In all studies the primary outcome assessed was weight source of their information regardless of whether or not the study change from baseline after one year of treatment and the is published. This information makes it easier for interested RIO-North America study also evaluated the prevention of readers to retrieve the relevant publications or documents. weight regain between the first and second year. All studies Reporting study-level data also allows the comparison of the evaluated adverse effects, including those of any kind and main characteristics of the studies included in the review. Authors serious events. Quality of life was measured in only one should present enough detail to allow readers to make their own study, but the results were not described (RIO-Europe). judgments about the relevance of included studies. Such Secondary and additional outcomes information also makes it possible for readers to conduct their These included prevalence of metabolic syndrome after one own subgroup analyses and interpret subgroups, based on study year and change in cardiometabolic risk factors such as characteristics. blood pressure, lipid profile, etc. Authors should avoid, whenever possible, assuming information No study included mortality and costs as outcome. when it is missing from a study report (e.g., sample size, method of The timing of outcome measures was variable and could randomization). Reviewers may contact the original investigators include monthly investigations, evaluations every three to try to obtain missing information or confirm the data extracted months or a single final evaluation after one year.’’ [134] for the systematic review. If this information is not obtained, this In table: See Table 2. should be noted in the report. If information is imputed, the reader PLoS Medicine | www.plosmedicine.org 15 July 2009 | Volume 6 | Issue 7 | e1000100 Figure 2. Example Figure: Example flow diagram of study selection. DDW, Digestive Disease Week; UEGW, United European Gastroenterology Week. Reproduced with permission from [130]. doi:10.1371/journal.pmed.1000100.g002 should be told how this was done and for which items. Presenting do not generally allow for the quantity of information available in study-level data makes it possible to clearly identify unpublished electronic journals or Cochrane reviews, this should not be information obtained from the original researchers and make it accepted as an excuse for omission of important aspects of the available for the public record. methods or results of included studies, since these can, if necessary, Typically, study-level characteristics are presented as a table as be shown on a Web site. in the example in Table 2. Such presentation ensures that all Following the presentation and description of each included pertinent items are addressed and that missing or unclear study, as discussed above, reviewers usually provide a narrative information is clearly indicated. Although paper-based journals summary of the studies. Such a summary provides readers with an PLoS Medicine | www.plosmedicine.org 16 July 2009 | Volume 6 | Issue 7 | e1000100 Table 2. Example Table: Summary of included studies evaluating the efficacy of antiemetic agents in acute gastroenteritis. No. of Source Setting Patients Age Range Inclusion Criteria Antiemetic Agent Route Follow-Up Freedman et al., 2006 ED 214 6 months–10 years GE with mild to moderate Ondansetron PO 1–2 weeks dehydration and vomiting in the preceding 4 hours Reeves et al., 2002 ED 107 1 month–22 years GE and vomiting requiring IV Ondansetron IV 5–7 days rehydration Roslund et al., 2007 ED 106 1–10 years GE with failed oral rehydration Ondansetron PO 1 week attempt in ED Stork et al., 2006 ED 137 6 months–12 years GE, recurrent emesis, mild Ondansetron and IV 1 and 2 days to moderate dehydration, dexamethasone and failed oral hydration ED, emergency department; GE, gastroenteritis; IV, intravenous; PO, by mouth. Adapted from [135]. doi:10.1371/journal.pmed.1000100.t002 overview of the included studies. It may for example address the Cochrane Collaboration’s new tool for assessing the risk of bias languages of the published papers, years of publication, and also requests that authors substantiate these assessments with any geographic origins of the included studies. relevant text from the original studies [11]. It is often easiest to The PICOS framework is often helpful in reporting the narrative provide these data in a tabular format, as in the example. summary indicating, for example, the clinical characteristics and However, a narrative summary describing the tabular data can disease severity of the participants and the main features of the also be helpful for readers. intervention and of the comparison group. For non-pharmacolog- Item 20: RESULTS OF INDIVIDUAL STUDIES. For all ical interventions, it may be helpful to specify for each study the key outcomes considered (benefits and harms), present, for each study: elements of the intervention received by each group. Full details of (a) simple summary data for each intervention group and (b) effect the interventions in included studies were reported in only three of estimates and confidence intervals, ideally with a forest plot. 25 systematic reviews relevant to general practice [84]. Item 19: RISK OF BIAS WITHIN STUDIES. Present data Examples. See Table 4 and Figure 3. on risk of bias of each study and, if available, any outcome-level assessment (see Item 12). Explanation. Publication of summary data from individual studies allows the analyses to be reproduced and other analyses and graphical displays to be investigated. Others may wish to Example. See Table 3. assess the impact of excluding particular studies or consider Explanation. We recommend that reviewers assess the risk of subgroup analyses not reported by the review authors. Displaying bias in the included studies using a standard approach with the results of each treatment group in included studies also enables defined criteria (see Item 12). They should report the results of any inspection of individual study features. For example, if only odds such assessments [89]. ratios are provided, readers cannot assess the variation in event rates across the studies, making the odds ratio impossible to Reporting only summary data (e.g., ‘‘two of eight trials adequately concealed allocation’’) is inadequate because it fails interpret [138]. Additionally, because data extraction errors in to inform readers which studies had the particular methodological meta-analyses are common and can be large [139], the presentation of the results from individual studies makes it easier shortcoming. A more informative approach is to explicitly report the methodological features evaluated for each study. The to identify errors. For continuous outcomes, readers may wish to Table 3. Example Table: Quality measures of the randomized controlled trials that failed to fulfill any one of six markers of validity. Concealment of RCT Stopped Patients Health Care Data Collectors Outcome Trials Randomisation Early Blinded Providers Blinded Blinded Assessors Blinded Liu No No Yes Yes Yes Yes Stone Yes No No Yes Yes Yes Polderman Yes Yes No No No Yes Zaugg Yes No No No Yes Yes Urban Yes Yes No No, except Yes Yes anesthesiologists RCT, randomized controlled trial. Adapted from [96]. doi:10.1371/journal.pmed.1000100.t003 PLoS Medicine | www.plosmedicine.org 17 July 2009 | Volume 6 | Issue 7 | e1000100 reviewers. For example, the standard deviation may be imputed Table 4. Example Table: Heterotopic ossification in trials using the typical standard deviations in the other trials [116,117] comparing radiotherapy to non-steroidal anti-inflammatory (see Item 14). Whenever relevant, authors should indicate which drugs after major hip procedures and fractures. results were not reported directly and had to be estimated from other information (see Item 13). In addition, the inclusion of unpublished data should be noted. Author (Year) Radiotherapy NSAID For all included studies it is important to present the estimated Kienapfel (1999) 12/49 24.5% 20/55 36.4% effect with a confidence interval. This information may be Sell (1998) 2/77 2.6% 18/77 23.4% incorporated in a table showing study characteristics or may be Kolbl (1997) 39/188 20.7% 18/113 15.9% shown in a forest plot [140]. The key elements of the forest plot are Kolbl (1998) 22/46 47.8% 6/54 11.1% the effect estimates and confidence intervals for each study shown graphically, but it is preferable also to include, for each study, the Moore (1998) 9/33 27.3% 18/39 46.2% numerical group-specific summary data, the effect size and Bremen-Kuhne (1997) 9/19 47.4% 11/31 35.5% confidence interval, and the percentage weight (see second example Knelles (1997) 5/101 5.0% 46/183 25.4% [Figure 3]). For discussion of the results of meta-analysis, see Item 21. In principle, all the above information should be provided for NSAID, non-steroidal anti-inflammatory drug. Adapted from [136]. every outcome considered in the review, including both benefits doi:10.1371/journal.pmed.1000100.t004 and harms. When there are too many outcomes for full information to be included, results for the most important examine the consistency of standard deviations across studies, for outcomes should be included in the main report with other example, to be reassured that standard deviation and standard information provided as a Web appendix. The choice of the error have not been confused [138]. information to present should be justified in light of what was For each study, the summary data for each intervention group originally stated in the protocol. Authors should explicitly mention are generally given for binary outcomes as frequencies with and if the planned main outcomes cannot be presented due to lack of without the event (or as proportions such as 12/45). It is not information. There is some evidence that information on harms is sufficient to report event rates per intervention group as only rarely reported in systematic reviews, even when it is available percentages. The required summary data for continuous outcomes in the original studies [141]. Selective omission of harms results are the mean, standard deviation, and sample size for each group. biases a systematic review and decreases its ability to contribute to In reviews that examine time-to-event data, the authors should informed decision making. report the log hazard ratio and its standard error (or confidence interval) for each included study. Sometimes, essential data are Item 21: SYNTHESES OF RESULTS. Present the main missing from the reports of the included studies and cannot be results of the review. If meta-analyses are done, include for each, calculated from other data but may need to be imputed by the confidence intervals and measures of consistency. Figure 3. Example Figure: Overall failure (defined as failure of assigned regimen or relapse) with tetracycline-rifampicin versus tetracycline-streptomycin. CI, confidence interval. Reproduced with permission from [137]. doi:10.1371/journal.pmed.1000100.g003 PLoS Medicine | www.plosmedicine.org 18 July 2009 | Volume 6 | Issue 7 | e1000100 studies, in which case forest plots are of little value and may be Examples. ‘‘Mortality data were available for all six trials, seriously biased. randomizing 311 patients and reporting data for 305 Of 300 systematic reviews indexed in MEDLINE in 2004, a patients. There were no deaths reported in the three little more than half (54%) included meta-analyses, of which the respiratory syncytial virus/severe bronchiolitis trials; thus majority (91%) reported assessing for inconsistency in results. our estimate is based on three trials randomizing 232 patients, 64 of whom died. In the pooled analysis, surfactant Item 22: RISK OF BIAS ACROSS STUDIES. Present was associated with significantly lower mortality (relative results of any assessment of risk of bias across studies (see Item 15). risk = 0.7, 95% confidence interval = 0.4–0.97, P = 0.04). There was no evidence of heterogeneity (I = 0%)’’. [142] Examples. ‘‘Strong evidence of heterogeneity (I = 79%, ‘‘Because the study designs, participants, interventions, and P,0.001) was observed. To explore this heterogeneity, a reported outcome measures varied markedly, we focused on funnel plot was drawn. The funnel plot in Figure 4 shows describing the studies, their results, their applicability, and evidence of considerable asymmetry.’’ [146] their limitations and on qualitative synthesis rather than ‘‘Specifically, four sertraline trials involving 486 participants meta-analysis.’’ [143] and one citalopram trial involving 274 participants were ‘‘We detected significant heterogeneity within this compar- 2 2 reported as having failed to achieve a statistically significant ison (I = 46.6%; x = 13.11, df = 7; P = 0.07). Retrospective drug effect, without reporting mean HRSD [Hamilton Rating exploration of the heterogeneity identified one trial that Scale for Depression] scores. We were unable to find data from seemed to differ from the others. It included only small these trials on pharmaceutical company Web sites or through ulcers (wound area less than 5 cm ). Exclusion of this trial our search of the published literature. These omissions removed the statistical heterogeneity and did not affect the represent 38% of patients in sertraline trials and 23% of finding of no evidence of a difference in healing rate patients in citalopram trials. Analyses with and without between hydrocolloids and simple low adherent dressings inclusion of these trials found no differences in the patterns (relative risk = 0.98, [95% confidence interval] 0.85 to 1.12; of results; similarly, the revealed patterns do not interact with I = 0%).’’ [144] drug type. The purpose of using the data obtained from the FDA was to avoid publication bias, by including unpublished Explanation. Results of systematic reviews should be as well as published trials. Inclusion of only those sertraline and presented in an orderly manner. Initial narrative descriptions of citalopram trials for which means were reported to the FDA the evidence covered in the review (see Item 18) may tell readers would constitute a form of reporting bias similar to publication important things about the study populations and the design and bias and would lead to overestimation of drug–placebo conduct of studies. These descriptions can facilitate the differences for these drug types. Therefore, we present analyses examination of patterns across studies. They may also provide only on data for medications for which complete clinical trials’ important information about applicability of evidence, suggest the change was reported.’’ [147] likely effects of any major biases, and allow consideration, in a systematic manner, of multiple explanations for possible Explanation. Authors should present the results of any differences of findings across studies. assessments of risk of bias across studies. If a funnel plot is If authors have conducted one or more meta-analyses, they reported, authors should specify the effect estimate and measure of should present the results as an estimated effect across studies precision used, presented typically on the x-axis and y-axis, with a confidence interval. It is often simplest to show each respectively. Authors should describe if and how they have tested meta-analysis summary with the actual results of included studies the statistical significance of any possible asymmetry (see Item 15). in a forest plot (see Item 20) [140]. It should always be clear Results of any investigations of selective reporting of outcomes which of the included studies contributed to each meta-analysis. within studies (as discussed in Item 15) should also be reported. Authors should also provide, for each meta-analysis, a measure Also, we advise authors to tell readers if any pre-specified analyses of the consistency of the results from the included studies such for assessing risk of bias across studies were not completed and the as I (heterogeneity; see Box 6); a confidence interval may also reasons (e.g., too few included studies). be given for this measure [145]. If no meta-analysis was performed, the qualitative inferences should be presented as Item 23: ADDITIONAL ANALYSES. Give results of systematically as possible with an explanation of why meta- additional analyses, if done (e.g., sensitivity or subgroup analyses, analysis was not done, as in the second example above [143]. meta-regression [see Item 16]). Readers may find a forest plot, without a summary estimate, helpful in such cases. Examples. ‘‘…benefits of chondroitin were smaller in trials Authors should in general report syntheses for all the outcome with adequate concealment of allocation compared with measures they set out to investigate (i.e., those described in the trials with unclear concealment (P for interaction = 0.050), in protocol; see Item 4) to allow readers to draw their own trials with an intention-to-treat analysis compared with those conclusions about the implications of the results. Readers should that had excluded patients from the analysis (P for be made aware of any deviations from the planned analysis. interaction = 0.017), and in large compared with small trials Authors should tell readers if the planned meta-analysis was not (P for interaction = 0.022).’’ [148] thought appropriate or possible for some of the outcomes and the ‘‘Subgroup analyses according to antibody status, antiviral reasons for that decision. It may not always be sensible to give meta-analysis results and medications, organ transplanted, treatment duration, use of forest plots for each outcome. If the review addresses a broad antilymphocyte therapy, time to outcome assessment, study question, there may be a very large number of outcomes. Also, quality and other aspects of study design did not some outcomes may have been reported in only one or two demonstrate any differences in treatment effects. Multivar- PLoS Medicine | www.plosmedicine.org 19 July 2009 | Volume 6 | Issue 7 | e1000100 Figure 4. Example Figure: Example of a funnel plot showing evidence of considerable asymmetry. SE, standard error. Adapted from [146], with permission. doi:10.1371/journal.pmed.1000100.g004 iate meta-regression showed no significant difference in Example. ‘‘Overall, the evidence is not sufficiently robust CMV [cytomegalovirus] disease after allowing for potential to determine the comparative effectiveness of angioplasty confounding or effect-modification by prophylactic drug (with or without stenting) and medical treatment alone. Only used, organ transplanted or recipient serostatus in CMV 2 randomized trials with long-term outcomes and a third positive recipients and CMV negative recipients of CMV randomized trial that allowed substantial crossover of positive donors.’’ [149] treatment after 3 months directly compared angioplasty and medical treatment…the randomized trials did not Explanation. Authors should report any subgroup or evaluate enough patients or did not follow patients for a sensitivity analyses and whether or not they were pre-specified sufficient duration to allow definitive conclusions to be made (see Items 5 and 16). For analyses comparing subgroups of about clinical outcomes, such as mortality and cardiovascu- studies (e.g., separating studies of low- and high-dose aspirin), the lar or kidney failure events. authors should report any tests for interactions, as well as Some acceptable evidence from comparison of medical estimates and confidence intervals from meta-analyses within treatment and angioplasty suggested no difference in long- each subgroup. Similarly, meta-regression results (see Item 16) term kidney function but possibly better blood pressure should not be limited to p-values, but should include effect sizes control after angioplasty, an effect that may be limited to and confidence intervals [150], as the first example reported patients with bilateral atherosclerotic renal artery stenosis. above does in a table. The amount of data included in each The evidence regarding other outcomes is weak. Because the additional analysis should be specified if different from that reviewed studies did not explicitly address patients with considered in the main analyses. This information is especially rapid clinical deterioration who may need acute interven- relevant for sensitivity analyses that exclude some studies; for tion, our conclusions do not apply to this important subset of example, those with high risk of bias. patients.’’ [143] Importantly, all additional analyses conducted should be reported, not just those that were statistically significant. This information will help avoid selective outcome reporting bias Explanation. Authors should give a brief and balanced within the review as has been demonstrated in reports of summary of the nature and findings of the review. Sometimes, randomized controlled trials [42,44,121,151,152]. Results from outcomes for which little or no data were found should be noted exploratory subgroup or sensitivity analyses should be interpret- due to potential relevance for policy decisions and future research. ed cautiously, bearing in mind the potential for multiple analyses Applicability of the review’s findings, to different patients, settings, to mislead. or target audiences, for example, should be mentioned. Although there is no standard way to assess applicability simultaneously to different audiences, some systems do exist [153]. Sometimes, DISCUSSION Item 24: SUMMARY OF EVIDENCE. Summarize the main authors formally rate or assess the overall body of evidence findings, including the strength of evidence for each main addressed in the review and can present the strength of their outcome; consider their relevance to key groups (e.g., health summary recommendations tied to their assessments of the quality care providers, users, and policy makers). of evidence (e.g., the GRADE system) [10]. PLoS Medicine | www.plosmedicine.org 20 July 2009 | Volume 6 | Issue 7 | e1000100 Authors need to keep in mind that statistical significance of the Item 26: CONCLUSIONS. Provide a general interpretation effects does not always suggest clinical or policy relevance. of the results in the context of other evidence, and implications for Likewise, a non-significant result does not demonstrate that a future research. treatment is ineffective. Authors should ideally clarify trade-offs and how the values attached to the main outcomes would lead different people to make different decisions. In addition, adroit Example. Implications for practice: authors consider factors that are important in translating the ‘‘Between 1995 and 1997 five different meta-analyses of the evidence to different settings and that may modify the estimates of effect of antibiotic prophylaxis on infection and mortality effects reported in the review [153]. Patients and health care were published. All confirmed a significant reduction in providers may be primarily interested in which intervention is infections, though the magnitude of the effect varied from most likely to provide a benefit with acceptable harms, while policy one review to another. The estimated impact on overall makers and administrators may value data on organizational mortality was less evident and has generated considerable impact and resource utilization. controversy on the cost effectiveness of the treatment. Only one among the five available reviews, however, suggested Item 25: LIMITATIONS. Discuss limitations at study and that a weak association between respiratory tract infections outcome level (e.g., risk of bias), and at review level (e.g., and mortality exists and lack of sufficient statistical power incomplete retrieval of identified research, reporting bias). may have accounted for the limited effect on mortality.’’ Implications for research: Examples. Outcome level: ‘‘A logical next step for future trials would thus be the ‘‘The meta-analysis reported here combines data across comparison of this protocol against a regimen of a systemic studies in order to estimate treatment effects with more antibiotic agent only to see whether the topical component precision than is possible in a single study. The main can be dropped. We have already identified six such trials limitation of this meta-analysis, as with any overview, is that but the total number of patients so far enrolled (n = 1056) is the patient population, the antibiotic regimen and the too small for us to be confident that the two treatments are outcome definitions are not the same across studies.’’ [154] really equally effective. If the hypothesis is therefore Study and review level: considered worth testing more and larger randomised ‘‘Our study has several limitations. The quality of the studies controlled trials are warranted. Trials of this kind, however, varied. Randomization was adequate in all trials; however, 7 would not resolve the relevant issue of treatment induced of the articles did not explicitly state that analysis of data resistance. To produce a satisfactory answer to this, studies adhered to the intention-to-treat principle, which could lead with a different design would be necessary. Though a to overestimation of treatment effect in these trials, and we detailed discussion goes beyond the scope of this paper, could not assess the quality of 4 of the 5 trials reported as studies in which the intensive care unit rather than the abstracts. Analyses did not identify an association between individual patient is the unit of randomisation and in which components of quality and re-bleeding risk, and the effect the occurrence of antibiotic resistance is monitored over a size in favour of combination therapy remained statistically long period of time should be undertaken.’’ [156] significant when we excluded trials that were reported as abstracts. Explanation. Systematic reviewers sometimes draw Publication bias might account for some of the effect we conclusions that are too optimistic [157] or do not consider observed. Smaller trials are, in general, analyzed with less the harms equally as carefully as the benefits, although some methodological rigor than larger studies, and an asymmet- evidence suggests these problems are decreasing [158]. If rical funnel plot suggests that selective reporting may have conclusions cannot be drawn because there are too few reliable led to an overestimation of effect sizes in small trials.’’ [155] studies, or too much uncertainty, this should be stated. Such a finding can be as important as finding consistent effects from Explanation. A discussion of limitations should address the several large studies. validity (i.e., risk of bias) and reporting (informativeness) of the Authors should try to relate the results of the review to other included studies, limitations of the review process, and evidence, as this helps readers to better interpret the results. For generalizability (applicability) of the review. Readers may find it example, there may be other systematic reviews about the same helpful if authors discuss whether studies were threatened by general topic that have used different methods or have addressed serious risks of bias, whether the estimates of the effect of the related but slightly different questions [159,160]. Similarly, there intervention are too imprecise, or if there were missing data for may be additional information relevant to decision makers, such as many participants or important outcomes. the cost-effectiveness of the intervention (e.g., health technology Limitations of the review process might include limitations of the assessment). Authors may discuss the results of their review in the search (e.g., restricting to English-language publications), and any context of existing evidence regarding other interventions. difficulties in the study selection, appraisal, and meta-analysis We advise authors also to make explicit recommendations for processes. For example, poor or incomplete reporting of study future research. In a sample of 2,535 Cochrane reviews, 82% designs, patient populations, and interventions may hamper included recommendations for research with specific interventions, interpretation and synthesis of the included studies [84]. Applica- 30% suggested the appropriate type of participants, and 52% bility of the review may be affected if there are limited data for suggested outcome measures for future research [161]. There is no certain populations or subgroups where the intervention might corresponding assessment about systematic reviews published in medical journals, but we believe that such recommendations are perform differently or few studies assessing the most important outcomes of interest; or if there is a substantial amount of data much less common in those reviews. relating to an outdated intervention or comparator or heavy reliance Clinical research should not be planned without a thorough on imputation of missing values for summary estimates (Item 14). knowledge of similar, existing research [162]. There is evidence PLoS Medicine | www.plosmedicine.org 21 July 2009 | Volume 6 | Issue 7 | e1000100 that this still does not occur as it should and that authors of here. A useful principle is for systematic review authors to ensure primary studies do not consider a systematic review when they that their methods are reported with adequate clarity and design their studies [163]. We believe systematic reviews have transparency to enable readers to critically judge the available great potential for guiding future clinical research. evidence and replicate or update the research. In some systematic reviews, the authors will seek the raw data from the original researchers to calculate the summary statistics. FUNDING These systematic reviews are called individual patient (or Item 27: FUNDING. Describe sources of funding or other participant) data reviews [40,41]. Individual patient data meta- support (e.g., supply of data) for the systematic review; role of analyses may also be conducted with prospective accumulation of funders for the systematic review. data rather than retrospective accumulation of existing data. Here too, extra information about the methods will need to be reported. Examples: ‘‘The evidence synthesis upon which this article Other types of systematic reviews exist. Realist reviews aim to was based was funded by the Centers for Disease Control determine how complex programs work in specific contexts and and Prevention for the Agency for Healthcare Research and settings [174]. Meta-narrative reviews aim to explain complex Quality and the U.S. Prevention Services Task Force.’’ bodies of evidence through mapping and comparing different [164] over-arching storylines [175]. Network meta-analyses, also known ‘‘Role of funding source: the funders played no role in study as multiple treatments meta-analyses, can be used to analyze data design, collection, analysis, interpretation of data, writing of from comparisons of many different treatments [176,177]. They the report, or in the decision to submit the paper for use both direct and indirect comparisons, and can be used to publication. They accept no responsibility for the contents.’’ compare interventions that have not been directly compared. [165] We believe that the issues we have highlighted in this paper are relevant to ensure transparency and understanding of the Explanation. Authors of systematic reviews, like those of any processes adopted and the limitations of the information presented other research study, should disclose any funding they received to in systematic reviews of different types. We hope that PRISMA carry out the review, or state if the review was not funded. Lexchin can be the basis for more detailed guidance on systematic reviews and colleagues [166] observed that outcomes of reports of of other types of research, including diagnostic accuracy and randomized trials and meta-analyses of clinical trials funded by epidemiological studies. the pharmaceutical industry are more likely to favor the sponsor’s product compared to studies with other sources of funding. Similar Discussion results have been reported elsewhere [167,168]. Analogous data suggest that similar biases may affect the conclusions of systematic We developed the PRISMA Statement using an approach for reviews [169]. developing reporting guidelines that has evolved over several years Given the potential role of systematic reviews in decision [178]. The overall aim of PRISMA is to help ensure the clarity making, we believe authors should be transparent about the and transparency of reporting of systematic reviews, and recent funding and the role of funders, if any. Sometimes the funders will data indicate that this reporting guidance is much needed [3]. provide services, such as those of a librarian to complete the PRISMA is not intended to be a quality assessment tool and it searches for relevant literature or access to commercial databases should not be used as such. not available to the reviewers. Any level of funding or services This PRISMA Explanation and Elaboration document was provided to the systematic review team should be reported. developed to facilitate the understanding, uptake, and dissemina- Authors should also report whether the funder had any role in the tion of the PRISMA Statement and hopefully provide a conduct or report of the review. Beyond funding issues, authors pedagogical framework for those interested in conducting and should report any real or perceived conflicts of interest related to reporting systematic reviews. It follows a format similar to that their role or the role of the funder in the reporting of the used in other explanatory documents [17,18,19]. Following the systematic review [170]. recommendations in the PRISMA checklist may increase the word In a survey of 300 systematic reviews published in November count of a systematic review report. We believe, however, that the 2004, funding sources were not reported in 41% of the benefit of readers being able to critically appraise a clear, reviews [3]. Only a minority of reviews (2%) reported being complete, and transparent systematic review report outweighs funded by for-profit sources, but the true proportion may be the possible slight increase in the length of the report. higher [171]. While the aims of PRISMA are to reduce the risk of flawed reporting of systematic reviews and improve the clarity and Additional Considerations for Systematic Reviews transparency in how reviews are conducted, we have little data to of Non-Randomized Intervention Studies or for state more definitively whether this ‘‘intervention’’ will achieve its Other Types of Systematic Reviews intended goal. A previous effort to evaluate QUOROM was not successfully completed [178]. Publication of the QUOROM The PRISMA Statement and this document have focused on Statement was delayed for two years while a research team systematic reviews of reports of randomized trials. Other study attempted to evaluate its effectiveness by conducting a randomized designs, including non-randomized studies, quasi-experimental controlled trial with the participation of eight major medical studies, and interrupted time series, are included in some journals. Unfortunately that trial was not completed due to accrual systematic reviews that evaluate the effects of health care problems (David Moher, personal communication). Other evalu- interventions [172,173]. The methods of these reviews may differ ation methods might be easier to conduct. At least one survey of to varying degrees from the typical intervention review, for example regarding the literature search, data abstraction, 139 published systematic reviews in the critical care literature assessment of risk of bias, and analysis methods. As such, their [179] suggests that their quality improved after the publication of reporting demands might also differ from what we have described QUOROM. PLoS Medicine | www.plosmedicine.org 22 July 2009 | Volume 6 | Issue 7 | e1000100 If the PRISMA Statement is endorsed by and adhered to in guidelines, such as CONSORT. We also encourage editors of journals, as other reporting guidelines have been health care journals to support PRISMA by updating their [17,18,19,180], there should be evidence of improved reporting ‘‘Instructions to Authors’’ and including the PRISMA Web of systematic reviews. For example, there have been several address, and by raising awareness through specific editorial evaluations of whether the use of CONSORT improves reports actions. of randomized controlled trials. A systematic review of these studies [181] indicates that use of CONSORT is associated Supporting Information with improved reporting of certain items, such as allocation Figure S1 Flow of information through the different phases of a concealment. We aim to evaluate the benefits (i.e., improved systematic review (downloadable template document for research- reporting) and possible adverse effects (e.g., increased word ers to re-use). length) of PRISMA and we encourage others to consider doing Found at: doi:10.1371/journal.pmed.1000100.s001 (0.08 MB likewise. DOC) Even though we did not carry out a systematic literature search to produce our checklist, and this is indeed a limitation Text S1 Checklist of items to include when reporting a of our effort, PRISMA was nevertheless developed using an systematic review or meta-analysis (downloadable template evidence-based approach, whenever possible. Checklist items document for researchers to re-use). were included if there was evidence that not reporting the item Found at: doi:10.1371/journal.pmed.1000100.s002 (0.04 MB was associated with increased risk of bias, or where it was DOC) clear that information was necessary to appraise the reliability of a review. To keep PRISMA up-to-date and as evidence- Acknowledgments based as possible requires regular vigilance of the literature, The following people contributed to this paper: which is growing rapidly. Currently the Cochrane Methodol- Doug Altman, DSc, Centre for Statistics in Medicine (Oxford, UK); ogy Register has more than 11,000 records pertaining to the Gerd Antes, PhD, University Hospital Freiburg (Freiburg, Germany); conduct and reporting of systematic reviews and other David Atkins, MD, MPH, Health Services Research and Development evaluations of health and social care. For some checklist items, Service, Veterans Health Administration (Washington, D. C., US); such as reporting the abstract (Item 2), we have used evidence Virginia Barbour, MRCP, DPhil, PLoS Medicine (Cambridge, UK); Nick from elsewhere in the belief that the issue applies equally well Barrowman, PhD, Children’s Hospital of Eastern Ontario (Ottawa, to reporting of systematic reviews. Yet for other items, Canada); Jesse A. Berlin, ScD, Johnson & Johnson Pharmaceutical Research and Development (Titusville, New Jersey, US); Jocalyn Clark, evidence does not exist; for example, whether a training PhD, PLoS Medicine (at the time of writing, BMJ, London, UK); Mike exercise improves the accuracy and reliability of data Clarke, PhD, UK Cochrane Centre (Oxford, UK) and School of extraction. We hope PRISMA will act as a catalyst to help Nursing and Midwifery, Trinity College (Dublin, Ireland); Deborah generate further evidence that can be considered when further Cook, MD, Departments of Medicine, Clinical Epidemiology and revising the checklist in the future. Biostatistics, McMaster University (Hamilton, Canada); Roberto More than ten years have passed between the development of D’Amico, PhD, Universita` di Modena e Reggio Emilia (Modena, the QUOROM Statement and its update, the PRISMA Italy) and Centro Cochrane Italiano, Istituto Ricerche Farmacologiche Mario Negri (Milan, Italy); Jonathan J. Deeks, PhD, University of Statement. We aim to update PRISMA more frequently. We Birmingham (Birmingham, UK); P. J. Devereaux, MD, PhD, hope that the implementation of PRISMA will be better than it Departments of Medicine, Clinical Epidemiology and Biostatistics, has been for QUOROM. There are at least two reasons to be McMaster University (Hamilton, Canada); Kay Dickersin, PhD, Johns optimistic. First, systematic reviews are increasingly used by health Hopkins Bloomberg School of Public Health (Baltimore, Maryland, care providers to inform ‘‘best practice’’ patient care. Policy US); Matthias Egger, MD, Department of Social and Preventive analysts and managers are using systematic reviews to inform Medicine, University of Bern (Bern, Switzerland); Edzard Ernst, MD, health care decision making, and to better target future research. PhD, FRCP, FRCP(Edin), Peninsula Medical School (Exeter, UK); Second, we anticipate benefits from the development of the Peter C. Gøtzsche, MD, MSc, The Nordic Cochrane Centre (Copenhagen, Denmark); Jeremy Grimshaw, MBChB, PhD, FRCFP, EQUATOR Network, described below. Ottawa Hospital Research Institute (Ottawa, Canada); Gordon Guyatt, Developing any reporting guideline requires considerable effort, MD, Departments of Medicine, Clinical Epidemiology and Biostatistics, experience, and expertise. While reporting guidelines have been McMaster University (Hamilton, Canada); Julian Higgins, PhD, MRC successful for some individual efforts [17,18,19], there are likely Biostatistics Unit (Cambridge, UK); John P. A. Ioannidis, MD, others who want to develop reporting guidelines who possess little University of Ioannina Campus (Ioannina, Greece); Jos Kleijnen, time, experience, or knowledge as to how to do so appropriately. MD, PhD, Kleijnen Systematic Reviews Ltd (York, UK) and School The EQUATOR Network (Enhancing the QUAlity and Trans- for Public Health and Primary Care (CAPHRI), University of Maastricht (Maastricht, Netherlands); Tom Lang, MA, Tom Lang parency Of health Research) aims to help such individuals and Communications and Training (Davis, California, US); Alessandro groups by serving as a global resource for anybody interested in Liberati, MD, Universita` di Modena e Reggio Emilia (Modena, Italy) developing reporting guidelines, regardless of the focus and Centro Cochrane Italiano, Istituto Ricerche Farmacologiche Mario [7,180,182]. The overall goal of EQUATOR is to improve the Negri (Milan, Italy); Nicola Magrini, MD, NHS Centre for the quality of reporting of all health science research through the Evaluation of the Effectiveness of Health Care – CeVEAS (Modena, development and translation of reporting guidelines. Beyond this Italy); David McNamee, PhD, The Lancet (London, UK); Lorenzo aim, the network plans to develop a large Web presence by Moja, MD, MSc, Centro Cochrane Italiano, Istituto Ricerche Farm- acologiche Mario Negri (Milan, Italy); David Moher, PhD, Ottawa developing and maintaining a resource center of reporting tools, Methods Centre, Ottawa Hospital Research Institute (Ottawa, Canada); and other information for reporting research (http://www. Cynthia Mulrow, MD, MSc, Annals of Internal Medicine (Philadelphia, equator-network.org/). Pennsylvania, US); Maryann Napoli, Center for Medical Consumers We encourage health care journals and editorial groups, such as (New York, New York, US); Andy Oxman, MD, Norwegian Health the World Association of Medical Editors and the International Services Research Centre (Oslo, Norway); Ba’ Pham, MMath, Toronto Committee of Medical Journal Editors, to endorse PRISMA in Health Economics and Technology Assessment Collaborative (Toronto, much the same way as they have endorsed other reporting Canada) (at the time of the first meeting of the group, GlaxoSmithK- PLoS Medicine | www.plosmedicine.org 23 July 2009 | Volume 6 | Issue 7 | e1000100 line Canada, Mississauga, Canada); Drummond Rennie, MD, FRCP, Dr. Lorenzo Moja helped with the preparation and the several updates FACP, University of California San Francisco (San Francisco, of the manuscript and assisted with the preparation of the reference list. California, US); Margaret Sampson, MLIS, Children’s Hospital of Alessandro Liberati is the guarantor of the manuscript. Eastern Ontario (Ottawa, Canada); Kenneth F. Schulz, PhD, MBA, Family Health International (Durham, North Carolina, US); Paul G. Author Contributions Shekelle, MD, PhD, Southern California Evidence Based Practice Center (Santa Monica, California, US); Jennifer Tetzlaff, BSc, Ottawa ICMJE criteria for authorship read and met: AL DGA JT CM PCG JPAI Methods Centre, Ottawa Hospital Research Institute (Ottawa, Canada); MC PJD JK DM. Wrote the first draft of the paper: AL DGA JT JPAI David Tovey, FRCGP, The Cochrane Library, Cochrane Collabora- DM. Contributed to the writing of the paper: AL DGA JT CM PCG JPAI tion (Oxford, UK) (at the time of the first meeting of the group, BMJ, MC PJD JK DM. Concept and design of the Explanation and Elaboration London, UK); Peter Tugwell, MD, MSc, FRCPC, Institute of statement: AL DGA JT DM. Agree with the recommendations: AL DGA Population Health, University of Ottawa (Ottawa, Canada). JT CM PCG JPAI MC PJD JK DM. References 1. Canadian Institutes of Health Research (2006) Randomized controlled trials 21. Bjelakovic G, Nikolova D, Gluud LL, Simonetti RG, Gluud C (2007) registration/application checklist (12/2006). Available: http://www.cihr-irsc. Mortality in randomized trials of antioxidant supplements for primary and gc.ca/e/documents/rct_reg_e.pdf. Accessed 26 May 2009. secondary prevention: Systematic review and meta-analysis. JAMA 297: 2. Young C, Horton R (2005) Putting clinical trials into context. Lancet 366: 842–857. 107–108. 22. Montori VM, Wilczynski NL, Morgan D, Haynes RB (2005) Optimal search 3. Moher D,TetzlaffJ,TriccoAC, SampsonM,AltmanDG(2007) strategies for retrieving systematic reviews from Medline: Analytical survey. Epidemiology and reporting characteristics of systematic reviews. PLoS Med BMJ 330: 68. 4: e78. doi:10.1371/journal.pmed.0040078. 23. Bischoff-Ferrari HA, Willett WC, Wong JB, Giovannucci E, Dietrich T, et al. 4. Dixon E, Hameed M, Sutherland F, Cook DJ, Doig C (2005) Evaluating meta- (2005) Fracture prevention with vitamin D supplementation: A meta-analysis of analyses in the general surgical literature: A critical appraisal. Ann Surg 241: randomized controlled trials. JAMA 293: 2257–2264. 450–459. 24. Hopewell S, Clarke M, Moher D, Wager E, Middleton P, et al. (2008) 5. Hemels ME, Vicente C, Sadri H, Masson MJ, Einarson TR (2004) Quality CONSORT for reporting randomised trials in journal and conference assessment of meta-analyses of RCTs of pharmacotherapy in major depressive abstracts. Lancet 371: 281–283. disorder. Curr Med Res Opin 20: 477–484. 25. Hopewell S, Clarke M, Moher D, Wager E, Middleton P, et al. (2008) 6. Jin W, Yu R, Li W, Youping L, Ya L, et al. (2008) The reporting quality of CONSORT for reporting randomized controlled trials in journal and meta-analyses improves: A random sampling study. J Clin Epidemiol 61: conference abstracts: Explanation and elaboration. PLoS Med 5: e20. 770–775. doi:10.1371/journal.pmed.0050020. 7. Moher D, Simera I, Schulz KF, Hoey J, Altman DG (2008) Helping editors, 26. Haynes RB, Mulrow CD, Huth EJ, Altman DG, Gardner MJ (1990) More peer reviewers and authors improve the clarity, completeness and transparency informative abstracts revisited. Ann Intern Med 113: 69–76. of reporting health research. BMC Med 6: 13. 27. Mulrow CD, Thacker SB, Pugh JA (1988) A proposal for more informative 8. Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, et al. (1999) Improving abstracts of review articles. Ann Intern Med 108: 613–615. the quality of reports of meta-analyses of randomised controlled trials: The 28. Froom P, Froom J (1993) Deficiencies in structured medical abstracts. J Clin QUOROM statement. Quality of Reporting of Meta-analyses. Lancet 354: Epidemiol 46: 591–594. 1896–1900. 29. Hartley J (2000) Clarifying the abstracts of systematic literature reviews. Bull 9. Green S, Higgins JPT, Alderson P, Clarke M, Mulrow CD, et al. (2008) Med Libr Assoc 88: 332–337. Chapter 1: What is a systematic review? In: Higgins JPT, Green S, editors. 30. Hartley J, Sydes M, Blurton A (1996) Obtaining information accurately and Cochrane handbook for systematic reviews of interventions version 5.0.0 quickly: Are structured abstract more efficient? J Infor Sci 22: 349–356. [updated February 2008]. The Cochrane Collaboration. Available: http:// 31. Pocock SJ, Hughes MD, Lee RJ (1987) Statistical problems in the reporting of www.cochrane-handbook.org/. Accessed 26 May 2009. clinical trials. A survey of three medical journals. N Engl J Med 317: 426–432. 10. Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, et al. (2008) GRADE: An emerging consensus on rating quality of evidence and strength of 32. Taddio A, Pain T, Fassos FF, Boon H, Ilersich AL, et al. (1994) Quality of recommendations. BMJ 336: 924–926. nonstructured and structured abstracts of original research articles in the British 11. Higgins JPT, Altman DG (2008) Chapter 8: Assessing risk of bias in included Medical Journal, the Canadian Medical Association Journal and the Journal of studies. In: Higgins JPT, Green S, eds. Cochrane handbook for systematic the American Medical Association. CMAJ 150: 1611–1615. reviews of interventions version 5.0.0 [updated February 2008]. The Cochrane 33. Harris KC, Kuramoto LK, Schulzer M, Retallack JE (2009) Effect of school- Collaboration, Available: http://www.cochrane-handbook.org/. Accessed 26 based physical activity interventions on body mass index in children: A meta- May 2009. analysis. CMAJ 180: 719–726. 12. Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group (2008) 34. James MT, Conley J, Tonelli M, Manns BJ, MacRae J, et al. (2008) Meta- Preferred reporting items for systematic reviews and meta-analyses: The analysis: Antibiotics for prophylaxis against hemodialysis catheter-related PRISMA Statement. PLoS Med 6: e1000097. 10.1371/journal.pmed.1000097. infections. Ann Intern Med 148: 596–605. 13. Atkins D, Fink K, Slutsky J (2005) Better information for better health care: 35. Counsell C (1997) Formulating questions and locating primary studies for The Evidence-based Practice Center program and the Agency for Healthcare inclusion in systematic reviews. Ann Intern Med 127: 380–387. Research and Quality. Ann Intern Med 142: 1035–1041. 36. Gotzsche PC (2000) Why we need a broad perspective on meta-analysis. It may 14. Helfand M, Balshem H (2009) Principles for developing guidance: AHRQ and be crucially important for patients. BMJ 321: 585–586. the effective health-care program. J Clin Epidemiol, In press. 37. Grossman P, Niemann L, Schmidt S, Walach H (2004) Mindfulness-based 15. Higgins JPT, Green S (2008) Cochrane handbook for systematic reviews of stress reduction and health benefits. A meta-analysis. J Psychosom Res 57: interventions version 5.0.0 [updated February 2008]. The Cochrane 35–43. Collaboration. Available: http://www.cochrane-handbook.org/. Accessed 26 38. Brunton G, Green S, Higgins JPT, Kjeldstrøm M, Jackson N, et al. (2008) May 2009. Chapter 2: Preparing a Cochrane review. In: Higgins JPT, Green S, eds. 16. Centre for Reviews and Dissemination (2009) Systematic reviews: CRD’s Cochrane handbook for systematic reviews of interventions version 5.0.0 guidance for undertaking reviews in health care. York: University of York, [updated February 2008]. The Cochrane Collaboration, Available: http:// Available: http://www.york.ac.uk/inst/crd/systematic_reviews_book.htm. Ac- www.cochrane-handbook.org/. Accessed 26 May 2009. cessed 26 May 2009. 39. Sutton AJ, Abrams KR, Jones DR, Sheldon TA, Song F (1998) Systematic 17. Altman DG, Schulz KF, Moher D, Egger M, Davidoff F, et al. (2001) The reviews of trials and other studies. Health Technol Assess 2: 1–276. revised CONSORT statement for reporting randomized trials: Explanation 40. Ioannidis JP, Rosenberg PS, Goedert JJ, O’Brien TR (2002) Commentary: and elaboration. Ann Intern Med 134: 663–694. Meta-analysis of individual participants’ data in genetic epidemiology. 18. Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, et al. (2003) Am J Epidemiol 156: 204–210. The STARD statement for reporting studies of diagnostic accuracy: 41. Stewart LA, Clarke MJ (1995) Practical methodology of meta-analyses Explanation and elaboration. Clin Chem 49: 7–18. (overviews) using updated individual patient data. Cochrane Working Group. 19. Vandenbroucke JP, von Elm E, Altman DG, Gøtzsche PC, Mulrow CD, et al. Stat Med 14: 2057–2079. (2007) Strengthening the Reporting of Observational Studies in Epidemiology 42. Chan AW, Hrobjartsson A, Haahr MT, Gøtzsche PC, Altman DG (2004) (STROBE): Explanation and elaboration. PLoS Med 4: e297. doi:10.1371/ Empirical evidence for selective reporting of outcomes in randomized trials: journal.pmed.0040297. Comparison of protocols to published articles. JAMA 291: 2457–2465. 20. Barker A, Maratos EC, Edmonds L, Lim E (2007) Recurrence rates of video- 43. Dwan K, Altman DG, Arnaiz JA, Bloom J, Chan AW, et al. (2008) Systematic assisted thoracoscopic versus open surgery in the prevention of recurrent pneumothoraces: A systematic review of randomised and non-randomised review of the empirical evidence of study publication bias and outcome trials. Lancet 370: 329–335. reporting bias. PLoS ONE 3: e3081. doi:10.1371/journal.pone.0003081. PLoS Medicine | www.plosmedicine.org 24 July 2009 | Volume 6 | Issue 7 | e1000100 44. Silagy CA, Middleton P, Hopewell S (2002) Publishing protocols of systematic 72. Major MP, Major PW, Flores-Mir C (2006) An evaluation of search and reviews: Comparing what was done to what was planned. JAMA 287: selection methods used in dental systematic reviews published in English. J Am 2831–2834. Dent Assoc 137: 1252–1257. 45. Centre for Reviews and Dissemination (2009) Research projects. York: 73. Major MP, Major PW, Flores-Mir C (2007) Benchmarking of reported search University of York, Available: http://www.crd.york.ac.uk/crdweb. Accessed and selection methods of systematic reviews by dental speciality. Evid Based 26 May 2009. Dent 8: 66–70. 46. The Joanna Briggs Institute (2009) Protocols & work in progress. Available: 74. Shah MR, Hasselblad V, Stevenson LW, Binanay C, O’Connor CM, et al. http://www.joannabriggs.edu.au/pubs/systematic_reviews_prot.php. Ac- (2005) Impact of the pulmonary artery catheter in critically ill patients: Meta- cessed 26 May 2009. analysis of randomized clinical trials. JAMA 294: 1664–1670. 47. Bagshaw SM, McAlister FA, Manns BJ, Ghali WA (2006) Acetylcysteine in the 75. Edwards P, Clarke M, DiGuiseppi C, Pratap S, Roberts I, et al. (2002) prevention of contrast-induced nephropathy: A case study of the pitfalls in the Identification of randomized controlled trials in systematic reviews: Accuracy evolution of evidence. Arch Intern Med 166: 161–166. and reliability of screening records. Stat Med 21: 1635–1640. 48. Biondi-Zoccai GG, Lotrionte M, Abbate A, Testa L, Remigi E, et al. (2006) 76. Cooper HM, Ribble RG (1989) Influences on the outcome of literature Compliance with QUOROM and quality of reporting of overlapping meta- searches for integrative research reviews. Knowledge 10: 179–201. analyses on the role of acetylcysteine in the prevention of contrast associated 77. Mistiaen P, Poot E (2006) Telephone follow-up, initiated by a hospital-based nephropathy: Case study. BMJ 332: 202–209. health professional, for postdischarge problems in patients discharged from 49. Sacks HS, Berrier J, Reitman D, Ancona-Berk VA, Chalmers TC (1987) Meta- hospital to home. Cochrane Database Syst Rev Issue 4: CD004510. analyses of randomized controlled trials. N Engl J Med 316: 450–455. doi:10.1002/14651858.CD004510.pub3. 50. Schroth RJ, Hitchon CA, Uhanova J, Noreddin A, Taback SP, et al. (2004) 78. Jones AP, Remmington T, Williamson PR, Ashby D, Smyth RL (2005) High Hepatitis B vaccination for patients with chronic renal failure. Cochrane prevalence but low impact of data extraction and reporting errors were found Database Syst Rev Issue 3: CD003775. doi:10.1002/14651858.CD003775. in Cochrane systematic reviews. J Clin Epidemiol 58: 741–742. pub2. 79. Clarke M, Hopewell S, Juszczak E, Eisinga A, Kjeldstrom M (2006) 51. Egger M, Zellweger-Zahner T, Schneider M, Junker C, Lengeler C, et al. Compression stockings for preventing deep vein thrombosis in airline (1997) Language bias in randomised controlled trials published in English and passengers. Cochrane Database Syst Rev Issue 2: CD004002. doi:10.1002/ German. Lancet 350: 326–329. 14651858.CD004002.pub2. 52. Gregoire G, Derderian F, Le Lorier J (1995) Selecting the language of the 80. Tramer MR, Reynolds DJ, Moore RA, McQuay HJ (1997) Impact of covert publications included in a meta-analysis: Is there a Tower of Babel bias? J Clin duplicate publication on meta-analysis: A case study. BMJ 315: 635–640. Epidemiol 48: 159–163. 81. von Elm E, Poglia G, Walder B, Tramer MR (2004) Different patterns of 53. Ju¨ni P, Holenstein F, Sterne J, Bartlett C, Egger M (2002) Direction and impact duplicate publication: An analysis of articles used in systematic reviews. JAMA of language bias in meta-analyses of controlled trials: Empirical study. 291: 974–980. Int J Epidemiol 31: 115–123. 82. Gotzsche PC (1989) Multiple publication of reports of drug trials. Eur J Clin 54. Moher D, Pham B, Klassen TP, Schulz KF, Berlin JA, et al. (2000) What Pharmacol 36: 429–432. contributions do languages other than English make on the results of meta- 83. Allen C, Hopewell S, Prentice A (2005) Non-steroidal anti-inflammatory drugs analyses? J Clin Epidemiol 53: 964–972. for pain in women with endometriosis. Cochrane Database Syst Rev Issue 4: 55. Pan Z, Trikalinos TA, Kavvoura FK, Lau J, Ioannidis JP (2005) Local CD004753. doi:10.1002/14651858.CD004753.pub2. literature bias in genetic epidemiology: An empirical evaluation of the Chinese 84. Glasziou P, Meats E, Heneghan C, Shepperd S (2008) What is missing from literature. PLoS Med 2: e334. doi:10.1371/journal.pmed.0020334. descriptions of treatment in trials and reviews? BMJ 336: 1472–1474. 56. Hopewell S, McDonald S, Clarke M, Egger M (2007) Grey literature in meta- 85. Tracz MJ, Sideras K, Bolona ER, Haddad RM, Kennedy CC, et al. (2006) analyses of randomized trials of health care interventions. Cochrane Database Testosterone use in men and its effects on bone health. A systematic review and Syst Rev Issue 2: MR000010. doi:10.1002/14651858.MR000010.pub3. meta-analysis of randomized placebo-controlled trials. J Clin Endocrinol Metab 57. Melander H, Ahlqvist-Rastad J, Meijer G, Beermann B (2003) Evidence 91: 2011–2016. b(i)ased medicine—Selective reporting from studies sponsored by pharmaceu- 86. Bucher HC, Hengstler P, Schindler C, Guyatt GH (2000) Percutaneous tical industry: Review of studies in new drug applications. BMJ 326: transluminal coronary angioplasty versus medical treatment for non-acute 1171–1173. coronary heart disease: Meta-analysis of randomised controlled trials. BMJ 321: 58. Sutton AJ, Duval SJ, Tweedie RL, Abrams KR, Jones DR (2000) Empirical 73–77. assessment of effect of publication bias on meta-analyses. BMJ 320: 1574–1577. 87. Gluud LL (2006) Bias in clinical intervention research. Am J Epidemiol 163: 59. Gotzsche PC (2006) Believability of relative risks and odds ratios in abstracts: 493–501. Cross sectional study. BMJ 333: 231–234. 88. Pildal J, Hro´bjartsson A, Jorgensen KJ, Hilden J, Altman DG, et al. (2007) 60. Bhandari M, Devereaux PJ, Guyatt GH, Cook DJ, Swiontkowski MF, et al. Impact of allocation concealment on conclusions drawn from meta-analyses of (2002) An observational study of orthopaedic abstracts and subsequent full-text randomized trials. Int J Epidemiol 36: 847–857. publications. J Bone Joint Surg Am 84-A: 615–621. 89. Moja LP, Telaro E, D’Amico R, Moschetti I, Coe L, et al. (2005) Assessment of 61. Rosmarakis ES, Soteriades ES, Vergidis PI, Kasiakou SK, Falagas ME (2005) methodological quality of primary studies by systematic reviews: Results of the From conference abstract to full paper: Differences between data presented in metaquality cross sectional study. BMJ 330: 1053. conferences and journals. Faseb J 19: 673–680. 90. Moher D, Jadad AR, Tugwell P (1996) Assessing the quality of randomized 62. Toma M, McAlister FA, Bialy L, Adams D, Vandermeer B, et al. (2006) controlled trials. Current issues and future directions. Int J Technol Assess Transition from meeting abstract to full-length journal article for randomized Health Care 12: 195–208. controlled trials. JAMA 295: 1281–1287. 91. Sanderson S, Tatt ID, Higgins JP (2007) Tools for assessing quality and 63. Saunders Y, Ross JR, Broadley KE, Edmonds PM, Patel S (2004) Systematic susceptibility to bias in observational studies in epidemiology: A systematic review of bisphosphonates for hypercalcaemia of malignancy. Palliat Med 18: review and annotated bibliography. Int J Epidemiol 36: 666–676. 418–431. 92. Greenland S (1994) Invited commentary: A critical look at some popular meta- 64. Shojania KG, Sampson M, Ansari MT, Ji J, Doucette S, et al. (2007) How analytic methods. Am J Epidemiol 140: 290–296. quickly do systematic reviews go out of date? A survival analysis. Ann Intern 93. Ju¨ni P, Altman DG, Egger M (2001) Systematic reviews in health care: Med 147: 224–233. Assessing the quality of controlled clinical trials. BMJ 323: 42–46. 65. Bergerhoff K, Ebrahim S, Paletta G (2004) Do we need to consider ‘in process 94. Kunz R, Oxman AD (1998) The unpredictability paradox: Review of empirical citations’ for search strategies? 12th Cochrane Colloquium; 2–6 October 2004; comparisons of randomised and non-randomised clinical trials. BMJ 317: Ottawa, Ontario, Canada. Available: http://www.cochrane.org/colloquia/ 1185–1190. abstracts/ottawa/P-039.htm. Accessed 26 May 2009. 95. Balk EM, Bonis PA, Moskowitz H, Schmid CH, Ioannidis JP, et al. (2002) 66. Zhang L, Sampson M, McGowan J (2006) Reporting of the role of expert Correlation of quality measures with estimates of treatment effect in meta- searcher in Cochrane reviews. Evid Based Libr Info Pract 1: 3–16. analyses of randomized controlled trials. JAMA 287: 2973–2982. 67. Turner EH, Matthews AM, Linardatos E, Tell RA, Rosenthal R (2008) 96. Devereaux PJ, Beattie WS, Choi PT, Badner NH, Guyatt GH, et al. (2005) Selective publication of antidepressant trials and its influence on apparent How strong is the evidence for the use of perioperative beta blockers in non- efficacy. N Engl J Med 358: 252–260. cardiac surgery? Systematic review and meta-analysis of randomised controlled 68. Alejandria MM, Lansang MA, Dans LF, Mantaring JB (2002) Intravenous trials. BMJ 331: 313–321. immunoglobulin for treating sepsis and septic shock. Cochrane Database Syst 97. Devereaux PJ, Bhandari M, Montori VM, Manns BJ, Ghali WA, et al. (2002) Rev Issue 1: CD001090. doi:10.1002/14651858.CD001090. Double blind, you are the weakest link—Good-bye! ACP J Club 136: A11. 69. Golder S, McIntosh HM, Duffy S, Glanville J (2006) Developing efficient 98. van Nieuwenhoven CA, Buskens E, van Tiel FH, Bonten MJ (2001) search strategies to identify reports of adverse effects in MEDLINE and Relationship between methodological trial quality and the effects of selective EMBASE. Health Info Libr J 23: 3–12. digestive decontamination on pneumonia and mortality in critically ill patients. 70. Sampson M, McGowan J, Cogo E, Grimshaw J, Moher D, et al. (2009) An JAMA 286: 335–340. evidence-based practice guideline for the peer review of electronic search strategies. J Clin Epidemiol, E-pub 2009 February 18. 99. Guyatt GH, Cook D, Devereaux PJ, Meade M, Straus S (2002) Therapy. 71. Flores-Mir C, Major MP, Major PW (2006) Search and selection methodology Users’ guides to the medical literature. AMA Press. pp 55–79. of systematic reviews in orthodontics (2000–2004). Am J Orthod Dentofacial 100. Sackett DL, Gent M (1979) Controversy in counting and attributing events in Orthop 130: 214–217. clinical trials. N Engl J Med 301: 1410–1412. PLoS Medicine | www.plosmedicine.org 25 July 2009 | Volume 6 | Issue 7 | e1000100 101. Montori VM, Devereaux PJ, Adhikari NK, Burns KE, Eggert CH, et al. (2005) 130. Fuccio L, Minardi ME, Zagari RM, Grilli D, Magrini N, et al. (2007) Meta- Randomized trials stopped early for benefit: A systematic review. JAMA 294: analysis: Duration of first-line proton-pump inhibitor based triple therapy for 2203–2209. Helicobacter pylori eradication. Ann Intern Med 147: 553–562. 102. Guyatt GH, Devereaux PJ (2002) Therapy and validity: The principle of 131. Egger M, Smith GD (1998) Bias in location and selection of studies. BMJ 316: intention-to-treat. In: Guyatt GH, Rennie DR, eds. Users’ guides to the 61–66. medical literature AMA Press. pp 267–273. 132. Ravnskov U (1992) Cholesterol lowering trials in coronary heart disease: 103. Berlin JA (1997) Does blinding of readers affect the results of meta-analyses? Frequency of citation and outcome. BMJ 305: 15–19. University of Pennsylvania Meta-analysis Blinding Study Group. Lancet 350: 133. Hind D, Booth A (2007) Do health technology assessments comply with 185–186. QUOROM diagram guidance? An empirical study. BMC Med Res Methodol 104. Jadad AR, Moore RA, Carroll D, Jenkinson C, Reynolds DJ, et al. (1996) 7: 49. Assessing the quality of reports of randomized clinical trials: Is blinding 134. Curioni C, Andre C (2006) Rimonabant for overweight or obesity. Cochrane necessary? Control Clin Trials 17: 1–12. Database Syst Rev Issue 4: CD006162. doi:10.1002/14651858.CD006162. 105. Pittas AG, Siegel RD, Lau J (2004) Insulin therapy for critically ill hospitalized pub2. patients: A meta-analysis of randomized controlled trials. Arch Intern Med 164: 135. DeCamp LR, Byerley JS, Doshi N, Steiner MJ (2008) Use of antiemetic agents 2005–2011. in acute gastroenteritis: A systematic review and meta-analysis. Arch Pediatr 106. Lakhdar R, Al-Mallah MH, Lanfear DE (2008) Safety and tolerability of Adolesc Med 162: 858–865. angiotensin-converting enzyme inhibitor versus the combination of angioten- 136. Pakos EE, Ioannidis JP (2004) Radiotherapy vs. nonsteroidal anti-inflammatory sin-converting enzyme inhibitor and angiotensin receptor blocker in patients drugs for the prevention of heterotopic ossification after major hip procedures: with left ventricular dysfunction: A systematic review and meta-analysis of A meta-analysis of randomized trials. Int J Radiat Oncol Biol Phys 60: randomized controlled trials. J Card Fail 14: 181–188. 888–895. 107. Bobat R, Coovadia H, Stephen C, Naidoo KL, McKerrow N, et al. (2005) 137. Skalsky K, Yahav D, Bishara J, Pitlik S, Leibovici L, et al. (2008) Treatment of Safety and efficacy of zinc supplementation for children with HIV-1 infection human brucellosis: Systematic review and meta-analysis of randomised in South Africa: A randomised double-blind placebo-controlled trial. Lancet controlled trials. BMJ 336: 701–704. 366: 1862–1867. 138. Altman DG, Cates C (2001) The need for individual trial results in reports of 108. Deeks JJ, Altman DG (2001) Effect measures for meta-analysis of trials with systematic reviews. BMJ. Rapid response. binary outcomes. In: Egger M, Smith GD, Altman DG, eds. Systematic reviews 139. Gotzsche PC, Hrobjartsson A, Maric K, Tendal B (2007) Data extraction in healthcare: Meta-analysis in context. 2nd edition. London: BMJ Publishing errors in meta-analyses that use standardized mean differences. JAMA 298: Group. 430–437. 109. Deeks JJ (2002) Issues in the selection of a summary statistic for meta-analysis of 140. Lewis S, Clarke M (2001) Forest plots: Trying to see the wood and the trees. clinical trials with binary outcomes. Stat Med 21: 1575–1600. BMJ 322: 1479–1480. 110. Engels EA, Schmid CH, Terrin N, Olkin I, Lau J (2000) Heterogeneity and 141. Papanikolaou PN, Ioannidis JP (2004) Availability of large-scale evidence on statistical significance in meta-analysis: An empirical study of 125 meta- specific harms from systematic reviews of randomized trials. Am J Med 117: analyses. Stat Med 19: 1707–1728. 582–589. 111. Tierney JF, Stewart LA, Ghersi D, Burdett S, Sydes MR (2007) Practical 142. Duffett M, Choong K, Ng V, Randolph A, Cook DJ (2007) Surfactant therapy methods for incorporating summary time-to-event data into meta-analysis. for acute respiratory failure in children: A systematic review and meta-analysis. Trials 8: 16. Crit Care 11: R66. 112. Michiels S, Piedbois P, Burdett S, Syz N, Stewart L, et al. (2005) Meta-analysis 143. Balk E, Raman G, Chung M, Ip S, Tatsioni A, et al. (2006) Effectiveness of when only the median survival times are known: A comparison with individual management strategies for renal artery stenosis: A systematic review. Ann patient data results. Int J Technol Assess Health Care 21: 119–125. Intern Med 145: 901–912. 113. Briel M, Studer M, Glass TR, Bucher HC (2004) Effects of statins on stroke 144. Palfreyman S, Nelson EA, Michaels JA (2007) Dressings for venous leg ulcers: prevention in patients with and without coronary heart disease: A meta-analysis Systematic review and meta-analysis. BMJ 335: 244. of randomized controlled trials. Am J Med 117: 596–606. 145. Ioannidis JP, Patsopoulos NA, Evangelou E (2007) Uncertainty in heteroge- 114. Jones M, Schenkel B, Just J, Fallowfield L (2004) Epoetin alfa improves quality neity estimates in meta-analyses. BMJ 335: 914–916. of life in patients with cancer: Results of metaanalysis. Cancer 101: 1720–1732. 146. Appleton KM, Hayward RC, Gunnell D, Peters TJ, Rogers PJ, et al. (2006) 115. Elbourne DR, Altman DG, Higgins JP, Curtin F, Worthington HV, et al. Effects of n-3 long-chain polyunsaturated fatty acids on depressed mood: (2002) Meta-analyses involving cross-over trials: Methodological issues. Systematic review of published trials. Am J Clin Nutr 84: 1308–1316. Int J Epidemiol 31: 140–149. 147. Kirsch I, Deacon BJ, Huedo-Medina TB, Scoboria A, Moore TJ, et al. (2008) 116. Follmann D, Elliott P, Suh I, Cutler J (1992) Variance imputation for overviews Initial severity and antidepressant benefits: A meta-analysis of data submitted to of clinical trials with continuous response. J Clin Epidemiol 45: 769–773. the Food and Drug Administration. PLoS Med 5: e45. doi:10.1371/ 117. Wiebe N, Vandermeer B, Platt RW, Klassen TP, Moher D, et al. (2006) A journal.pmed.0050045. systematic review identifies a lack of standardization in methods for handling 148. Reichenbach S, Sterchi R, Scherer M, Trelle S, Burgi E, et al. (2007) Meta- missing variance data. J Clin Epidemiol 59: 342–353. analysis: Chondroitin for osteoarthritis of the knee or hip. Ann Intern Med 146: 118. Hrobjartsson A, Gotzsche PC (2004) Placebo interventions for all clinical 580–590. conditions. Cochrane Database Syst Rev Issue 2: CD003974. doi:10.1002/ 149. Hodson EM, Craig JC, Strippoli GF, Webster AC (2008) Antiviral medications 14651858.CD003974.pub2. for preventing cytomegalovirus disease in solid organ transplant recipients. 119. Shekelle PG, Morton SC, Maglione M, Suttorp M, Tu W, et al. (2004) Cochrane Database Syst Rev Issue 2: CD003774. doi:10.1002/ Pharmacological and surgical treatment of obesity. Evid Rep Technol Assess 14651858.CD003774.pub3. (Summ). pp 1–6. 150. Thompson SG, Higgins JP (2002) How should meta-regression analyses be 120. Chan AW, Altman DG (2005) Identifying outcome reporting bias in undertaken and interpreted? Stat Med 21: 1559–1573. randomised trials on PubMed: Review of publications and survey of authors. 151. Chan AW, Krleza-Jeric K, Schmid I, Altman DG (2004) Outcome reporting BMJ 330: 753. bias in randomized trials funded by the Canadian Institutes of Health 121. Williamson PR, Gamble C (2005) Identification and impact of outcome Research. CMAJ 171: 735–740. selection bias in meta-analysis. Stat Med 24: 1547–1561. 152. Hahn S, Williamson PR, Hutton JL, Garner P, Flynn EV (2000) Assessing the 122. Williamson PR, Gamble C, Altman DG, Hutton JL (2005) Outcome selection potential for bias in meta-analysis due to selective reporting of subgroup bias in meta-analysis. Stat Methods Med Res 14: 515–524. analyses within studies. Stat Med 19: 3325–3336. 123. Ioannidis JP, Trikalinos TA (2007) The appropriateness of asymmetry tests for 153. Green LW, Glasgow RE (2006) Evaluating the relevance, generalization, and publication bias in meta-analyses: A large survey. CMAJ 176: 1091–1096. applicability of research: Issues in external validation and translation 124. Briel M, Schwartz GG, Thompson PL, de Lemos JA, Blazing MA, et al. (2006) methodology. Eval Health Prof 29: 126–153. Effects of early treatment with statins on short-term clinical outcomes in acute 154. Liberati A, D’Amico R, Pifferi, Torri V, Brazzi L (2004) Antibiotic prophylaxis coronary syndromes: A meta-analysis of randomized controlled trials. JAMA to reduce respiratory tract infections and mortality in adults receiving intensive 295: 2046–2056. care. Cochrane Database Syst Rev Issue 1: CD000022. doi:10.1002/ 125. Song F, Eastwood AJ, Gilbody S, Duley L, Sutton AJ (2000) Publication and 14651858.CD000022.pub2. related biases. Health Technol Assess 4: 1–115. 155. Gonzalez R, Zamora J, Gomez-Camarero J, Molinero LM, Banares R, et al. 126. Schmid CH, Stark PC, Berlin JA, Landais P, Lau J (2004) Meta-regression (2008) Meta-analysis: Combination endoscopic and drug therapy to prevent detected associations between heterogeneous treatment effects and study-level, variceal rebleeding in cirrhosis. Ann Intern Med 149: 109–122. but not patient-level, factors. J Clin Epidemiol 57: 683–697. 156. D’Amico R, Pifferi S, Leonetti C, Torri V, Tinazzi A, et al. (1998) Effectiveness 127. Higgins JP, Thompson SG (2004) Controlling the risk of spurious findings from of antibiotic prophylaxis in critically ill adult patients: Systematic review of meta-regression. Stat Med 23: 1663–1682. randomised controlled trials. BMJ 316: 1275–1285. 128. Thompson SG, Higgins JP (2005) Treating individuals 4: Can meta-analysis 157. Olsen O, Middleton P, Ezzo J, Gotzsche PC, Hadhazy V, et al. (2001) Quality help target interventions at individuals most likely to benefit? Lancet 365: 341–346. of Cochrane reviews: Assessment of sample from 1998. BMJ 323: 829–832. 129. Uitterhoeve RJ, Vernooy M, Litjens M, Potting K, Bensing J, et al. (2004) 158. Hopewell S, Wolfenden L, Clarke M (2008) Reporting of adverse events in Psychosocial interventions for patients with advanced cancer—A systematic systematic reviews can be improved: Survey results. J Clin Epidemiol 61: review of the literature. Br J Cancer 91: 1050–1062. 597–602. PLoS Medicine | www.plosmedicine.org 26 July 2009 | Volume 6 | Issue 7 | e1000100 Available: http://www.york.ac.uk/inst/crd/pdf/appm.pdf. Accessed 26 May 159. Cook DJ, Reeve BK, Guyatt GH, Heyland DK, Griffith LE, et al. (1996) Stress ulcer prophylaxis in critically ill patients. Resolving discordant meta-analyses. JAMA 275: 308–314. 188. Moher D, Cook DJ, Jadad AR, Tugwell P, Moher M, et al. (1999) Assessing the 160. Jadad AR, Cook DJ, Browman GP (1997) A guide to interpreting discordant quality of reports of randomised trials: Implications for the conduct of meta- systematic reviews. CMAJ 156: 1411–1416. analyses. Health Technol Assess 3: i–iv, 1–98. 161. Clarke L, Clarke M, Clarke T (2007) How useful are Cochrane reviews in 189. Devereaux PJ, Choi PT, El-Dika S, Bhandari M, Montori VM, et al. (2004) An identifying research needs? J Health Serv Res Policy 12: 101–103. observational study found that authors of randomized controlled trials frequently use concealment of randomization and blinding, despite the failure 162. [No authors listed] (2000) World Medical Association Declaration of Helsinki: to report these methods. J Clin Epidemiol 57: 1232–1236. Ethical principles for medical research involving human subjects. JAMA 284: 190. Soares HP, Daniels S, Kumar A, Clarke M, Scott C, et al. (2004) Bad reporting 3043–3045. does not mean bad methods for randomised trials: Observational study of 163. Clarke M, Hopewell S, Chalmers I (2007) Reports of clinical trials should begin randomised controlled trials performed by the Radiation Therapy Oncology and end with up-to-date systematic reviews of other relevant evidence: A status Group. BMJ 328: 22–24. report. J R Soc Med 100: 187–190. 191. Liberati A, Himel HN, Chalmers TC (1986) A quality assessment of 164. Dube C, Rostom A, Lewin G, Tsertsvadze A, Barrowman N, et al. (2007) The randomized control trials of primary treatment of breast cancer. J Clin Oncol use of aspirin for primary prevention of colorectal cancer: A systematic review 4: 942–951. prepared for the U.S. Preventive Services Task Force. Ann Intern Med 146: 192. Moher D, Jadad AR, Nichol G, Penman M, Tugwell P, et al. (1995) Assessing 365–375. the quality of randomized controlled trials: An annotated bibliography of scales 165. Critchley J, Bates I (2005) Haemoglobin colour scale for anaemia diagnosis and checklists. Control Clin Trials 16: 62–73. where there is no laboratory: A systematic review. Int J Epidemiol 34: 193. Greenland S, O’Rourke K (2001) On the bias produced by quality scores in 1425–1434. meta-analysis, and a hierarchical view of proposed solutions. Biostatistics 2: 166. Lexchin J, Bero LA, Djulbegovic B, Clark O (2003) Pharmaceutical industry 463–471. sponsorship and research outcome and quality: Systematic review. BMJ 326: 194. Ju¨ni P, Witschi A, Bloch R, Egger M (1999) The hazards of scoring the quality 1167–1170. of clinical trials for meta-analysis. JAMA 282: 1054–1060. 167. Als-Nielsen B, Chen W, Gluud C, Kjaergard LL (2003) Association of funding 195. Fleiss JL (1993) The statistical basis of meta-analysis. Stat Methods Med Res 2: and conclusions in randomized drug trials: A reflection of treatment effect or 121–145. adverse events? JAMA 290: 921–928. 196. Villar J, Mackey ME, Carroli G, Donner A (2001) Meta-analyses in systematic 168. Peppercorn J, Blood E, Winer E, Partridge A (2007) Association between reviews of randomized controlled trials in perinatal medicine: Comparison of pharmaceutical involvement and outcomes in breast cancer clinical trials. fixed and random effects models. Stat Med 20: 3635–3647. Cancer 109: 1239–1246. 197. Lau J, Ioannidis JP, Schmid CH (1998) Summing up evidence: One answer is 169. Yank V, Rennie D, Bero LA (2007) Financial ties and concordance between not always enough. Lancet 351: 123–127. results and conclusions in meta-analyses: Retrospective cohort study. BMJ 335: 198. DerSimonian R, Laird N (1986) Meta-analysis in clinical trials. Control Clin 1202–1205. Trials 7: 177–188. 170. Jorgensen AW, Hilden J, Gøtzsche PC (2006) Cochrane reviews compared with 199. Hunter JE, Schmidt FL (2000) Fixed effects vs. random effects meta-analysis industry supported meta-analyses and other meta-analyses of the same drugs: models: Implications for cumulative research knowledge. Int J Sel Assess 8: Systematic review. BMJ 333: 782. 275–292. 171. Gotzsche PC, Hrobjartsson A, Johansen HK, Haahr MT, Altman DG, et al. 200. Deeks JJ, Altman DG, Bradburn MJ (2001) Statistical methods for examining (2007) Ghost authorship in industry-initiated randomised trials. PLoS Med 4: heterogeneity and combining results from several studies in meta-analysis. In: e19. doi:10.1371/journal.pmed.0040019. Egger M, Davey Smith G, Altman DG, eds. Systematic reviews in healthcare: 172. Akbari A, Mayhew A, Al-Alawi M, Grimshaw J, Winkens R, et al. (2008) Meta-analysis in context. London: BMJ Publishing Group. pp 285–312. Interventions to improve outpatient referrals from primary care to secondary 201. Warn DE, Thompson SG, Spiegelhalter DJ (2002) Bayesian random effects care. Cochrane Database Syst Rev Issue 2: CD005471. doi:10.1002/ meta-analysis of trials with binary outcomes: Methods for the absolute risk 14651858.CD005471.pub2. difference and relative risk scales. Stat Med 21: 1601–1623. 173. Davies P, Boruch R (2001) The Campbell Collaboration. BMJ 323: 294–295. 202. Higgins JP, Thompson SG, Deeks JJ, Altman DG (2003) Measuring 174. Pawson R, Greenhalgh T, Harvey G, Walshe K (2005) Realist review—A new inconsistency in meta-analyses. BMJ 327: 557–560. method of systematic review designed for complex policy interventions. J Health 203. Higgins JP, Thompson SG (2002) Quantifying heterogeneity in a meta- Serv Res Policy 10(Suppl 1): 21–34. analysis. Stat Med 21: 1539–1558. 175. Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O, et al. (2005) 204. Huedo-Medina TB, Sanchez-Meca J, Marin-Martinez F, Botella J (2006) Storylines of research in diffusion of innovation: A meta-narrative approach to Assessing heterogeneity in meta-analysis: Q statistic or I2 index? Psychol systematic review. Soc Sci Med 61: 417–430. Methods 11: 193–206. 176. Lumley T (2002) Network meta-analysis for indirect treatment comparisons. 205. Thompson SG, Turner RM, Warn DE (2001) Multilevel models for meta- Stat Med 21: 2313–2324. analysis, and their application to absolute risk differences. Stat Methods Med 177. Salanti G, Higgins JP, Ades AE, Ioannidis JP (2008) Evaluation of networks of Res 10: 375–392. randomized trials. Stat Methods Med Res 17: 279–301. 206. Dickersin K (2005) Publication bias: Recognising the problem, understanding 178. Altman DG, Moher D (2005) [Developing guidelines for reporting healthcare its origin and scope, and preventing harm. In: Rothstein HR, Sutton AJ, research: Scientific rationale and procedures.]. Med Clin (Barc) 125(Suppl 1): Borenstein M, eds. Publication bias in meta-analysis—Prevention, assessment 8–13. and adjustments. West Sussex: John Wiley & Sons. 356 p. 179. Delaney A, Bagshaw SM, Ferland A, Manns B, Laupland KB, et al. (2005) A 207. Scherer RW, Langenberg P, von Elm E (2007) Full publication of results systematic evaluation of the quality of meta-analyses in the critical care initially presented in abstracts. Cochrane Database Syst Rev Issue 2: literature. Crit Care 9: R575–582. MR000005. doi:10.1002/14651858.MR000005.pub3. 180. Altman DG, Simera I, Hoey J, Moher D, Schulz K (2008) EQUATOR: 208. Krzyzanowska MK, Pintilie M, Tannock IF (2003) Factors associated with Reporting guidelines for health research. Lancet 371: 1149–1150. failure to publish large randomized trials presented at an oncology meeting. 181. Plint AC, Moher D, Morrison A, Schulz K, Altman DG, et al. (2006) Does the JAMA 290: 495–501. CONSORT checklist improve the quality of reports of randomised controlled 209. Hopewell S, Clarke M (2001) Methodologists and their methods. Do trials? A systematic review. Med J Aust 185: 263–267. methodologists write up their conference presentations or is it just 15 minutes 182. Simera I, Altman DG, Moher D, Schulz KF, Hoey J (2008) Guidelines for of fame? Int J Technol Assess Health Care 17: 601–603. reporting health research: The EQUATOR network’s survey of guideline 210. Ghersi D (2006) Issues in the design, conduct and reporting of clinical trials that authors. PLoS Med 5: e139. doi:10.1371/journal.pmed.0050139. impact on the quality of decision making. PhD thesis. Sydney: School of Public 183. Last JM (2001) A dictionary of epidemiology. Oxford: Oxford University Press Health, Faculty of Medicine, University of Sydney. & International Epidemiological Association. 211. von Elm E, Rollin A, Blumle A, Huwiler K, Witschi M, et al. (2008) Publication 184. Antman EM, Lau J, Kupelnick B, Mosteller F, Chalmers TC (1992) A and non-publication of clinical trials: Longitudinal study of applications comparison of results of meta-analyses of randomized control trials and submitted to a research ethics committee. Swiss Med Wkly 138: 197–203. recommendations of clinical experts. Treatments for myocardial infarction. 212. Sterne JA, Egger M (2001) Funnel plots for detecting bias in meta-analysis: JAMA 268: 240–248. Guidelines on choice of axis. J Clin Epidemiol 54: 1046–1055. 185. Oxman AD, Guyatt GH (1993) The science of reviewing research. 213. Harbord RM, Egger M, Sterne JA (2006) A modified test for small-study effects Ann N Y Acad Sci 703: 125–133; discussion 133–124. in meta-analyses of controlled trials with binary endpoints. Stat Med 25: 186. O’Connor D, Green S, Higgins JPT (2008) Chapter 5: Defining the review 3443–3457. question and developing criteria for including studies. In: Higgins JPT, Green S, 214. Peters JL, Sutton AJ, Jones DR, Abrams KR, Rushton L (2006) Comparison of eds. Cochrane handbook for systematic reviews of interventions version 5.0.0 two methods to detect publication bias in meta-analysis. JAMA 295: 676–680. [updated February 2008]. The Cochrane Collaboration, Available: http:// 215. Rothstein HR, Sutton AJ, Borenstein M (2005) Publication bias in meta- www.cochrane-handbook.org/. Accessed 26 May 2009. analysis: Prevention, assessment and adjustments. West Sussex: John Wiley & 187. McDonagh M, Whiting P, Bradley M, Cooper J, Sutton A, et al. (2000) A Sons. systematic review of public water fluoridation. Protocol changes (Appendix M). 216. Lau J, Ioannidis JP, Terrin N, Schmid CH, Olkin I (2006) The case of the NHS Centre for Reviews and Dissemination. York: University of York, misleading funnel plot. BMJ 333: 597–600. PLoS Medicine | www.plosmedicine.org 27 July 2009 | Volume 6 | Issue 7 | e1000100 217. Terrin N, Schmid CH, Lau J (2005) In an empirical evaluation of the funnel 220. Sterne JAC, Egger M, Moher D (2008) Chapter 10: Addressing reporting plot, researchers could not visually identify publication bias. J Clin Epidemiol biases. In: Higgins JPT, Green S, eds. Cochrane handbook for systematic 58: 894–901. reviews of interventions version 5.0.0 [updated February 2008]. The Cochrane 218. Egger M, Davey Smith G, Schneider M, Minder C (1997) Bias in meta-analysis Collaboration, Available: http://www.cochrane-handbook.org/. Accessed 26 detected by a simple, graphical test. BMJ 315: 629–634. May 2009. 219. Ioannidis JP, Trikalinos TA (2007) An exploratory test for an excess of significant findings. Clin Trials 4: 245–253. PLoS Medicine | www.plosmedicine.org 28 July 2009 | Volume 6 | Issue 7 | e1000100 http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Annals of Internal Medicine Unpaywall

The PRISMA Statement for Reporting Systematic Reviews and Meta-Analyses of Studies That Evaluate Health Care Interventions: Explanation and Elaboration

Annals of Internal MedicineAug 18, 2009

Loading next page...
 
/lp/unpaywall/the-prisma-statement-for-reporting-systematic-reviews-and-meta-8HXj0VXhEM

References

References for this paper are not available at this time. We will be adding them shortly, thank you for your patience.

Publisher
Unpaywall
ISSN
0003-4819
DOI
10.7326/0003-4819-151-4-200908180-00136
Publisher site
See Article on Publisher Site

Abstract

Guidelines and Guidance The PRISMA Statement for Reporting Systematic Reviews and Meta-Analyses of Studies That Evaluate Health Care Interventions: Explanation and Elaboration 1,2 3 4 5 6 Alessandro Liberati *, Douglas G. Altman , Jennifer Tetzlaff , Cynthia Mulrow , Peter C. Gøtzsche , 7 8,9 10 11,12 4,13 John P. A. Ioannidis , Mike Clarke , P. J. Devereaux , Jos Kleijnen , David Moher 1 Universita` di Modena e Reggio Emilia, Modena, Italy, 2 Centro Cochrane Italiano, Istituto Ricerche Farmacologiche Mario Negri, Milan, Italy, 3 Centre for Statistics in Medicine, University of Oxford, Oxford, United Kingdom, 4 Ottawa Methods Centre, Ottawa Hospital Research Institute, Ottawa, Ontario, Canada, 5 Annals of Internal Medicine, Philadelphia, Pennsylvania, United States of America, 6 The Nordic Cochrane Centre, Copenhagen, Denmark, 7 Department of Hygiene and Epidemiology, University of Ioannina School of Medicine, Ioannina, Greece, 8 UK Cochrane Centre, Oxford, United Kingdom, 9 School of Nursing and Midwifery, Trinity College, Dublin, Ireland, 10 Departments of Medicine, Clinical Epidemiology and Biostatistics, McMaster University, Hamilton, Ontario, Canada, 11 Kleijnen Systematic Reviews Ltd, York, United Kingdom, 12 School for Public Health and Primary Care (CAPHRI), University of Maastricht, Maastricht, The Netherlands, 13 Department of Epidemiology and Community Medicine, Faculty of Medicine, Ottawa, Ontario, Canada Recent data suggest that at least 2,500 new systematic reviews Abstract: Systematic reviews and meta-analyses are reported in English are indexed in MEDLINE annually [3]. essential to summarize evidence relating to efficacy and Unfortunately, there is considerable evidence that key informa- safety of health care interventions accurately and reliably. tion is often poorly reported in systematic reviews, thus The clarity and transparency of these reports, however, is diminishing their potential usefulness [3,4,5,6]. As is true for all not optimal. Poor reporting of systematic reviews research, systematic reviews should be reported fully and diminishes their value to clinicians, policy makers, and transparently to allow readers to assess the strengths and other users. Since the development of the QUOROM weaknesses of the investigation [7]. That rationale led to the (QUality Of Reporting Of Meta-analysis) Statement—a development of the QUOROM (QUality Of Reporting Of Meta- reporting guideline published in 1999—there have been analyses) Statement; those detailed reporting recommendations several conceptual, methodological, and practical advanc- were published in 1999 [8]. In this paper we describe the updating es regarding the conduct and reporting of systematic reviews and meta-analyses. Also, reviews of published systematic reviews have found that key information about these studies is often poorly reported. Realizing these Citation: Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gøtzsche PC, et al. (2009) The issues, an international group that included experienced PRISMA Statement for Reporting Systematic Reviews and Meta-Analyses of authors and methodologists developed PRISMA (Preferred Studies That Evaluate Health Care Interventions: Explanation and Elaboration. PLoS Med 6(7): e1000100. doi:10.1371/journal.pmed.1000100 Reporting Items for Systematic reviews and Meta-Analy- ses) as an evolution of the original QUOROM guideline for Published July 21, 2009 systematic reviews and meta-analyses of evaluations of Copyright:  2009 Liberati et al. This is an open-access article distributed health care interventions. The PRISMA Statement con- under the terms of the Creative Commons Attribution License, which permits sists of a 27-item checklist and a four-phase flow diagram. unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. The checklist includes items deemed essential for transparent reporting of a systematic review. In this Funding: PRISMA was funded by the Canadian Institutes of Health Research; Universita di Modena e Reggio Emilia, Italy; Cancer Research UK; Clinical Evidence Explanation and Elaboration document, we explain the BMJ Knowledge; The Cochrane Collaboration; and GlaxoSmithKline, Canada. AL is meaning and rationale for each checklist item. For each funded, in part, through grants of the Italian Ministry of University (COFIN - PRIN item, we include an example of good reporting and, 2002 prot. 2002061749 and COFIN - PRIN 2006 prot. 2006062298). DGA is funded where possible, references to relevant empirical studies by Cancer Research UK. DM is funded by a University of Ottawa Research Chair. None of the sponsors had any involvement in the planning, execution, or write-up and methodological literature. The PRISMA Statement, of the PRISMA documents. Additionally, no funder played a role in drafting the this document, and the associated Web site (http://www. manuscript. prisma-statement.org/) should be helpful resources to Competing Interests: MC’s employment is as Director of the UK Cochrane improve reporting of systematic reviews and meta- Centre. He is employed by the Oxford Radcliffe Hospitals Trust on behalf of the analyses. Department of Health and the National Institute for Health Research in England. This is a fixed term contract, the renewal of which is dependent upon the value placed upon his work, that of the UK Cochrane Centre, and of The Cochrane Collaboration more widely by the Department of Health. His work involves the conduct of systematic reviews and the support of the conduct and use of Introduction systematic reviews. Therefore, work–such as this manuscript–relating to systematic reviews might have an impact on his employment. Systematic reviews and meta-analyses are essential tools for summarizing evidence accurately and reliably. They help Abbreviations: PICOS, participants, interventions, comparators, outcomes, and study design; PRISMA, Preferred Reporting Items for Systematic reviews and clinicians keep up-to-date; provide evidence for policy makers to Meta-Analyses; QUOROM, QUality Of Reporting Of Meta-analyses. judge risks, benefits, and harms of health care behaviors and * E-mail: alesslib@mailbase.it interventions; gather together and summarize related research for Provenance: Not commissioned; externally peer reviewed. In order to patients and their carers; provide a starting point for clinical encourage dissemination of the PRISMA explanatory paper, this article is freely practice guideline developers; provide summaries of previous accessible on the PLoS Medicine, Annals of Internal Medicine,and BMJ Web sites. research for funders wishing to support new research [1]; and help The authors jointly hold the copyright of this article. For details on further use see editors judge the merits of publishing reports of new studies [2]. the PRISMA Web site (http://www.prisma-statement.org/). PLoS Medicine | www.plosmedicine.org 1 July 2009 | Volume 6 | Issue 7 | e1000100 of that guidance. Our aim is to ensure clear presentation of what Box 1. Terminology was planned, done, and found in a systematic review. The terminology used to describe systematic reviews and Terminology used to describe systematic reviews and meta- meta-analyses has evolved over time and varies between analyses has evolved over time and varies across different groups of fields. Different terms have been used by different groups, researchers and authors (see Box 1). In this document we adopt the such as educators and psychologists. The conduct of a definitions used by the Cochrane Collaboration [9]. A systematic systematic review comprises several explicit and repro- review attempts to collate all empirical evidence that fits pre- ducible steps, such as identifying all likely relevant records, specified eligibility criteria to answer a specific research question. selecting eligible studies, assessing the risk of bias, It uses explicit, systematic methods that are selected to minimize extracting data, qualitative synthesis of the included bias, thus providing reliable findings from which conclusions can studies, and possibly meta-analyses. be drawn and decisions made. Meta-analysis is the use of statistical Initially this entire process was termed a meta-analysis methods to summarize and combine the results of independent and was so defined in the QUOROM Statement [8]. More studies. Many systematic reviews contain meta-analyses, but not recently, especially in health care research, there has been a all. trend towards preferring the term systematic review. If quantitative synthesis is performed, this last stage alone is The QUOROM Statement and Its Evolution into referred to as a meta-analysis. The Cochrane Collaboration uses this terminology [9], under which a meta-analysis, if PRISMA performed, is a component of a systematic review. The QUOROM Statement, developed in 1996 and published Regardless of the question addressed and the complexities in 1999 [8], was conceived as a reporting guidance for authors involved, it is always possible to complete a systematic reporting a meta-analysis of randomized trials. Since then, much review of existing data, but not always possible, or has happened. First, knowledge about the conduct and reporting desirable, to quantitatively synthesize results, due to clinical, methodological, or statistical differences across the includ- of systematic reviews has expanded considerably. For example, ed studies. Conversely, with prospective accumulation of The Cochrane Library’s Methodology Register (which includes studies and datasets where the plan is eventually to reports of studies relevant to the methods for systematic reviews) combine them, the term ‘‘(prospective) meta-analysis’’ now contains more than 11,000 entries (March 2009). Second, may make more sense than ‘‘systematic review.’’ there have been many conceptual advances, such as ‘‘outcome- For retrospective efforts, one possibility is to use the level’’ assessments of the risk of bias [10,11], that apply to term systematic review for the whole process up to the systematic reviews. Third, authors have increasingly used point when one decides whether to perform a quantitative systematic reviews to summarize evidence other than that synthesis. If a quantitative synthesis is performed, some provided by randomized trials. researchers refer to this as a meta-analysis. This definition However, despite advances, the quality of the conduct and is similar to that found in the current edition of the reporting of systematic reviews remains well short of ideal Dictionary of Epidemiology [183]. [3,4,5,6]. All of these issues prompted the need for an update While we recognize that the use of these terms is and expansion of the QUOROM Statement. Of note, recognizing inconsistent and there is residual disagreement among the that the updated statement now addresses the above conceptual members of the panel working on PRISMA, we have adopted the definitions used by the Cochrane Collaboration [9]. and methodological issues and may also have broader applicability Systematic review: A systematic review attempts to than the original QUOROM Statement, we changed the name of collate all empirical evidence that fits pre-specified the reporting guidance to PRISMA (Preferred Reporting Items for eligibility criteria to answer a specific research question. Systematic reviews and Meta-Analyses). It uses explicit, systematic methods that are selected with a view to minimizing bias, thus providing reliable findings Development of PRISMA from which conclusions can be drawn and decisions made [184,185]. The key characteristics of a systematic review The PRISMA Statement was developed by a group of 29 review are: (a) a clearly stated set of objectives with an explicit, authors, methodologists, clinicians, medical editors, and consum- reproducible methodology; (b) a systematic search that ers [12]. They attended a three-day meeting in 2005 and attempts to identify all studies that would meet the participated in extensive post-meeting electronic correspondence. eligibility criteria; (c) an assessment of the validity of the A consensus process that was informed by evidence, whenever findings of the included studies, for example through the possible, was used to develop a 27-item checklist (Table 1; see also assessment of risk of bias; and (d) systematic presentation, Text S1 for a downloadable template checklist for researchers to and synthesis, of the characteristics and findings of the re-use) and a four-phase flow diagram (Figure 1; see Figure S1 for included studies. a downloadable template document for researchers to re-use). Meta-analysis: Meta-analysis is the use of statistical Items deemed essential for transparent reporting of a systematic techniques to integrate and summarize the results of review were included in the checklist. The flow diagram originally included studies. Many systematic reviews contain meta- proposed by QUOROM was also modified to show numbers of analyses, but not all. By combining information from all identified records, excluded articles, and included studies. After 11 relevant studies, meta-analyses can provide more precise estimates of the effects of health care than those derived revisions the group approved the checklist, flow diagram, and this from the individual studies included within a review. explanatory paper. The PRISMA Statement itself provides further details regarding its background and development [12]. This accom- panying Explanation and Elaboration document explains the meaning and rationale for each checklist item. A few PRISMA on several occasions to further refine the document, which was Group participants volunteered to help draft specific items for circulated and ultimately approved by the larger PRISMA this document, and four of these (DGA, AL, DM, and JT) met Group. PLoS Medicine | www.plosmedicine.org 2 July 2009 | Volume 6 | Issue 7 | e1000100 Table 1. Checklist of items to include when reporting a systematic review (with or without meta-analysis). Section/Topic # Checklist Item Reported on Page # TITLE Title 1 Identify the report as a systematic review, meta-analysis, or both. ABSTRACT Structured summary 2 Provide a structured summary including, as applicable: background; objectives; data sources; study eligibility criteria, participants, and interventions; study appraisal and synthesis methods; results; limitations; conclusions and implications of key findings; systematic review registration number. INTRODUCTION Rationale 3 Describe the rationale for the review in the context of what is already known. Objectives 4 Provide an explicit statement of questions being addressed with reference to participants, interventions, comparisons, outcomes, and study design (PICOS). METHODS Protocol and registration 5 Indicate if a review protocol exists, if and where it can be accessed (e.g., Web address), and, if available, provide registration information including registration number. Eligibility criteria 6 Specify study characteristics (e.g., PICOS, length of follow-up) and report characteristics (e.g., years considered, language, publication status) used as criteria for eligibility, giving rationale. Information sources 7 Describe all information sources (e.g., databases with dates of coverage, contact with study authors to identify additional studies) in the search and date last searched. Search 8 Present full electronic search strategy for at least one database, including any limits used, such that it could be repeated. Study selection 9 State the process for selecting studies (i.e., screening, eligibility, included in systematic review, and, if applicable, included in the meta-analysis). Data collection process 10 Describe method of data extraction from reports (e.g., piloted forms, independently, in duplicate) and any processes for obtaining and confirming data from investigators. Data items 11 List and define all variables for which data were sought (e.g., PICOS, funding sources) and any assumptions and simplifications made. Risk of bias in individual 12 Describe methods used for assessing risk of bias of individual studies (including specification of whether this was studies done at the study or outcome level), and how this information is to be used in any data synthesis. Summary measures 13 State the principal summary measures (e.g., risk ratio, difference in means). Synthesis of results 14 Describe the methods of handling data and combining results of studies, if done, including measures of consistency (e.g., I ) for each meta-analysis. Risk of bias across studies 15 Specify any assessment of risk of bias that may affect the cumulative evidence (e.g., publication bias, selective reporting within studies). Additional analyses 16 Describe methods of additional analyses (e.g., sensitivity or subgroup analyses, meta-regression), if done, indicating which were pre-specified. RESULTS Study selection 17 Give numbers of studies screened, assessed for eligibility, and included in the review, with reasons for exclusions at each stage, ideally with a flow diagram. Study characteristics 18 For each study, present characteristics for which data were extracted (e.g., study size, PICOS, follow-up period) and provide the citations. Risk of bias within studies 19 Present data on risk of bias of each study and, if available, any outcome-level assessment (see Item 12). Results of individual 20 For all outcomes considered (benefits or harms), present, for each study: (a) simple summary data for each studies intervention group and (b) effect estimates and confidence intervals, ideally with a forest plot. Synthesis of results 21 Present results of each meta-analysis done, including confidence intervals and measures of consistency. Risk of bias across studies 22 Present results of any assessment of risk of bias across studies (see Item 15). Additional analysis 23 Give results of additional analyses, if done (e.g., sensitivity or subgroup analyses, meta-regression [see Item 16]). DISCUSSION Summary of evidence 24 Summarize the main findings including the strength of evidence for each main outcome; consider their relevance to key groups (e.g., health care providers, users, and policy makers). Limitations 25 Discuss limitations at study and outcome level (e.g., risk of bias), and at review level (e.g., incomplete retrieval of identified research, reporting bias). Conclusions 26 Provide a general interpretation of the results in the context of other evidence, and implications for future research. FUNDING Funding 27 Describe sources of funding for the systematic review and other support (e.g., supply of data); role of funders for the systematic review. doi:10.1371/journal.pmed.1000100.t001 PLoS Medicine | www.plosmedicine.org 3 July 2009 | Volume 6 | Issue 7 | e1000100 Figure 1. Flow of information through the different phases of a systematic review. doi:10.1371/journal.pmed.1000100.g001 examples by removing citations or Web addresses, or by spelling Scope of PRISMA out abbreviations.) We then explain the pertinent issue, the PRISMA focuses on ways in which authors can ensure the rationale for including the item, and relevant evidence from the transparent and complete reporting of systematic reviews and literature, whenever possible. No systematic search was carried out meta-analyses. It does not address directly or in a detailed manner to identify exemplars and evidence. We also include seven Boxes the conduct of systematic reviews, for which other guides are that provide a more comprehensive explanation of certain available [13,14,15,16]. thematic aspects of the methodology and conduct of systematic We developed the PRISMA Statement and this explanatory reviews. document to help authors report a wide array of systematic Although we focus on a minimal list of items to consider when reviews to assess the benefits and harms of a health care reporting a systematic review, we indicate places where additional intervention. We consider most of the checklist items relevant information is desirable to improve transparency of the review when reporting systematic reviews of non-randomized studies process. We present the items numerically from 1 to 27; however, assessing the benefits and harms of interventions. However, we authors need not address items in this particular order in their recognize that authors who address questions relating to reports. Rather, what is important is that the information for each etiology, diagnosis, or prognosis, for example, and who review item is given somewhere within the report. epidemiological or diagnostic accuracy studies may need to modify or incorporate additional items for their systematic The PRISMA Checklist reviews. TITLE and ABSTRACT How To Use This Paper Item 1: TITLE. Identify the report as a systematic review, meta-analysis, or both. We modeled this Explanation and Elaboration document after those prepared for other reporting guidelines [17,18,19]. To Examples. ‘‘Recurrence rates of video-assisted thoraco- maximize the benefit of this document, we encourage people to read it in conjunction with the PRISMA Statement [11]. scopic versus open surgery in the prevention of recurrent We present each checklist item and follow it with a published pneumothoraces: a systematic review of randomised and exemplar of good reporting for that item. (We edited some non-randomised trials’’ [20] PLoS Medicine | www.plosmedicine.org 4 July 2009 | Volume 6 | Issue 7 | e1000100 ‘‘Mortality in randomized trials of antioxidant supplements Box 2. Helping To Develop the Research for primary and secondary prevention: systematic review Question(s): The PICOS Approach and meta-analysis’’ [21] Formulating relevant and precise questions that can be answered in a systematic review can be complex and time Explanation. Authors should identify their report as a consuming. A structured approach for framing questions systematic review or meta-analysis. Terms such as ‘‘review’’ or that uses five components may help facilitate the process. ‘‘overview’’ do not describe for readers whether the review was This approach is commonly known by the acronym ‘‘PICOS’’ systematic or whether a meta-analysis was performed. A recent where each letter refers to a component: the patient survey found that 50% of 300 authors did not mention the terms population or the disease being addressed (P), the ‘‘systematic review’’ or ‘‘meta-analysis’’ in the title or abstract of interventions or exposure (I), the comparator group (C), their systematic review [3]. Although sensitive search strategies the outcome or endpoint (O), and the study design chosen have been developed to identify systematic reviews [22], inclusion (S) [186]. Issues relating to PICOS impact several PRISMA of the terms systematic review or meta-analysis in the title may items (i.e., Items 6, 8, 9, 10, 11, and 18). improve indexing and identification. Providing information about the population requires a We advise authors to use informative titles that make key precise definition of a group of participants (often patients), such as men over the age of 65 years, their defining information easily accessible to readers. Ideally, a title reflecting the characteristics of interest (often disease), and possibly the PICOS approach (participants, interventions, comparators, out- setting of care considered, such as an acute care hospital. comes, and study design) (see Item 11 and Box 2) may help readers as The interventions (exposures) under consideration in it provides key information about the scope of the review. Specifying the systematic review need to be transparently reported. the design(s) of the studies included, as shown in the examples, may For example, if the reviewers answer a question regarding also help some readers and those searching databases. the association between a woman’s prenatal exposure to Some journals recommend ‘‘indicative titles’’ that indicate the folic acid and subsequent offspring’s neural tube defects, topic matter of the review, while others require declarative titles reporting the dose, frequency, and duration of folic acid that give the review’s main conclusion. Busy practitioners may used in different studies is likely to be important for prefer to see the conclusion of the review in the title, but readers to interpret the review’s results and conclusions. declarative titles can oversimplify or exaggerate findings. Thus, Other interventions (exposures) might include diagnostic, many journals and methodologists prefer indicative titles as used in preventative, or therapeutic treatments, arrangements of the examples above. specific processes of care, lifestyle changes, psychosocial or educational interventions, or risk factors. Item 2: STRUCTURED SUMMARY. Provide a structured Clearly reporting the comparator (control) group summary including, as applicable: background; objectives; data intervention(s), such as usual care, drug, or placebo, is sources; study eligibility criteria, participants, and interventions; essential for readers to fully understand the selection criteria study appraisal and synthesis methods; results; limitations; of primary studies included in systematic reviews, and conclusions and implications of key findings; funding for the might be a source of heterogeneity investigators have to deal with. Comparators are often very poorly described. systematic review; and systematic review registration number. Clearly reporting what the intervention is compared with is very important and may sometimes have implications for Example. ‘‘Context: The role and dose of oral vitamin D the inclusion of studies in a review—many reviews compare supplementation in nonvertebral fracture prevention have with ‘‘standard care,’’ which is otherwise undefined; this not been well established. should be properly addressed by authors. Objective: To estimate the effectiveness of vitamin D The outcomes of the intervention being assessed, such as mortality, morbidity, symptoms, or quality of life improve- supplementation in preventing hip and nonvertebral frac- ments, should be clearly specified as they are required to tures in older persons. interpret the validity and generalizability of the systematic Data Sources: A systematic review of English and non-English review’s results. articles using MEDLINE and the Cochrane Controlled Finally, the type of study design(s) included in the Trials Register (1960–2005), and EMBASE (1991–2005). review should be reported. Some reviews only include Additional studies were identified by contacting clinical reports of randomized trials whereas others have broader experts and searching bibliographies and abstracts presented design criteria and include randomized trials and certain at the American Society for Bone and Mineral Research types of observational studies. Still other reviews, such as (1995–2004). Search terms included randomized controlled those specifically answering questions related to harms, may include a wide variety of designs ranging from cohort trial (RCT), controlled clinical trial, random allocation, studies to case reports. Whatever study designs are double-blind method, cholecalciferol, ergocalciferol, 25- included in the review, these should be reported. hydroxyvitamin D, fractures, humans, elderly, falls, and Independently from how difficult it is to identify the bone density. components of the research question, the important point is Study Selection: Only double-blind RCTs of oral vitamin D that a structured approach is preferable, and this extends supplementation (cholecalciferol, ergocalciferol) with or beyond systematic reviews of effectiveness. Ideally the without calcium supplementation vs calcium supplementa- PICOS criteria should be formulated a priori, in the tion or placebo in older persons (.60 years) that examined systematic review’s protocol, although some revisions might be required due to the iterative nature of the review process. hip or nonvertebral fractures were included. Authors are encouraged to report their PICOS criteria and Data Extraction: Independent extraction of articles by 2 whether any modifications were made during the review authors using predefined data fields, including study quality process. A useful example in this realm is the Appendix of indicators. the ‘‘Systematic Reviews of Water Fluoridation’’ undertaken Data Synthesis: All pooled analyses were based on random- by the Centre for Reviews and Dissemination [187]. effects models. Five RCTs for hip fracture (n = 9294) and 7 PLoS Medicine | www.plosmedicine.org 5 July 2009 | Volume 6 | Issue 7 | e1000100 heading, authors might describe the most important weaknesses RCTs for nonvertebral fracture risk (n = 9820) met our of included studies as well as limitations of the review process. inclusion criteria. All trials used cholecalciferol. Heteroge- Then authors should provide clear and balanced Conclusions that neity among studies for both hip and nonvertebral fracture are closely linked to the objective and findings of the review. prevention was observed, which disappeared after pooling Additionally, it would be helpful if authors included some RCTs with low-dose (400 IU/d) and higher-dose vitamin D information about funding for the review. Finally, although (700–800 IU/d), separately. A vitamin D dose of 700 to protocol registration for systematic reviews is still not common 800 IU/d reduced the relative risk (RR) of hip fracture by practice, if authors have registered their review or received a 26% (3 RCTs with 5572 persons; pooled RR, 0.74; 95% registration number, we recommend providing the registration confidence interval [CI], 0.61–0.88) and any nonvertebral information at the end of the abstract. fracture by 23% (5 RCTs with 6098 persons; pooled RR, Taking all the above considerations into account, the intrinsic 0.77; 95% CI, 0.68–0.87) vs calcium or placebo. No tension between the goal of completeness of the abstract and its significant benefit was observed for RCTs with 400 IU/d keeping into the space limit often set by journal editors is vitamin D (2 RCTs with 3722 persons; pooled RR for hip recognized as a major challenge. fracture, 1.15; 95% CI, 0.88–1.50; and pooled RR for any nonvertebral fracture, 1.03; 95% CI, 0.86–1.24). INTRODUCTION Conclusions: Oral vitamin D supplementation between 700 to Item 3: RATIONALE. Describe the rationale for the review 800 IU/d appears to reduce the risk of hip and any in the context of what is already known. nonvertebral fractures in ambulatory or institutionalized elderly persons. An oral vitamin D dose of 400 IU/d is not sufficient for fracture prevention.’’ [23] Example. ‘‘Reversing the trend of increasing weight for height in children has proven difficult. It is widely accepted Explanation. Abstracts provide key information that enables that increasing energy expenditure and reducing energy readers to understand the scope, processes, and findings of a intake form the theoretical basis for management. There- review and to decide whether to read the full report. The abstract fore, interventions aiming to increase physical activity and may be all that is readily available to a reader, for example, in a improve diet are the foundation of efforts to prevent and bibliographic database. The abstract should present a balanced treat childhood obesity. Such lifestyle interventions have and realistic assessment of the review’s findings that mirrors, albeit been supported by recent systematic reviews, as well as by briefly, the main text of the report. the Canadian Paediatric Society, the Royal College of We agree with others that the quality of reporting in abstracts Paediatrics and Child Health, and the American Academy presented at conferences and in journal publications needs of Pediatrics. However, these interventions are fraught with improvement [24,25]. While we do not uniformly favor a specific poor adherence. Thus, school-based interventions are format over another, we generally recommend structured abstracts. theoretically appealing because adherence with interven- Structured abstracts provide readers with a series of headings tions can be improved. Consequently, many local govern- pertaining to the purpose, conduct, findings, and conclusions of the ments have enacted or are considering policies that mandate systematic review being reported [26,27]. They give readers more complete information and facilitate finding information more easily increased physical activity in schools, although the effect of than unstructured abstracts [28,29,30,31,32]. such interventions on body composition has not been A highly structured abstract of a systematic review could include assessed.’’ [33] the following headings: Context (or Background); Objective (or Explanation. Readers need to understand the rationale Purpose); Data Sources; Study Selection (or Eligibility Criteria); behind the study and what the systematic review may add to Study Appraisal and Synthesis Methods (or Data Extraction and what is already known. Authors should tell readers whether their Data Synthesis); Results; Limitations; and Conclusions (or report is a new systematic review or an update of an existing one. Implications). Alternatively, a simpler structure could cover but If the review is an update, authors should state reasons for the collapse some of the above headings (e.g., label Study Selection update, including what has been added to the evidence base since and Study Appraisal as Review Methods) or omit some headings the previous version of the review. such as Background and Limitations. An ideal background or introduction that sets context for In the highly structured abstract mentioned above, authors use readers might include the following. First, authors might define the the Background heading to set the context for readers and explain importance of the review question from different perspectives (e.g., the importance of the review question. Under the Objectives public health, individual patient, or health policy). Second, authors heading, they ideally use elements of PICOS (see Box 2) to state might briefly mention the current state of knowledge and its the primary objective of the review. Under a Data Sources heading, they summarize sources that were searched, any limitations. As in the above example, information about the effects of several different interventions may be available that helps language or publication type restrictions, and the start and end dates of searches. Study Selection statements then ideally describe readers understand why potential relative benefits or harms of particular interventions need review. Third, authors might whet who selected studies using what inclusion criteria. Data Extraction Methods statements describe appraisal methods during data readers’ appetites by clearly stating what the review aims to add. abstraction and the methods used to integrate or summarize They also could discuss the extent to which the limitations of the the data. The Data Synthesis section is where the main results of existing evidence base may be overcome by the review. the review are reported. If the review includes meta-analyses, Item 4: OBJECTIVES. Provide an explicit statement of authors should provide numerical results with confidence questions being addressed with reference to participants, intervals for the most important outcomes. Ideally, they should interventions, comparisons, outcomes, and study design specify the amount of evidence in these analyses (numbers of (PICOS). studies and numbers of participants). Under a Limitations PLoS Medicine | www.plosmedicine.org 6 July 2009 | Volume 6 | Issue 7 | e1000100 Cochrane reviews revealed indirect evidence for possible selective Example. ‘‘To examine whether topical or intraluminal reporting bias for systematic reviews. Almost all (n = 43) contained antibiotics reduce catheter-related bloodstream infection, we a major change, such as the addition or deletion of outcomes, reviewed randomized, controlled trials that assessed the between the protocol and the full publication [44]. Whether (or to efficacy of these antibiotics for primary prophylaxis against what extent) the changes reflected bias, however, was not clear. catheter-related bloodstream infection and mortality com- For example, it has been rather common not to describe outcomes pared with no antibiotic therapy in adults undergoing that were not presented in any of the included studies. hemodialysis.’’ [34] Registration of a systematic review, typically with a protocol and registration number, is not yet common, but some opportunities Explanation. The questions being addressed, and the exist [45,46]. Registration may possibly reduce the risk of multiple rationale for them, are one of the most critical parts of a reviews addressing the same question [45,46,47,48], reduce systematic review. They should be stated precisely and explicitly publication bias, and provide greater transparency when updating so that readers can understand quickly the review’s scope and the systematic reviews. Of note, a survey of systematic reviews indexed potential applicability of the review to their interests [35]. in MEDLINE in November 2004 found that reports of protocol Framing questions so that they include the following five use had increased to about 46% [3] from 8% noted in previous ‘‘PICOS’’ components may improve the explicitness of review surveys [49]. The improvement was due mostly to Cochrane questions: (1) the patient population or disease being addressed reviews, which, by requirement, have a published protocol [3]. (P), (2) the interventions or exposure of interest (I), (3) the comparators (C), (4) the main outcome or endpoint of interest Item 6: ELIGIBILITY CRITERIA. Specify study charac- (O), and (5) the study designs chosen (S). For more detail teristics (e.g., PICOS, length of follow-up) and report regarding PICOS, see Box 2. characteristics (e.g., years considered, language, publication Good review questions may be narrowly focused or broad, status) used as criteria for eligibility, giving rationale. depending on the overall objectives of the review. Sometimes broad questions might increase the applicability of the results and facilitate detection of bias, exploratory analyses, and sensitivity Examples. Types of studies: ‘‘Randomised clinical trials analyses [35,36]. Whether narrowly focused or broad, precisely studying the administration of hepatitis B vaccine to CRF stated review objectives are critical as they help define other [chronic renal failure] patients, with or without dialysis. No components of the review process such as the eligibility criteria language, publication date, or publication status restrictions (Item 6) and the search for relevant literature (Items 7 and 8). were imposed…’’ Types of participants: ‘‘Participants of any age with CRF or METHODS receiving dialysis (haemodialysis or peritoneal dialysis) were Item 5: PROTOCOL AND REGISTRATION. Indicate if a considered. CRF was defined as serum creatinine greater review protocol exists, if and where it can be accessed (e.g., Web than 200 mmol/L for a period of more than six months or address) and, if available, provide registration information individuals receiving dialysis (haemodialysis or peritoneal including the registration number. dialysis)…Renal transplant patients were excluded from this review as these individuals are immunosuppressed and are Example. ‘‘Methods of the analysis and inclusion criteria receiving immunosuppressant agents to prevent rejection of were specified in advance and documented in a protocol.’’ [37] their transplanted organs, and they have essentially normal renal function…’’ Explanation. A protocol is important because it pre-specifies Types of intervention: ‘‘Trials comparing the beneficial and the objectives and methods of the systematic review. For instance, harmful effects of hepatitis B vaccines with adjuvant or a protocol specifies outcomes of primary interest, how reviewers will extract information about those outcomes, and methods that cytokine co-interventions [and] trials comparing the bene- reviewers might use to quantitatively summarize the outcome data ficial and harmful effects of immunoglobulin prophylaxis. (see Item 13). Having a protocol can help restrict the likelihood of This review was limited to studies looking at active biased post hoc decisions in review methods, such as selective immunization. Hepatitis B vaccines (plasma or recombinant outcome reporting. Several sources provide guidance about (yeast) derived) of all types, dose, and regimens versus elements to include in the protocol for a systematic review placebo, control vaccine, or no vaccine…’’ [16,38,39]. For meta-analyses of individual patient-level data, we Types of outcome measures: ‘‘Primary outcome measures: advise authors to describe whether a protocol was explicitly Seroconversion, ie, proportion of patients with adequate designed and whether, when, and how participating collaborators anti-HBs response (.10 IU/L or Sample Ratio Units). endorsed it [40,41]. Hepatitis B infections (as measured by hepatitis B core Authors may modify protocols during the research, and readers antigen (HBcAg) positivity or persistent HBsAg positivity), should not automatically consider such modifications inappropri- both acute and chronic. Acute (primary) HBV [hepatitis B ate. For example, legitimate modifications may extend the period virus] infections were defined as seroconversion to HBsAg of searches to include older or newer studies, broaden eligibility positivity or development of IgM anti-HBc. Chronic HBV criteria that proved too narrow, or add analyses if the primary infections were defined as the persistence of HBsAg for more analyses suggest that additional ones are warranted. Authors than six months or HBsAg positivity and liver biopsy should, however, describe the modifications and explain their compatible with a diagnosis or chronic hepatitis B. rationale. Secondary outcome measures: Adverse events of hepatitis Although worthwhile protocol amendments are common, one B vaccinations…[and]…mortality.’’ [50] must consider the effects that protocol modifications may have on the results of a systematic review, especially if the primary outcome is changed. Bias from selective outcome reporting in randomized Explanation. Knowledge of the eligibility criteria is essential trials has been well documented [42,43]. An examination of 47 in appraising the validity, applicability, and comprehensiveness of PLoS Medicine | www.plosmedicine.org 7 July 2009 | Volume 6 | Issue 7 | e1000100 a review. Thus, authors should unambiguously specify eligibility In addition to searching databases, authors should report the use of supplementary approaches to identify studies, such as hand criteria used in the review. Carefully defined eligibility criteria inform various steps of the review methodology. They influence searching of journals, checking reference lists, searching trials registries or regulatory agency Web sites [67], contacting the development of the search strategy and serve to ensure that studies are selected in a systematic and unbiased manner. manufacturers, or contacting authors. Authors should also report if they attempted to acquire any missing information (e.g., on study A study may be described in multiple reports, and one report may methods or results) from investigators or sponsors; it is useful to describe multiple studies. Therefore, we separate eligibility criteria describe briefly who was contacted and what unpublished into the following two components: study characteristics and report information was obtained. characteristics. Both need to be reported. Study eligibility criteria are likely to include the populations, interventions, comparators, Item 8: SEARCH. Present the full electronic search strategy outcomes, and study designs of interest (PICOS; see Box 2), as well for at least one major database, including any limits used, such that as other study-specific elements, such as specifying a minimum it could be repeated. length of follow-up. Authors should state whether studies will be excluded because they do not include (or report) specific outcomes to help readers ascertain whether the systematic review may be Examples. In text: ‘‘We used the following search terms to biased as a consequence of selective reporting [42,43]. search all trials registers and databases: immunoglobulin*; Report eligibility criteria are likely to include language of IVIG; sepsis; septic shock; septicaemia; and septicemia…’’ publication, publication status (e.g., inclusion of unpublished [68] material and abstracts), and year of publication. Inclusion or not of In appendix: ‘‘Search strategy: MEDLINE (OVID) non-English language literature [51,52,53,54,55], unpublished 01. immunoglobulins/ data, or older data can influence the effect estimates in meta- 02. immunoglobulin$.tw. analyses [56,57,58,59]. Caution may need to be exercised in 03. ivig.tw. including all identified studies due to potential differences in the 04. 1 or 2 or 3 risk of bias such as, for example, selective reporting in abstracts 05. sepsis/ [60,61,62]. 06. sepsis.tw. 07. septic shock/ Item 7: INFORMATION SOURCES. Describe all 08. septic shock.tw. information sources in the search (e.g., databases with dates of 09. septicemia/ coverage, contact with study authors to identify additional studies) and date last searched. 10. septicaemia.tw. 11. septicemia.tw. 12. 5 or 6 or 7 or 8 or 9 or 10 or 11 Example. ‘‘Studies were identified by searching electronic 13. 4 and 12 databases, scanning reference lists of articles and consulta- 14. randomized controlled trials/ tion with experts in the field and drug companies…No limits 15. randomized-controlled-trial.pt. were applied for language and foreign papers were 16. controlled-clinical-trial.pt. translated. This search was applied to Medline (1966– 17. random allocation/ Present), CancerLit (1975–Present), and adapted for Embase 18. double-blind method/ (1980–Present), Science Citation Index Expanded (1981– 19. single-blind method/ Present) and Pre-Medline electronic databases. Cochrane 20. 14 or 15 or 16 or 17 or 18 or 19 and DARE (Database of Abstracts of Reviews of Effective- 21. exp clinical trials/ ness) databases were reviewed…The last search was run on 22. clinical-trial.pt. 19 June 2001. In addition, we handsearched contents pages 23. (clin$ adj trial$).ti,ab. of Journal of Clinical Oncology 2001, European Journal of 24. ((singl$ or doubl$ or trebl$ or tripl$) adj (blind$)).ti,ab. Cancer 2001 and Bone 2001, together with abstracts printed 25. placebos/ in these journals 1999–2001. A limited update literature 26. placebo$.ti,ab. search was performed from 19 June 2001 to 31 December 27. random$.ti,ab. 2003.’’ [63] 28. 21 or 22 or 23 or 24 or 25 or 26 or 27 29. research design/ Explanation. The National Library of Medicine’s 30. comparative study/ MEDLINE database is one of the most comprehensive sources 31. exp evaluation studies/ of health care information in the world. Like any database, 32. follow-up studies/ however, its coverage is not complete and varies according to the 33. prospective studies/ field. Retrieval from any single database, even by an experienced 34. (control$ or prospective$ or volunteer$).ti,ab. searcher, may be imperfect, which is why detailed reporting is important within the systematic review. 35. 30 or 31 or 32 or 33 or 34 At a minimum, for each database searched, authors should 36. 20 or 28 or 29 or 35 report the database, platform, or provider (e.g., Ovid, Dialog, 37. 13 and 36’’ [68] PubMed) and the start and end dates for the search of each database. This information lets readers assess the currency of the Explanation. The search strategy is an essential part of the review, which is important because the publication time-lag report of any systematic review. Searches may be complicated and outdates the results of some reviews [64]. This information should iterative, particularly when reviewers search unfamiliar databases also make updating more efficient [65]. Authors should also report or their review is addressing a broad or new topic. Perusing the who developed and conducted the search [66]. search strategy allows interested readers to assess the PLoS Medicine | www.plosmedicine.org 8 July 2009 | Volume 6 | Issue 7 | e1000100 comprehensiveness and completeness of the search, and to efforts were made to resolve disagreements (e.g., by contact with replicate it. Thus, we advise authors to report their full the authors of the original studies). electronic search strategy for at least one major database. As an Item 10: DATA COLLECTION PROCESS. Describe the alternative to presenting search strategies for all databases, authors method of data extraction from reports (e.g., piloted forms, could indicate how the search took into account other databases independently by two reviewers) and any processes for obtaining searched, as index terms vary across databases. If different and confirming data from investigators. searches are used for different parts of a wider question (e.g., questions relating to benefits and questions relating to harms), we recommend authors provide at least one example of a strategy for Example. ‘‘We developed a data extraction sheet (based on each part of the objective [69]. We also encourage authors to state the Cochrane Consumers and Communication Review whether search strategies were peer reviewed as part of the Group’s data extraction template), pilot-tested it on ten systematic review process [70]. randomly-selected included studies, and refined it accord- We realize that journal restrictions vary and that having the ingly. One review author extracted the following data from search strategy in the text of the report is not always feasible. We included studies and the second author checked the strongly encourage all journals, however, to find ways, such as a extracted data…Disagreements were resolved by discussion ‘‘Web extra,’’ appendix, or electronic link to an archive, to make between the two review authors; if no agreement could be search strategies accessible to readers. We also advise all authors to reached, it was planned a third author would decide. We archive their searches so that (1) others may access and review contacted five authors for further information. All responded them (e.g., replicate them or understand why their review of a and one provided numerical data that had only been similar topic did not identify the same reports), and (2) future updates of their review are facilitated. presented graphically in the published paper.’’ [77] Several sources provide guidance on developing search strategies [71,72,73]. Most searches have constraints, for example Explanation. Reviewers extract information from each relating to limited time or financial resources, inaccessible or included study so that they can critique, present, and summarize inadequately indexed reports and databases, unavailability of evidence in a systematic review. They might also contact authors experts with particular language or database searching skills, or of included studies for information that has not been, or is review questions for which pertinent evidence is not easy to find. unclearly, reported. In meta-analysis of individual patient data, Authors should be straightforward in describing their search this phase involves collection and scrutiny of detailed raw constraints. Apart from the keywords used to identify or exclude databases. The authors should describe these methods, including records, they should report any additional limitations relevant to any steps taken to reduce bias and mistakes during data collection the search, such as language and date restrictions (see also and data extraction [78] (Box 3). eligibility criteria, Item 6) [51]. Some systematic reviewers use a data extraction form that could be reported as an appendix or ‘‘Web extra’’ to their report. These Item 9: STUDY SELECTION. State the process for selecting forms could show the reader what information reviewers sought studies (i.e., for screening, for determining eligibility, for inclusion (see Item 11) and how they extracted it. Authors could tell readers in the systematic review, and, if applicable, for inclusion in the if the form was piloted. Regardless, we advise authors to tell meta-analysis). readers who extracted what data, whether any extractions were completed in duplicate, and, if so, whether duplicate abstraction was done independently and how disagreements were resolved. Example. ‘‘Eligibility assessment…[was] performed inde- Published reports of the included studies may not provide all the pendently in an unblinded standardized manner by 2 information required for the review. Reviewers should describe reviewers…Disagreements between reviewers were resolved any actions they took to seek additional information from the by consensus.’’ [74] original researchers (see Item 7). The description might include how they attempted to contact researchers, what they asked for, Explanation. There is no standard process for selecting and their success in obtaining the necessary information. Authors studies to include in a systematic review. Authors usually start with should also tell readers when individual patient data were sought a large number of identified records from their search and from the original researchers [41] (see Item 11) and indicate the sequentially exclude records according to eligibility criteria. We studies for which such data were used in the analyses. The advise authors to report how they screened the retrieved records reviewers ideally should also state whether they confirmed the (typically a title and abstract), how often it was necessary to review accuracy of the information included in their review with the the full text publication, and if any types of record (e.g., letters to original researchers, for example, by sending them a copy of the the editor) were excluded. We also advise using the PRISMA flow draft review [79]. diagram to summarize study selection processes (see Item 17; Box Some studies are published more than once. Duplicate 3). publications may be difficult to ascertain, and their inclusion Efforts to enhance objectivity and avoid mistakes in study may introduce bias [80,81]. We advise authors to describe any selection are important. Thus authors should report whether each steps they used to avoid double counting and piece together data stage was carried out by one or several people, who these people from multiple reports of the same study (e.g., juxtaposing author were, and, whenever multiple independent investigators per- formed the selection, what the process was for resolving names, treatment comparisons, sample sizes, or outcomes). We also advise authors to indicate whether all reports on a study were disagreements. The use of at least two investigators may reduce the possibility of rejecting relevant reports [75]. The benefit may considered, as inconsistencies may reveal important limitations. For example, a review of multiple publications of drug trials be greatest for topics where selection or rejection of an article requires difficult judgments [76]. For these topics, authors should showed that reported study characteristics may differ from report ideally tell readers the level of inter-rater agreement, how to report, including the description of the design, number of commonly arbitration about selection was required, and what patients analyzed, chosen significance level, and outcomes [82]. PLoS Medicine | www.plosmedicine.org 9 July 2009 | Volume 6 | Issue 7 | e1000100 Box 3. Identification of Study Reports and Box 4. Study Quality and Risk of Bias Data Extraction In this paper, and elsewhere [11], we sought to use a new Comprehensive searches usually result in a large number term for many readers, namely, risk of bias, for evaluating of identified records, a much smaller number of studies each included study in a systematic review. Previous included in the systematic review, and even fewer of these papers [89,188] tended to use the term ‘‘quality’’. When studies included in any meta-analyses. Reports of system- carrying out a systematic review we believe it is important atic reviews often provide little detail as to the methods to distinguish between quality and risk of bias and to focus used by the review team in this process. Readers are often on evaluating and reporting the latter. Quality is often the left with what can be described as the ‘‘X-files’’ phenom- best the authors have been able to do. For example, enon, as it is unclear what occurs between the initial set of authors may report the results of surgical trials in which identified records and those finally included in the review. blinding of the outcome assessors was not part of the Sometimes, review authors simply report the number of trial’s conduct. Even though this may have been the best included studies; more often they report the initial number methodology the researchers were able to do, there are of identified records and the number of included studies. still theoretical grounds for believing that the study was Rarely, although this is optimal for readers, do review susceptible to (risk of) bias. authors report the number of identified records, the Assessing the risk of bias should be part of the conduct smaller number of potentially relevant studies, and the and reporting of any systematic review. In all situations, we even smaller number of included studies, by outcome. encourage systematic reviewers to think ahead carefully Review authors also need to differentiate between the about what risks of bias (methodological and clinical) may number of reports and studies. Often there will not be a have a bearing on the results of their systematic reviews. 1:1 ratio of reports to studies and this information needs to For systematic reviewers, understanding the risk of bias be described in the systematic review report. on the results of studies is often difficult, because the Ideally, the identification of study reports should be report is only a surrogate of the actual conduct of the reported as text in combination with use of the PRISMA study. There is some suggestion [189,190] that the report flow diagram. While we recommend use of the flow may not be a reasonable facsimile of the study, although diagram, a small number of reviews might be particularly this view is not shared by all [88,191]. There are three main simple and can be sufficiently described with a few brief ways to assess risk of bias: individual components, sentences of text. More generally, review authors will need checklists, and scales. There are a great many scales to report the process used for each step: screening the available [192], although we caution their use based on identified records; examining the full text of potentially theoretical grounds [193] and emerging empirical evi- relevant studies (and reporting the number that could not dence [194]. Checklists are less frequently used and be obtained); and applying eligibility criteria to select the potentially run the same problems as scales. We advocate included studies. using a component approach and one that is based on Such descriptions should also detail how potentially domains for which there is good empirical evidence and eligible records were promoted to the next stage of the perhaps strong clinical grounds. The new Cochrane risk of review (e.g., full text screening) and to the final stage of bias tool [11] is one such component approach. this process, the included studies. Often review teams have The Cochrane risk of bias tool consists of five items for three response options for excluding records or promoting which there is empirical evidence for their biasing them to the next stage of the winnowing process: ‘‘yes,’’ influence on the estimates of an intervention’s effective- ‘‘no,’’ and ‘‘maybe.’’ ness in randomized trials (sequence generation, allocation Similarly, some detail should be reported on who concealment, blinding, incomplete outcome data, and participated and how such processes were completed. For selective outcome reporting) and a catch-all item called example, a single person may screen the identified records ‘‘other sources of bias’’ [11]. There is also some consensus while a second person independently examines a small that these items can be applied for evaluation of studies sample of them. The entire winnowing process is one of across very diverse clinical areas [93]. Other risk of bias ‘‘good book keeping’’ whereby interested readers should items may be topic or even study specific, i.e., they may be able to work backwards from the included studies to stem from some peculiarity of the research topic or some come up with the same numbers of identified records. special feature of the design of a specific study. These There is often a paucity of information describing the peculiarities need to be investigated on a case-by-case data extraction processes in reports of systematic reviews. basis, based on clinical and methodological acumen, and Authors may simply report that ‘‘relevant’’ data were there can be no general recipe. In all situations, systematic extracted from each included study with little information reviewers need to think ahead carefully about what about the processes used for data extraction. It may be aspects of study quality may have a bearing on the results. useful for readers to know whether a systematic review’s authors developed, a priori or not, a data extraction form, Authors ideally should present any algorithm that they used to whether multiple forms were used, the number of select data from overlapping reports and any efforts they used to questions, whether the form was pilot tested, and who solve logical inconsistencies across reports. completed the extraction. For example, it is important for readers to know whether one or more people extracted Item 11: DATA ITEMS. List and define all variables for data, and if so, whether this was completed independent- which data were sought (e.g., PICOS, funding sources), and any ly, whether ‘‘consensus’’ data were used in the analyses, assumptions and simplifications made. and if the review team completed an informal training exercise or a more formal reliability exercise. Examples. ‘‘Information was extracted from each included trial on: (1) characteristics of trial participants (including age, stage and severity of disease, and method of diagnosis), and PLoS Medicine | www.plosmedicine.org 10 July 2009 | Volume 6 | Issue 7 | e1000100 Explanation. The likelihood that the treatment effect reported the trial’s inclusion and exclusion criteria; (2) type of in a systematic review approximates the truth depends on the validity intervention (including type, dose, duration and frequency of the included studies, as certain methodological characteristics may of the NSAID [non-steroidal anti-inflammatory drug]; be associated with effect sizes [87,88]. For example, trials without versus placebo or versus the type, dose, duration and reported adequate allocation concealment exaggerate treatment frequency of another NSAID; or versus another pain effects on average compared to those with adequate concealment management drug; or versus no treatment); (3) type of [88]. Therefore, it is important for authors to describe any methods outcome measure (including the level of pain reduction, that they used to gauge the risk of bias in the included studies and how improvement in quality of life score (using a validated scale), that information was used [89]. Additionally, authors should provide effect on daily activities, absence from work or school, length a rationale if no assessment of risk of bias was undertaken. The most of follow up, unintended effects of treatment, number of popular term to describe the issues relevant to this item is ‘‘quality,’’ women requiring more invasive treatment).’’ [83] but for the reasons that are elaborated in Box 4 we prefer to name this item as ‘‘assessment of risk of bias.’’ Explanation. It is important for readers to know what Many methods exist to assess the overall risk of bias in included information review authors sought, even if some of this studies, including scales, checklists, and individual components information was not available [84]. If the review is limited to [90,91]. As discussed in Box 4, scales that numerically summarize reporting only those variables that were obtained, rather than multiple components into a single number are misleading and those that were deemed important but could not be obtained, bias unhelpful [92,93]. Rather, authors should specify the methodolog- might be introduced and the reader might be misled. It is therefore ical components that they assessed. Common markers of validity for helpful if authors can refer readers to the protocol (see Item 5), and randomized trials include the following: appropriate generation of archive their extraction forms (see Item 10), including definitions random allocation sequence [94]; concealment of the allocation of variables. The published systematic review should include a sequence [93]; blinding of participants, health care providers, data description of the processes used with, if relevant, specification of collectors, and outcome adjudicators [95,96,97,98]; proportion of how readers can get access to additional materials. patients lost to follow-up [99,100]; stopping of trials early for benefit We encourage authors to report whether some variables were [101]; and whether the analysis followed the intention-to-treat added after the review started. Such variables might include those principle [100,102]. The ultimate decision regarding which found in the studies that the reviewers identified (e.g., important methodological features to evaluate requires consideration of the outcome measures that the reviewers initially overlooked). Authors strength of the empiric data, theoretical rationale, and the unique should describe the reasons for adding any variables to those circumstances of the included studies. already pre-specified in the protocol so that readers can Authors should report how they assessed risk of bias; whether understand the review process. it was in a blind manner; and if assessments were completed by We advise authors to report any assumptions they made about more than one person, and if so, whether they were completed missing or unclear information and to explain those processes. For independently [103,104]. Similarly, we encourage authors to example, in studies of women aged 50 or older it is reasonable to report any calibration exercises among review team members assume that none were pregnant, even if this is not reported. that were done. Finally, authors need to report how their Likewise, review authors might make assumptions about the route assessments of risk of bias are used subsequently in the data of administration of drugs assessed. However, special care should synthesis (see Item 16). Despite the often difficult task of be taken in making assumptions about qualitative information. For assessing the risk of bias in included studies, authors are example, the upper age limit for ‘‘children’’ can vary from 15 years sometimes silent on what they did with the resultant assessments to 21 years, ‘‘intense’’ physiotherapy might mean very different [89]. If authors exclude studies from the review or any things to different researchers at different times and for different subsequent analyses on the basis of the risk of bias, they should patients, and the volume of blood associated with ‘‘heavy’’ blood tell readers which studies they excluded and explain the reasons loss might vary widely depending on the setting. for those exclusions (see Item 6). Authors should also describe any planned sensitivity or subgroup analyses related to bias Item 12: RISK OF BIAS IN INDIVIDUAL STUDIES. assessments (see Item 16). Describe methods used for assessing risk of bias in individual studies (including specification of whether this was done at the Item 13: SUMMARY MEASURES. State the principal study or outcome level, or both), and how this information is to be summary measures (e.g., risk ratio, difference in means). used in any data synthesis. Examples. ‘‘Relative risk of mortality reduction was the Example. ‘‘To ascertain the validity of eligible randomized primary measure of treatment effect.’’ [105] trials, pairs of reviewers working independently and with ‘‘The meta-analyses were performed by computing relative adequate reliability determined the adequacy of randomi- risks (RRs) using random-effects model. Quantitative zation and concealment of allocation, blinding of patients, analyses were performed on an intention-to-treat basis and health care providers, data collectors, and outcome were confined to data derived from the period of follow-up. assessors; and extent of loss to follow-up (i.e. proportion of RR and 95% confidence intervals for each side effect (and patients in whom the investigators were not able to ascertain all side effects) were calculated.’’ [106] outcomes).’’ [85] ‘‘The primary outcome measure was the mean difference in ‘‘To explore variability in study results (heterogeneity) we log HIV-1 viral load comparing zinc supplementation to specified the following hypotheses before conducting the placebo…’’ [107] analysis. We hypothesised that effect size may differ according to the methodological quality of the studies.’’ Explanation. When planning a systematic review, it is [86] generally desirable that authors pre-specify the outcomes of PLoS Medicine | www.plosmedicine.org 11 July 2009 | Volume 6 | Issue 7 | e1000100 primary interest (see Item 5) as well as the intended summary effect depression) are used across studies, the sign of some scores may measure for each outcome. The chosen summary effect measure need to be reversed to ensure that all scales are aligned (e.g., so low may differ from that used in some of the included studies. If values represent good health on all scales). Standard deviations possible the choice of effect measures should be explained, though may have to be reconstructed from other statistics such as p-values it is not always easy to judge in advance which measure is the most and t statistics [115,116], or occasionally they may be imputed appropriate. from the standard deviations observed in other studies [117]. For binary outcomes, the most common summary measures are Time-to-event data also usually need careful conversions to a the risk ratio, odds ratio, and risk difference [108]. Relative effects consistent format [111]. Authors should report details of any such are more consistent across studies than absolute effects [109,110], data processing. although absolute differences are important when interpreting Statistical combination of data from two or more separate findings (see Item 24). studies in a meta-analysis may be neither necessary nor desirable For continuous outcomes, the natural effect measure is the (see Box 5 and Item 21). Regardless of the decision to combine difference in means [108]. Its use is appropriate when outcome individual study results, authors should report how they planned to measurements in all studies are made on the same scale. The evaluate between-study variability (heterogeneity or inconsistency) standardized difference in means is used when the studies do not (Box 6). The consistency of results across trials may influence the yield directly comparable data. Usually this occurs when all studies decision of whether to combine trial results in a meta-analysis. assess the same outcome but measure it in a variety of ways (e.g., When meta-analysis is done, authors should specify the effect different scales to measure depression). measure (e.g., relative risk or mean difference) (see Item 13), the For time-to-event outcomes, the hazard ratio is the most statistical method (e.g., inverse variance), and whether a fixed- or common summary measure. Reviewers need the log hazard ratio random-effects approach, or some other method (e.g., Bayesian) and its standard error for a study to be included in a meta-analysis was used (see Box 6). If possible, authors should explain the [111]. This information may not be given for all studies, but reasons for those choices. methods are available for estimating the desired quantities from Item 15: RISK OF BIAS ACROSS STUDIES. Specify any other reported information [111]. Risk ratio and odds ratio (in assessment of risk of bias that may affect the cumulative evidence relation to events occurring by a fixed time) are not equivalent to (e.g., publication bias, selective reporting within studies). the hazard ratio, and median survival times are not a reliable basis for meta-analysis [112]. If authors have used these measures they should describe their methods in the report. Examples. ‘‘For each trial we plotted the effect by the inverse of its standard error. The symmetry of such ‘funnel Item 14: PLANNED METHODS OF ANALYSIS. Describe plots’ was assessed both visually, and formally with Egger’s the methods of handling data and combining results of studies, if test, to see if the effect decreased with increasing sample done, including measures of consistency (e.g., I ) for each meta- size.’’ [118] analysis. ‘‘We assessed the possibility of publication bias by evaluating a funnel plot of the trial mean differences for asymmetry, Examples. ‘‘We tested for heterogeneity with the Breslow- which can result from the non publication of small trials with Day test, and used the method proposed by Higgins et al. to negative results…Because graphical evaluation can be measure inconsistency (the percentage of total variation subjective, we also conducted an adjusted rank correlation across studies due to heterogeneity) of effects across lipid- test and a regression asymmetry test as formal statistical tests lowering interventions. The advantages of this measure of for publication bias…We acknowledge that other factors, inconsistency (termed I ) are that it does not inherently such as differences in trial quality or true study heteroge- depend on the number of studies and is accompanied by an neity, could produce asymmetry in funnel plots.’’ [119] uncertainty interval.’’ [113] ‘‘In very few instances, estimates of baseline mean or mean Explanation. Reviewers should explore the possibility that QOL [Quality of life] responses were obtained without the available data are biased. They may examine results from the corresponding estimates of variance (standard deviation available studies for clues that suggest there may be missing studies [SD] or standard error). In these instances, an SD was (publication bias) or missing data from the included studies imputed from the mean of the known SDs. In a number of (selective reporting bias) (see Box 7). Authors should report in cases, the response data available were the mean and detail any methods used to investigate possible bias across studies. variance in a pre study condition and after therapy. The It is difficult to assess whether within-study selective reporting is within-patient variance in these cases could not be present in a systematic review. If a protocol of an individual study is calculated directly and was approximated by assuming available, the outcomes in the protocol and the published report can independence.’’ [114] be compared. Even in the absence of a protocol, outcomes listed in the methods section of the published report can be compared with Explanation. The data extracted from the studies in the those for which results are presented [120]. In only half of 196 trial reports describing comparisons of two drugs in arthritis were all the review may need some transformation (processing) before they are suitable for analysis or for presentation in an evidence table. effect variables in the methods and results sections the same [82]. In other cases, knowledge of the clinical area may suggest that it is Although such data handling may facilitate meta-analyses, it is sometimes needed even when meta-analyses are not done. For likely that the outcome was measured even if it was not reported. For example, in a particular disease, if one of two linked outcomes is example, in trials with more than two intervention groups it may be necessary to combine results for two or more groups (e.g., reported but the other is not, then one should question whether the latter has been selectively omitted [121,122]. receiving similar but non-identical interventions), or it may be desirable to include only a subset of the data to match the review’s Only 36% (76 of 212) of therapeutic systematic reviews inclusion criteria. When several different scales (e.g., for published in November 2004 reported that study publication PLoS Medicine | www.plosmedicine.org 12 July 2009 | Volume 6 | Issue 7 | e1000100 Sensitivity analyses are used to explore the degree to which the Box 5. Whether or Not To Combine Data main findings of a systematic review are affected by changes in Deciding whether or not to combine data involves its methods or in the data used from individual studies (e.g., statistical, clinical, and methodological considerations. study inclusion criteria, results of risk of bias assessment). The statistical decisions are perhaps the most technical Subgroup analyses address whether the summary effects vary and evidence-based. These are more thoroughly discussed in relation to specific (usually clinical) characteristics of the in Box 6. The clinical and methodological decisions are included studies or their participants. Meta-regression extends generally based on discussions within the review team and the idea of subgroup analysis to the examination of the may be more subjective. quantitative influence of study characteristics on the effect size Clinical considerations will be influenced by the [126]. Meta-regression also allows authors to examine the question the review is attempting to address. Broad contribution of different variables to the heterogeneity in study questions might provide more ‘‘license’’ to combine more findings. Readers of systematic reviews should be aware that disparate studies, such as whether ‘‘Ritalin is effective in meta-regression has many limitations, including a danger of increasing focused attention in people diagnosed with over-interpretation of findings [127,128]. attention deficit hyperactivity disorder (ADHD).’’ Here Even with limited data, many additional analyses can be authors might elect to combine reports of studies undertaken. The choice of which analysis to undertake will depend involving children and adults. If the clinical question is on the aims of the review. None of these analyses, however, are more focused, such as whether ‘‘Ritalin is effective in exempt from producing potentially misleading results. It is increasing classroom attention in previously undiagnosed important to inform readers whether these analyses were ADHD children who have no comorbid conditions,’’ it is performed, their rationale, and which were pre-specified. likely that different decisions regarding synthesis of studies are taken by authors. In any case authors should describe their clinical decisions in the systematic review report. RESULTS Deciding whether or not to combine data also has a Item 17: STUDY SELECTION. Give numbers of studies methodological component. Reviewers may decide not to screened, assessed for eligibility, and included in the review, with combine studies of low risk of bias with those of high risk reasons for exclusions at each stage, ideally with a flow diagram. of bias (see Items 12 and 19). For example, for subjective outcomes, systematic review authors may not wish to Examples. In text: combine assessments that were completed under blind ‘‘A total of 10 studies involving 13 trials were identified for conditions with those that were not. For any particular question there may not be a ‘‘right’’ inclusion in the review. The search of Medline, PsycInfo and or ‘‘wrong’’ choice concerning synthesis, as such decisions Cinahl databases provided a total of 584 citations. After are likely complex. However, as the choice may be adjusting for duplicates 509 remained. Of these, 479 studies subjective, authors should be transparent as to their key were discarded because after reviewing the abstracts it decisions and describe them for readers. appeared that these papers clearly did not meet the criteria. Three additional studies…were discarded because full text of the study was not available or the paper could not be feasibly translated into English. The full text of the bias was considered, and only a quarter of those intended to remaining 27 citations was examined in more detail. It carry out a formal assessment for that bias [3]. Of 60 meta- appeared that 22 studies did not meet the inclusion criteria analyses in 24 articles published in 2005 in which formal as described. Five studies…met the inclusion criteria and assessments were reported, most were based on fewer than ten were included in the systematic review. An additional five studies; most displayed statistically significant heterogeneity; and studies…that met the criteria for inclusion were identified by many reviewers misinterpreted the results of the tests employed checking the references of located, relevant papers and [123]. A review of trials of antidepressants found that meta- searching for studies that have cited these papers. No analysis of only the published trials gave effect estimates 32% unpublished relevant studies were obtained.’’ [129] larger on average than when all trials sent to the drug agency See flow diagram Figure 2. were analyzed [67]. Item 16: ADDITIONAL ANALYSES. Describe methods of Explanation. Authors should report, ideally with a flow additional analyses (e.g., sensitivity or subgroup analyses, meta- diagram, the total number of records identified from electronic regression), if done, indicating which were pre-specified. bibliographic sources (including specialized database or registry searches), hand searches of various sources, reference lists, citation indices, and experts. It is useful if authors delineate for readers the Example. ‘‘Sensitivity analyses were pre-specified. The number of selected articles that were identified from the different treatment effects were examined according to quality sources so that they can see, for example, whether most articles were components (concealed treatment allocation, blinding of identified through electronic bibliographic sources or from references patients and caregivers, blinded outcome assessment), time or experts. Literature identified primarily from references or experts to initiation of statins, and the type of statin. One post-hoc may be prone to citation or publication bias [131,132]. sensitivity analysis was conducted including unpublished The flow diagram and text should describe clearly the process of data from a trial using cerivastatin.’’ [124] report selection throughout the review. Authors should report: unique records identified in searches; records excluded after Explanation. Authors may perform additional analyses to preliminary screening (e.g., screening of titles and abstracts); help understand whether the results of their review are robust, all reports retrieved for detailed evaluation; potentially eligible reports of which should be reported. Such analyses include sensitivity that were not retrievable; retrieved reports that did not meet analysis, subgroup analysis, and meta-regression [125]. inclusion criteria and the primary reasons for exclusion; and the PLoS Medicine | www.plosmedicine.org 13 July 2009 | Volume 6 | Issue 7 | e1000100 Box 6. Meta-Analysis and Assessment of Consistency (Heterogeneity) geneity (see below), some consider that the use of a fixed- Meta-Analysis: Statistical Combination of the Results effect analysis is counterintuitive because their main of Multiple Studies If it is felt that studies should have assumption is violated. Others argue that it is inappropriate their results combined statistically, other issues must be to conduct any meta-analysis when there is unexplained considered because there are many ways to conduct a meta- variability across trial results. If the reviewers decide not to analysis. Different effect measures can be used for both combine the data quantitatively, a danger is that eventually binary and continuous outcomes (see Item 13). Also, there are two commonly used statistical models for combining they may end up using quasi-quantitative rules of poor data in a meta-analysis [195]. The fixed-effect model assumes validity (e.g., vote counting of how many studies have that there is a common treatment effect for all included nominally significant results) for interpreting the evidence. studies [196]; it is assumed that the observed differences in Statistical methods to combine data exist for almost any results across studies reflect random variation [196]. The complex situation that may arise in a systematic review, but random-effects model assumes that there is no common one has to be aware of their assumptions and limitations to treatment effect for all included studies but rather that the avoid misapplying or misinterpreting these methods. variation of the effects across studies follows a particular Assessment of Consistency (Heterogeneity) We expect distribution [197]. In a random-effects model it is believed some variation (inconsistency) in the results of different that the included studies represent a random sample from a studies due to chance alone. Variability in excess of that due larger population of studies addressing the question of to chance reflects true differences in the results of the trials, interest [198]. and is called ‘‘heterogeneity.’’ The conventional statistical There is no consensus about whether to use fixed- or approach to evaluating heterogeneity is a chi-squared test random-effects models, and both are in wide use. The (Cochran’s Q), but it has low power when there are few following differences have influenced some researchers studies and excessive power when there are many studies regarding their choice between them. The random-effects [202]. By contrast, the I statistic quantifies the amount of model gives more weight to the results of smaller trials than variation in results across studies beyond that expected by does the fixed-effect analysis, which may be undesirable as chance and so is preferable to Q [202,203]. I represents the small trials may be inferior and most prone to publication percentage of the total variation in estimated effects across bias. The fixed-effect model considers only within-study studies that is due to heterogeneity rather than to chance; variability whereas the random-effects model considers both some authors consider an I value less than 25% as low [202]. within- and between-study variability. This is why a fixed- However, I also suffers from large uncertainty in the effect analysis tends to give narrower confidence intervals common situation where only a few studies are available (i.e., provide greater precision) than a random-effects [204], and reporting the uncertainty in I (e.g., as the 95% analysis [110,196,199]. In the absence of any between-study confidence interval) may be helpful [145]. When there are heterogeneity, the fixed- and random-effects estimates will few studies, inferences about heterogeneity should be coincide. cautious. In addition, there are different methods for performing When considerable heterogeneity is observed, it is both types of meta-analysis [200]. Common fixed-effect advisable to consider possible reasons [205]. In particular, approaches are Mantel-Haenszel and inverse variance, the heterogeneity may be due to differences between whereas random-effects analyses usually use the DerSimo- subgroups of studies (see Item 16). Also, data extraction nian and Laird approach, although other methods exist, errors are a common cause of substantial heterogeneity in including Bayesian meta-analysis [201]. results with continuous outcomes [139]. In the presence of demonstrable between-study hetero- studies included in the review. Indeed, the most appropriate layout Examples. In text: may vary for different reviews. ‘‘Characteristics of included studies Authors should also note the presence of duplicate or Methods supplementary reports so that readers understand the number of All four studies finally selected for the review were randomised individual studies compared to the number of reports that were controlled trials published in English. The duration of the included in the review. Authors should be consistent in their use of intervention was 24 months for the RIO-North America and terms, such as whether they are reporting on counts of citations, 12 months for the RIO-Diabetes, RIO-Lipids and RIO- records, publications, or studies. We believe that reporting the Europe study. Although the last two described a period of 24 number of studies is the most important. months during which they were conducted, only the first 12- A flow diagram can be very useful; it should depict all the months results are provided. All trials had a run-in, as a single studies included based upon fulfilling the eligibility criteria, blind period before the randomisation. whether or not data have been combined for statistical analysis. Participants A recent review of 87 systematic reviews found that about half The included studies involved 6625 participants. The main included a QUOROM flow diagram [133]. The authors of this inclusion criteria entailed adults (18 years or older), with a research recommended some important ways that reviewers can body mass index greater than 27 kg/m and less than 5 kg improve the use of a flow diagram when describing the flow of variation in body weight within the three months before information throughout the review process, including a separate study entry. flow diagram for each important outcome reported [133]. Intervention All trials were multicentric. The RIO-North America was Item 18: STUDY CHARACTERISTICS. For each study, present characteristics for which data were extracted (e.g., study conducted in the USA and Canada, RIO-Europe in Europe size, PICOS, follow-up period) and provide the citation. and the USA, RIO-Diabetes in the USA and 10 other PLoS Medicine | www.plosmedicine.org 14 July 2009 | Volume 6 | Issue 7 | e1000100 Box 7. Bias Caused by Selective Publication of Studies or Results within Studies Systematic reviews aim to incorporate information from all multiple ways and the choice of presentation influenced by relevant studies. The absence of information from some the results obtained. In a study of 102 randomized trials, studies may pose a serious threat to the validity of a review. comparison of published reports with trial protocols showed Data may be incomplete because some studies were not that a median of 38% efficacy and 50% safety outcomes per published, or because of incomplete or inadequate reporting trial, respectively, were not available for meta-analysis. within a published article. These problems are often summa- Statistically significant outcomes had a higher odds of being rized as ‘‘publication bias’’ although in fact the bias arises from fully reported in publications when compared with non- non-publication of full studies and selective publication of significant outcomes for both efficacy (pooled odds ratio 2.4; results in relation to their findings. Non-publication of research 95% confidence interval 1.4 to 4.0) and safety (4.7, 1.8 to 12) findings dependent on the actual results is an important risk of data. Several other studies have had similar findings [210,211]. bias to a systematic review and meta-analysis. Detection of Missing Information Missing studies may Missing Studies Several empirical investigations have increasingly be identified from trials registries. Evidence of shown that the findings from clinical trials are more likely missing outcomes may come from comparison with the to be published if the results are statistically significant study protocol, if available, or by careful examination of (p,0.05) than if they are not [125,206,207]. For example, of published articles [11]. Study publication bias and selective 500 oncology trials with more than 200 participants for outcome reporting are difficult to exclude or verify from the which preliminary results were presented at a conference of available results, especially when few studies are available. the American Society of Clinical Oncology, 81% with p,0.05 If the available data are affected by either (or both) of the were published in full within five years compared to only above biases, smaller studies would tend to show larger 68% of those with p.0.05 [208]. estimates of the effects of the intervention. Thus one Also, among published studies, those with statistically possibility is to investigate the relation between effect size significant results are published sooner than those with non- and sample size (or more specifically, precision of the effect significant findings [209]. When some studies are missing for estimate). Graphical methods, especially the funnel plot these reasons, the available results will be biased towards [212], and analytic methods (e.g., Egger’s test) are often used exaggerating the effect of an intervention. [213,214,215], although their interpretation can be problem- atic [216,217]. Strictly speaking, such analyses investigate Missing Outcomes In many systematic reviews only some ‘‘small study bias’’; there may be many reasons why smaller of the eligible studies (often a minority) can be included in a studies have systematically different effect sizes than larger meta-analysis for a specific outcome. For some studies, the studies, of which reporting bias is just one [218]. Several outcome may not be measured or may be measured but not alternative tests for bias have also been proposed, beyond reported. The former will not lead to bias, but the latter the ones testing small study bias [215,219,220], but none can could. be considered a gold standard. Although evidence that Evidence is accumulating that selective reporting bias is smaller studies had larger estimated effects than large ones widespread and of considerable importance [42,43]. In may suggest the possibility that the available evidence is addition, data for a given outcome may be analyzed in biased, misinterpretation of such data is common [123]. Explanation. For readers to gauge the validity and different countries not specified, and RIO-Lipids in eight applicability of a systematic review’s results, they need to know unspecified different countries. something about the included studies. Such information includes The intervention received was placebo, 5 mg of rimonabant PICOS (Box 2) and specific information relevant to the review or 20 mg of rimonabant once daily in addition to a mild question. For example, if the review is examining the long-term hypocaloric diet (600 kcal/day deficit). effects of antidepressants for moderate depressive disorder, authors Outcomes should report the follow-up periods of the included studies. For Primary each included study, authors should provide a citation for the In all studies the primary outcome assessed was weight source of their information regardless of whether or not the study change from baseline after one year of treatment and the is published. This information makes it easier for interested RIO-North America study also evaluated the prevention of readers to retrieve the relevant publications or documents. weight regain between the first and second year. All studies Reporting study-level data also allows the comparison of the evaluated adverse effects, including those of any kind and main characteristics of the studies included in the review. Authors serious events. Quality of life was measured in only one should present enough detail to allow readers to make their own study, but the results were not described (RIO-Europe). judgments about the relevance of included studies. Such Secondary and additional outcomes information also makes it possible for readers to conduct their These included prevalence of metabolic syndrome after one own subgroup analyses and interpret subgroups, based on study year and change in cardiometabolic risk factors such as characteristics. blood pressure, lipid profile, etc. Authors should avoid, whenever possible, assuming information No study included mortality and costs as outcome. when it is missing from a study report (e.g., sample size, method of The timing of outcome measures was variable and could randomization). Reviewers may contact the original investigators include monthly investigations, evaluations every three to try to obtain missing information or confirm the data extracted months or a single final evaluation after one year.’’ [134] for the systematic review. If this information is not obtained, this In table: See Table 2. should be noted in the report. If information is imputed, the reader PLoS Medicine | www.plosmedicine.org 15 July 2009 | Volume 6 | Issue 7 | e1000100 Figure 2. Example Figure: Example flow diagram of study selection. DDW, Digestive Disease Week; UEGW, United European Gastroenterology Week. Reproduced with permission from [130]. doi:10.1371/journal.pmed.1000100.g002 should be told how this was done and for which items. Presenting do not generally allow for the quantity of information available in study-level data makes it possible to clearly identify unpublished electronic journals or Cochrane reviews, this should not be information obtained from the original researchers and make it accepted as an excuse for omission of important aspects of the available for the public record. methods or results of included studies, since these can, if necessary, Typically, study-level characteristics are presented as a table as be shown on a Web site. in the example in Table 2. Such presentation ensures that all Following the presentation and description of each included pertinent items are addressed and that missing or unclear study, as discussed above, reviewers usually provide a narrative information is clearly indicated. Although paper-based journals summary of the studies. Such a summary provides readers with an PLoS Medicine | www.plosmedicine.org 16 July 2009 | Volume 6 | Issue 7 | e1000100 Table 2. Example Table: Summary of included studies evaluating the efficacy of antiemetic agents in acute gastroenteritis. No. of Source Setting Patients Age Range Inclusion Criteria Antiemetic Agent Route Follow-Up Freedman et al., 2006 ED 214 6 months–10 years GE with mild to moderate Ondansetron PO 1–2 weeks dehydration and vomiting in the preceding 4 hours Reeves et al., 2002 ED 107 1 month–22 years GE and vomiting requiring IV Ondansetron IV 5–7 days rehydration Roslund et al., 2007 ED 106 1–10 years GE with failed oral rehydration Ondansetron PO 1 week attempt in ED Stork et al., 2006 ED 137 6 months–12 years GE, recurrent emesis, mild Ondansetron and IV 1 and 2 days to moderate dehydration, dexamethasone and failed oral hydration ED, emergency department; GE, gastroenteritis; IV, intravenous; PO, by mouth. Adapted from [135]. doi:10.1371/journal.pmed.1000100.t002 overview of the included studies. It may for example address the Cochrane Collaboration’s new tool for assessing the risk of bias languages of the published papers, years of publication, and also requests that authors substantiate these assessments with any geographic origins of the included studies. relevant text from the original studies [11]. It is often easiest to The PICOS framework is often helpful in reporting the narrative provide these data in a tabular format, as in the example. summary indicating, for example, the clinical characteristics and However, a narrative summary describing the tabular data can disease severity of the participants and the main features of the also be helpful for readers. intervention and of the comparison group. For non-pharmacolog- Item 20: RESULTS OF INDIVIDUAL STUDIES. For all ical interventions, it may be helpful to specify for each study the key outcomes considered (benefits and harms), present, for each study: elements of the intervention received by each group. Full details of (a) simple summary data for each intervention group and (b) effect the interventions in included studies were reported in only three of estimates and confidence intervals, ideally with a forest plot. 25 systematic reviews relevant to general practice [84]. Item 19: RISK OF BIAS WITHIN STUDIES. Present data Examples. See Table 4 and Figure 3. on risk of bias of each study and, if available, any outcome-level assessment (see Item 12). Explanation. Publication of summary data from individual studies allows the analyses to be reproduced and other analyses and graphical displays to be investigated. Others may wish to Example. See Table 3. assess the impact of excluding particular studies or consider Explanation. We recommend that reviewers assess the risk of subgroup analyses not reported by the review authors. Displaying bias in the included studies using a standard approach with the results of each treatment group in included studies also enables defined criteria (see Item 12). They should report the results of any inspection of individual study features. For example, if only odds such assessments [89]. ratios are provided, readers cannot assess the variation in event rates across the studies, making the odds ratio impossible to Reporting only summary data (e.g., ‘‘two of eight trials adequately concealed allocation’’) is inadequate because it fails interpret [138]. Additionally, because data extraction errors in to inform readers which studies had the particular methodological meta-analyses are common and can be large [139], the presentation of the results from individual studies makes it easier shortcoming. A more informative approach is to explicitly report the methodological features evaluated for each study. The to identify errors. For continuous outcomes, readers may wish to Table 3. Example Table: Quality measures of the randomized controlled trials that failed to fulfill any one of six markers of validity. Concealment of RCT Stopped Patients Health Care Data Collectors Outcome Trials Randomisation Early Blinded Providers Blinded Blinded Assessors Blinded Liu No No Yes Yes Yes Yes Stone Yes No No Yes Yes Yes Polderman Yes Yes No No No Yes Zaugg Yes No No No Yes Yes Urban Yes Yes No No, except Yes Yes anesthesiologists RCT, randomized controlled trial. Adapted from [96]. doi:10.1371/journal.pmed.1000100.t003 PLoS Medicine | www.plosmedicine.org 17 July 2009 | Volume 6 | Issue 7 | e1000100 reviewers. For example, the standard deviation may be imputed Table 4. Example Table: Heterotopic ossification in trials using the typical standard deviations in the other trials [116,117] comparing radiotherapy to non-steroidal anti-inflammatory (see Item 14). Whenever relevant, authors should indicate which drugs after major hip procedures and fractures. results were not reported directly and had to be estimated from other information (see Item 13). In addition, the inclusion of unpublished data should be noted. Author (Year) Radiotherapy NSAID For all included studies it is important to present the estimated Kienapfel (1999) 12/49 24.5% 20/55 36.4% effect with a confidence interval. This information may be Sell (1998) 2/77 2.6% 18/77 23.4% incorporated in a table showing study characteristics or may be Kolbl (1997) 39/188 20.7% 18/113 15.9% shown in a forest plot [140]. The key elements of the forest plot are Kolbl (1998) 22/46 47.8% 6/54 11.1% the effect estimates and confidence intervals for each study shown graphically, but it is preferable also to include, for each study, the Moore (1998) 9/33 27.3% 18/39 46.2% numerical group-specific summary data, the effect size and Bremen-Kuhne (1997) 9/19 47.4% 11/31 35.5% confidence interval, and the percentage weight (see second example Knelles (1997) 5/101 5.0% 46/183 25.4% [Figure 3]). For discussion of the results of meta-analysis, see Item 21. In principle, all the above information should be provided for NSAID, non-steroidal anti-inflammatory drug. Adapted from [136]. every outcome considered in the review, including both benefits doi:10.1371/journal.pmed.1000100.t004 and harms. When there are too many outcomes for full information to be included, results for the most important examine the consistency of standard deviations across studies, for outcomes should be included in the main report with other example, to be reassured that standard deviation and standard information provided as a Web appendix. The choice of the error have not been confused [138]. information to present should be justified in light of what was For each study, the summary data for each intervention group originally stated in the protocol. Authors should explicitly mention are generally given for binary outcomes as frequencies with and if the planned main outcomes cannot be presented due to lack of without the event (or as proportions such as 12/45). It is not information. There is some evidence that information on harms is sufficient to report event rates per intervention group as only rarely reported in systematic reviews, even when it is available percentages. The required summary data for continuous outcomes in the original studies [141]. Selective omission of harms results are the mean, standard deviation, and sample size for each group. biases a systematic review and decreases its ability to contribute to In reviews that examine time-to-event data, the authors should informed decision making. report the log hazard ratio and its standard error (or confidence interval) for each included study. Sometimes, essential data are Item 21: SYNTHESES OF RESULTS. Present the main missing from the reports of the included studies and cannot be results of the review. If meta-analyses are done, include for each, calculated from other data but may need to be imputed by the confidence intervals and measures of consistency. Figure 3. Example Figure: Overall failure (defined as failure of assigned regimen or relapse) with tetracycline-rifampicin versus tetracycline-streptomycin. CI, confidence interval. Reproduced with permission from [137]. doi:10.1371/journal.pmed.1000100.g003 PLoS Medicine | www.plosmedicine.org 18 July 2009 | Volume 6 | Issue 7 | e1000100 studies, in which case forest plots are of little value and may be Examples. ‘‘Mortality data were available for all six trials, seriously biased. randomizing 311 patients and reporting data for 305 Of 300 systematic reviews indexed in MEDLINE in 2004, a patients. There were no deaths reported in the three little more than half (54%) included meta-analyses, of which the respiratory syncytial virus/severe bronchiolitis trials; thus majority (91%) reported assessing for inconsistency in results. our estimate is based on three trials randomizing 232 patients, 64 of whom died. In the pooled analysis, surfactant Item 22: RISK OF BIAS ACROSS STUDIES. Present was associated with significantly lower mortality (relative results of any assessment of risk of bias across studies (see Item 15). risk = 0.7, 95% confidence interval = 0.4–0.97, P = 0.04). There was no evidence of heterogeneity (I = 0%)’’. [142] Examples. ‘‘Strong evidence of heterogeneity (I = 79%, ‘‘Because the study designs, participants, interventions, and P,0.001) was observed. To explore this heterogeneity, a reported outcome measures varied markedly, we focused on funnel plot was drawn. The funnel plot in Figure 4 shows describing the studies, their results, their applicability, and evidence of considerable asymmetry.’’ [146] their limitations and on qualitative synthesis rather than ‘‘Specifically, four sertraline trials involving 486 participants meta-analysis.’’ [143] and one citalopram trial involving 274 participants were ‘‘We detected significant heterogeneity within this compar- 2 2 reported as having failed to achieve a statistically significant ison (I = 46.6%; x = 13.11, df = 7; P = 0.07). Retrospective drug effect, without reporting mean HRSD [Hamilton Rating exploration of the heterogeneity identified one trial that Scale for Depression] scores. We were unable to find data from seemed to differ from the others. It included only small these trials on pharmaceutical company Web sites or through ulcers (wound area less than 5 cm ). Exclusion of this trial our search of the published literature. These omissions removed the statistical heterogeneity and did not affect the represent 38% of patients in sertraline trials and 23% of finding of no evidence of a difference in healing rate patients in citalopram trials. Analyses with and without between hydrocolloids and simple low adherent dressings inclusion of these trials found no differences in the patterns (relative risk = 0.98, [95% confidence interval] 0.85 to 1.12; of results; similarly, the revealed patterns do not interact with I = 0%).’’ [144] drug type. The purpose of using the data obtained from the FDA was to avoid publication bias, by including unpublished Explanation. Results of systematic reviews should be as well as published trials. Inclusion of only those sertraline and presented in an orderly manner. Initial narrative descriptions of citalopram trials for which means were reported to the FDA the evidence covered in the review (see Item 18) may tell readers would constitute a form of reporting bias similar to publication important things about the study populations and the design and bias and would lead to overestimation of drug–placebo conduct of studies. These descriptions can facilitate the differences for these drug types. Therefore, we present analyses examination of patterns across studies. They may also provide only on data for medications for which complete clinical trials’ important information about applicability of evidence, suggest the change was reported.’’ [147] likely effects of any major biases, and allow consideration, in a systematic manner, of multiple explanations for possible Explanation. Authors should present the results of any differences of findings across studies. assessments of risk of bias across studies. If a funnel plot is If authors have conducted one or more meta-analyses, they reported, authors should specify the effect estimate and measure of should present the results as an estimated effect across studies precision used, presented typically on the x-axis and y-axis, with a confidence interval. It is often simplest to show each respectively. Authors should describe if and how they have tested meta-analysis summary with the actual results of included studies the statistical significance of any possible asymmetry (see Item 15). in a forest plot (see Item 20) [140]. It should always be clear Results of any investigations of selective reporting of outcomes which of the included studies contributed to each meta-analysis. within studies (as discussed in Item 15) should also be reported. Authors should also provide, for each meta-analysis, a measure Also, we advise authors to tell readers if any pre-specified analyses of the consistency of the results from the included studies such for assessing risk of bias across studies were not completed and the as I (heterogeneity; see Box 6); a confidence interval may also reasons (e.g., too few included studies). be given for this measure [145]. If no meta-analysis was performed, the qualitative inferences should be presented as Item 23: ADDITIONAL ANALYSES. Give results of systematically as possible with an explanation of why meta- additional analyses, if done (e.g., sensitivity or subgroup analyses, analysis was not done, as in the second example above [143]. meta-regression [see Item 16]). Readers may find a forest plot, without a summary estimate, helpful in such cases. Examples. ‘‘…benefits of chondroitin were smaller in trials Authors should in general report syntheses for all the outcome with adequate concealment of allocation compared with measures they set out to investigate (i.e., those described in the trials with unclear concealment (P for interaction = 0.050), in protocol; see Item 4) to allow readers to draw their own trials with an intention-to-treat analysis compared with those conclusions about the implications of the results. Readers should that had excluded patients from the analysis (P for be made aware of any deviations from the planned analysis. interaction = 0.017), and in large compared with small trials Authors should tell readers if the planned meta-analysis was not (P for interaction = 0.022).’’ [148] thought appropriate or possible for some of the outcomes and the ‘‘Subgroup analyses according to antibody status, antiviral reasons for that decision. It may not always be sensible to give meta-analysis results and medications, organ transplanted, treatment duration, use of forest plots for each outcome. If the review addresses a broad antilymphocyte therapy, time to outcome assessment, study question, there may be a very large number of outcomes. Also, quality and other aspects of study design did not some outcomes may have been reported in only one or two demonstrate any differences in treatment effects. Multivar- PLoS Medicine | www.plosmedicine.org 19 July 2009 | Volume 6 | Issue 7 | e1000100 Figure 4. Example Figure: Example of a funnel plot showing evidence of considerable asymmetry. SE, standard error. Adapted from [146], with permission. doi:10.1371/journal.pmed.1000100.g004 iate meta-regression showed no significant difference in Example. ‘‘Overall, the evidence is not sufficiently robust CMV [cytomegalovirus] disease after allowing for potential to determine the comparative effectiveness of angioplasty confounding or effect-modification by prophylactic drug (with or without stenting) and medical treatment alone. Only used, organ transplanted or recipient serostatus in CMV 2 randomized trials with long-term outcomes and a third positive recipients and CMV negative recipients of CMV randomized trial that allowed substantial crossover of positive donors.’’ [149] treatment after 3 months directly compared angioplasty and medical treatment…the randomized trials did not Explanation. Authors should report any subgroup or evaluate enough patients or did not follow patients for a sensitivity analyses and whether or not they were pre-specified sufficient duration to allow definitive conclusions to be made (see Items 5 and 16). For analyses comparing subgroups of about clinical outcomes, such as mortality and cardiovascu- studies (e.g., separating studies of low- and high-dose aspirin), the lar or kidney failure events. authors should report any tests for interactions, as well as Some acceptable evidence from comparison of medical estimates and confidence intervals from meta-analyses within treatment and angioplasty suggested no difference in long- each subgroup. Similarly, meta-regression results (see Item 16) term kidney function but possibly better blood pressure should not be limited to p-values, but should include effect sizes control after angioplasty, an effect that may be limited to and confidence intervals [150], as the first example reported patients with bilateral atherosclerotic renal artery stenosis. above does in a table. The amount of data included in each The evidence regarding other outcomes is weak. Because the additional analysis should be specified if different from that reviewed studies did not explicitly address patients with considered in the main analyses. This information is especially rapid clinical deterioration who may need acute interven- relevant for sensitivity analyses that exclude some studies; for tion, our conclusions do not apply to this important subset of example, those with high risk of bias. patients.’’ [143] Importantly, all additional analyses conducted should be reported, not just those that were statistically significant. This information will help avoid selective outcome reporting bias Explanation. Authors should give a brief and balanced within the review as has been demonstrated in reports of summary of the nature and findings of the review. Sometimes, randomized controlled trials [42,44,121,151,152]. Results from outcomes for which little or no data were found should be noted exploratory subgroup or sensitivity analyses should be interpret- due to potential relevance for policy decisions and future research. ed cautiously, bearing in mind the potential for multiple analyses Applicability of the review’s findings, to different patients, settings, to mislead. or target audiences, for example, should be mentioned. Although there is no standard way to assess applicability simultaneously to different audiences, some systems do exist [153]. Sometimes, DISCUSSION Item 24: SUMMARY OF EVIDENCE. Summarize the main authors formally rate or assess the overall body of evidence findings, including the strength of evidence for each main addressed in the review and can present the strength of their outcome; consider their relevance to key groups (e.g., health summary recommendations tied to their assessments of the quality care providers, users, and policy makers). of evidence (e.g., the GRADE system) [10]. PLoS Medicine | www.plosmedicine.org 20 July 2009 | Volume 6 | Issue 7 | e1000100 Authors need to keep in mind that statistical significance of the Item 26: CONCLUSIONS. Provide a general interpretation effects does not always suggest clinical or policy relevance. of the results in the context of other evidence, and implications for Likewise, a non-significant result does not demonstrate that a future research. treatment is ineffective. Authors should ideally clarify trade-offs and how the values attached to the main outcomes would lead different people to make different decisions. In addition, adroit Example. Implications for practice: authors consider factors that are important in translating the ‘‘Between 1995 and 1997 five different meta-analyses of the evidence to different settings and that may modify the estimates of effect of antibiotic prophylaxis on infection and mortality effects reported in the review [153]. Patients and health care were published. All confirmed a significant reduction in providers may be primarily interested in which intervention is infections, though the magnitude of the effect varied from most likely to provide a benefit with acceptable harms, while policy one review to another. The estimated impact on overall makers and administrators may value data on organizational mortality was less evident and has generated considerable impact and resource utilization. controversy on the cost effectiveness of the treatment. Only one among the five available reviews, however, suggested Item 25: LIMITATIONS. Discuss limitations at study and that a weak association between respiratory tract infections outcome level (e.g., risk of bias), and at review level (e.g., and mortality exists and lack of sufficient statistical power incomplete retrieval of identified research, reporting bias). may have accounted for the limited effect on mortality.’’ Implications for research: Examples. Outcome level: ‘‘A logical next step for future trials would thus be the ‘‘The meta-analysis reported here combines data across comparison of this protocol against a regimen of a systemic studies in order to estimate treatment effects with more antibiotic agent only to see whether the topical component precision than is possible in a single study. The main can be dropped. We have already identified six such trials limitation of this meta-analysis, as with any overview, is that but the total number of patients so far enrolled (n = 1056) is the patient population, the antibiotic regimen and the too small for us to be confident that the two treatments are outcome definitions are not the same across studies.’’ [154] really equally effective. If the hypothesis is therefore Study and review level: considered worth testing more and larger randomised ‘‘Our study has several limitations. The quality of the studies controlled trials are warranted. Trials of this kind, however, varied. Randomization was adequate in all trials; however, 7 would not resolve the relevant issue of treatment induced of the articles did not explicitly state that analysis of data resistance. To produce a satisfactory answer to this, studies adhered to the intention-to-treat principle, which could lead with a different design would be necessary. Though a to overestimation of treatment effect in these trials, and we detailed discussion goes beyond the scope of this paper, could not assess the quality of 4 of the 5 trials reported as studies in which the intensive care unit rather than the abstracts. Analyses did not identify an association between individual patient is the unit of randomisation and in which components of quality and re-bleeding risk, and the effect the occurrence of antibiotic resistance is monitored over a size in favour of combination therapy remained statistically long period of time should be undertaken.’’ [156] significant when we excluded trials that were reported as abstracts. Explanation. Systematic reviewers sometimes draw Publication bias might account for some of the effect we conclusions that are too optimistic [157] or do not consider observed. Smaller trials are, in general, analyzed with less the harms equally as carefully as the benefits, although some methodological rigor than larger studies, and an asymmet- evidence suggests these problems are decreasing [158]. If rical funnel plot suggests that selective reporting may have conclusions cannot be drawn because there are too few reliable led to an overestimation of effect sizes in small trials.’’ [155] studies, or too much uncertainty, this should be stated. Such a finding can be as important as finding consistent effects from Explanation. A discussion of limitations should address the several large studies. validity (i.e., risk of bias) and reporting (informativeness) of the Authors should try to relate the results of the review to other included studies, limitations of the review process, and evidence, as this helps readers to better interpret the results. For generalizability (applicability) of the review. Readers may find it example, there may be other systematic reviews about the same helpful if authors discuss whether studies were threatened by general topic that have used different methods or have addressed serious risks of bias, whether the estimates of the effect of the related but slightly different questions [159,160]. Similarly, there intervention are too imprecise, or if there were missing data for may be additional information relevant to decision makers, such as many participants or important outcomes. the cost-effectiveness of the intervention (e.g., health technology Limitations of the review process might include limitations of the assessment). Authors may discuss the results of their review in the search (e.g., restricting to English-language publications), and any context of existing evidence regarding other interventions. difficulties in the study selection, appraisal, and meta-analysis We advise authors also to make explicit recommendations for processes. For example, poor or incomplete reporting of study future research. In a sample of 2,535 Cochrane reviews, 82% designs, patient populations, and interventions may hamper included recommendations for research with specific interventions, interpretation and synthesis of the included studies [84]. Applica- 30% suggested the appropriate type of participants, and 52% bility of the review may be affected if there are limited data for suggested outcome measures for future research [161]. There is no certain populations or subgroups where the intervention might corresponding assessment about systematic reviews published in medical journals, but we believe that such recommendations are perform differently or few studies assessing the most important outcomes of interest; or if there is a substantial amount of data much less common in those reviews. relating to an outdated intervention or comparator or heavy reliance Clinical research should not be planned without a thorough on imputation of missing values for summary estimates (Item 14). knowledge of similar, existing research [162]. There is evidence PLoS Medicine | www.plosmedicine.org 21 July 2009 | Volume 6 | Issue 7 | e1000100 that this still does not occur as it should and that authors of here. A useful principle is for systematic review authors to ensure primary studies do not consider a systematic review when they that their methods are reported with adequate clarity and design their studies [163]. We believe systematic reviews have transparency to enable readers to critically judge the available great potential for guiding future clinical research. evidence and replicate or update the research. In some systematic reviews, the authors will seek the raw data from the original researchers to calculate the summary statistics. FUNDING These systematic reviews are called individual patient (or Item 27: FUNDING. Describe sources of funding or other participant) data reviews [40,41]. Individual patient data meta- support (e.g., supply of data) for the systematic review; role of analyses may also be conducted with prospective accumulation of funders for the systematic review. data rather than retrospective accumulation of existing data. Here too, extra information about the methods will need to be reported. Examples: ‘‘The evidence synthesis upon which this article Other types of systematic reviews exist. Realist reviews aim to was based was funded by the Centers for Disease Control determine how complex programs work in specific contexts and and Prevention for the Agency for Healthcare Research and settings [174]. Meta-narrative reviews aim to explain complex Quality and the U.S. Prevention Services Task Force.’’ bodies of evidence through mapping and comparing different [164] over-arching storylines [175]. Network meta-analyses, also known ‘‘Role of funding source: the funders played no role in study as multiple treatments meta-analyses, can be used to analyze data design, collection, analysis, interpretation of data, writing of from comparisons of many different treatments [176,177]. They the report, or in the decision to submit the paper for use both direct and indirect comparisons, and can be used to publication. They accept no responsibility for the contents.’’ compare interventions that have not been directly compared. [165] We believe that the issues we have highlighted in this paper are relevant to ensure transparency and understanding of the Explanation. Authors of systematic reviews, like those of any processes adopted and the limitations of the information presented other research study, should disclose any funding they received to in systematic reviews of different types. We hope that PRISMA carry out the review, or state if the review was not funded. Lexchin can be the basis for more detailed guidance on systematic reviews and colleagues [166] observed that outcomes of reports of of other types of research, including diagnostic accuracy and randomized trials and meta-analyses of clinical trials funded by epidemiological studies. the pharmaceutical industry are more likely to favor the sponsor’s product compared to studies with other sources of funding. Similar Discussion results have been reported elsewhere [167,168]. Analogous data suggest that similar biases may affect the conclusions of systematic We developed the PRISMA Statement using an approach for reviews [169]. developing reporting guidelines that has evolved over several years Given the potential role of systematic reviews in decision [178]. The overall aim of PRISMA is to help ensure the clarity making, we believe authors should be transparent about the and transparency of reporting of systematic reviews, and recent funding and the role of funders, if any. Sometimes the funders will data indicate that this reporting guidance is much needed [3]. provide services, such as those of a librarian to complete the PRISMA is not intended to be a quality assessment tool and it searches for relevant literature or access to commercial databases should not be used as such. not available to the reviewers. Any level of funding or services This PRISMA Explanation and Elaboration document was provided to the systematic review team should be reported. developed to facilitate the understanding, uptake, and dissemina- Authors should also report whether the funder had any role in the tion of the PRISMA Statement and hopefully provide a conduct or report of the review. Beyond funding issues, authors pedagogical framework for those interested in conducting and should report any real or perceived conflicts of interest related to reporting systematic reviews. It follows a format similar to that their role or the role of the funder in the reporting of the used in other explanatory documents [17,18,19]. Following the systematic review [170]. recommendations in the PRISMA checklist may increase the word In a survey of 300 systematic reviews published in November count of a systematic review report. We believe, however, that the 2004, funding sources were not reported in 41% of the benefit of readers being able to critically appraise a clear, reviews [3]. Only a minority of reviews (2%) reported being complete, and transparent systematic review report outweighs funded by for-profit sources, but the true proportion may be the possible slight increase in the length of the report. higher [171]. While the aims of PRISMA are to reduce the risk of flawed reporting of systematic reviews and improve the clarity and Additional Considerations for Systematic Reviews transparency in how reviews are conducted, we have little data to of Non-Randomized Intervention Studies or for state more definitively whether this ‘‘intervention’’ will achieve its Other Types of Systematic Reviews intended goal. A previous effort to evaluate QUOROM was not successfully completed [178]. Publication of the QUOROM The PRISMA Statement and this document have focused on Statement was delayed for two years while a research team systematic reviews of reports of randomized trials. Other study attempted to evaluate its effectiveness by conducting a randomized designs, including non-randomized studies, quasi-experimental controlled trial with the participation of eight major medical studies, and interrupted time series, are included in some journals. Unfortunately that trial was not completed due to accrual systematic reviews that evaluate the effects of health care problems (David Moher, personal communication). Other evalu- interventions [172,173]. The methods of these reviews may differ ation methods might be easier to conduct. At least one survey of to varying degrees from the typical intervention review, for example regarding the literature search, data abstraction, 139 published systematic reviews in the critical care literature assessment of risk of bias, and analysis methods. As such, their [179] suggests that their quality improved after the publication of reporting demands might also differ from what we have described QUOROM. PLoS Medicine | www.plosmedicine.org 22 July 2009 | Volume 6 | Issue 7 | e1000100 If the PRISMA Statement is endorsed by and adhered to in guidelines, such as CONSORT. We also encourage editors of journals, as other reporting guidelines have been health care journals to support PRISMA by updating their [17,18,19,180], there should be evidence of improved reporting ‘‘Instructions to Authors’’ and including the PRISMA Web of systematic reviews. For example, there have been several address, and by raising awareness through specific editorial evaluations of whether the use of CONSORT improves reports actions. of randomized controlled trials. A systematic review of these studies [181] indicates that use of CONSORT is associated Supporting Information with improved reporting of certain items, such as allocation Figure S1 Flow of information through the different phases of a concealment. We aim to evaluate the benefits (i.e., improved systematic review (downloadable template document for research- reporting) and possible adverse effects (e.g., increased word ers to re-use). length) of PRISMA and we encourage others to consider doing Found at: doi:10.1371/journal.pmed.1000100.s001 (0.08 MB likewise. DOC) Even though we did not carry out a systematic literature search to produce our checklist, and this is indeed a limitation Text S1 Checklist of items to include when reporting a of our effort, PRISMA was nevertheless developed using an systematic review or meta-analysis (downloadable template evidence-based approach, whenever possible. Checklist items document for researchers to re-use). were included if there was evidence that not reporting the item Found at: doi:10.1371/journal.pmed.1000100.s002 (0.04 MB was associated with increased risk of bias, or where it was DOC) clear that information was necessary to appraise the reliability of a review. To keep PRISMA up-to-date and as evidence- Acknowledgments based as possible requires regular vigilance of the literature, The following people contributed to this paper: which is growing rapidly. Currently the Cochrane Methodol- Doug Altman, DSc, Centre for Statistics in Medicine (Oxford, UK); ogy Register has more than 11,000 records pertaining to the Gerd Antes, PhD, University Hospital Freiburg (Freiburg, Germany); conduct and reporting of systematic reviews and other David Atkins, MD, MPH, Health Services Research and Development evaluations of health and social care. For some checklist items, Service, Veterans Health Administration (Washington, D. C., US); such as reporting the abstract (Item 2), we have used evidence Virginia Barbour, MRCP, DPhil, PLoS Medicine (Cambridge, UK); Nick from elsewhere in the belief that the issue applies equally well Barrowman, PhD, Children’s Hospital of Eastern Ontario (Ottawa, to reporting of systematic reviews. Yet for other items, Canada); Jesse A. Berlin, ScD, Johnson & Johnson Pharmaceutical Research and Development (Titusville, New Jersey, US); Jocalyn Clark, evidence does not exist; for example, whether a training PhD, PLoS Medicine (at the time of writing, BMJ, London, UK); Mike exercise improves the accuracy and reliability of data Clarke, PhD, UK Cochrane Centre (Oxford, UK) and School of extraction. We hope PRISMA will act as a catalyst to help Nursing and Midwifery, Trinity College (Dublin, Ireland); Deborah generate further evidence that can be considered when further Cook, MD, Departments of Medicine, Clinical Epidemiology and revising the checklist in the future. Biostatistics, McMaster University (Hamilton, Canada); Roberto More than ten years have passed between the development of D’Amico, PhD, Universita` di Modena e Reggio Emilia (Modena, the QUOROM Statement and its update, the PRISMA Italy) and Centro Cochrane Italiano, Istituto Ricerche Farmacologiche Mario Negri (Milan, Italy); Jonathan J. Deeks, PhD, University of Statement. We aim to update PRISMA more frequently. We Birmingham (Birmingham, UK); P. J. Devereaux, MD, PhD, hope that the implementation of PRISMA will be better than it Departments of Medicine, Clinical Epidemiology and Biostatistics, has been for QUOROM. There are at least two reasons to be McMaster University (Hamilton, Canada); Kay Dickersin, PhD, Johns optimistic. First, systematic reviews are increasingly used by health Hopkins Bloomberg School of Public Health (Baltimore, Maryland, care providers to inform ‘‘best practice’’ patient care. Policy US); Matthias Egger, MD, Department of Social and Preventive analysts and managers are using systematic reviews to inform Medicine, University of Bern (Bern, Switzerland); Edzard Ernst, MD, health care decision making, and to better target future research. PhD, FRCP, FRCP(Edin), Peninsula Medical School (Exeter, UK); Second, we anticipate benefits from the development of the Peter C. Gøtzsche, MD, MSc, The Nordic Cochrane Centre (Copenhagen, Denmark); Jeremy Grimshaw, MBChB, PhD, FRCFP, EQUATOR Network, described below. Ottawa Hospital Research Institute (Ottawa, Canada); Gordon Guyatt, Developing any reporting guideline requires considerable effort, MD, Departments of Medicine, Clinical Epidemiology and Biostatistics, experience, and expertise. While reporting guidelines have been McMaster University (Hamilton, Canada); Julian Higgins, PhD, MRC successful for some individual efforts [17,18,19], there are likely Biostatistics Unit (Cambridge, UK); John P. A. Ioannidis, MD, others who want to develop reporting guidelines who possess little University of Ioannina Campus (Ioannina, Greece); Jos Kleijnen, time, experience, or knowledge as to how to do so appropriately. MD, PhD, Kleijnen Systematic Reviews Ltd (York, UK) and School The EQUATOR Network (Enhancing the QUAlity and Trans- for Public Health and Primary Care (CAPHRI), University of Maastricht (Maastricht, Netherlands); Tom Lang, MA, Tom Lang parency Of health Research) aims to help such individuals and Communications and Training (Davis, California, US); Alessandro groups by serving as a global resource for anybody interested in Liberati, MD, Universita` di Modena e Reggio Emilia (Modena, Italy) developing reporting guidelines, regardless of the focus and Centro Cochrane Italiano, Istituto Ricerche Farmacologiche Mario [7,180,182]. The overall goal of EQUATOR is to improve the Negri (Milan, Italy); Nicola Magrini, MD, NHS Centre for the quality of reporting of all health science research through the Evaluation of the Effectiveness of Health Care – CeVEAS (Modena, development and translation of reporting guidelines. Beyond this Italy); David McNamee, PhD, The Lancet (London, UK); Lorenzo aim, the network plans to develop a large Web presence by Moja, MD, MSc, Centro Cochrane Italiano, Istituto Ricerche Farm- acologiche Mario Negri (Milan, Italy); David Moher, PhD, Ottawa developing and maintaining a resource center of reporting tools, Methods Centre, Ottawa Hospital Research Institute (Ottawa, Canada); and other information for reporting research (http://www. Cynthia Mulrow, MD, MSc, Annals of Internal Medicine (Philadelphia, equator-network.org/). Pennsylvania, US); Maryann Napoli, Center for Medical Consumers We encourage health care journals and editorial groups, such as (New York, New York, US); Andy Oxman, MD, Norwegian Health the World Association of Medical Editors and the International Services Research Centre (Oslo, Norway); Ba’ Pham, MMath, Toronto Committee of Medical Journal Editors, to endorse PRISMA in Health Economics and Technology Assessment Collaborative (Toronto, much the same way as they have endorsed other reporting Canada) (at the time of the first meeting of the group, GlaxoSmithK- PLoS Medicine | www.plosmedicine.org 23 July 2009 | Volume 6 | Issue 7 | e1000100 line Canada, Mississauga, Canada); Drummond Rennie, MD, FRCP, Dr. Lorenzo Moja helped with the preparation and the several updates FACP, University of California San Francisco (San Francisco, of the manuscript and assisted with the preparation of the reference list. California, US); Margaret Sampson, MLIS, Children’s Hospital of Alessandro Liberati is the guarantor of the manuscript. Eastern Ontario (Ottawa, Canada); Kenneth F. Schulz, PhD, MBA, Family Health International (Durham, North Carolina, US); Paul G. Author Contributions Shekelle, MD, PhD, Southern California Evidence Based Practice Center (Santa Monica, California, US); Jennifer Tetzlaff, BSc, Ottawa ICMJE criteria for authorship read and met: AL DGA JT CM PCG JPAI Methods Centre, Ottawa Hospital Research Institute (Ottawa, Canada); MC PJD JK DM. Wrote the first draft of the paper: AL DGA JT JPAI David Tovey, FRCGP, The Cochrane Library, Cochrane Collabora- DM. Contributed to the writing of the paper: AL DGA JT CM PCG JPAI tion (Oxford, UK) (at the time of the first meeting of the group, BMJ, MC PJD JK DM. Concept and design of the Explanation and Elaboration London, UK); Peter Tugwell, MD, MSc, FRCPC, Institute of statement: AL DGA JT DM. Agree with the recommendations: AL DGA Population Health, University of Ottawa (Ottawa, Canada). JT CM PCG JPAI MC PJD JK DM. References 1. Canadian Institutes of Health Research (2006) Randomized controlled trials 21. Bjelakovic G, Nikolova D, Gluud LL, Simonetti RG, Gluud C (2007) registration/application checklist (12/2006). Available: http://www.cihr-irsc. Mortality in randomized trials of antioxidant supplements for primary and gc.ca/e/documents/rct_reg_e.pdf. Accessed 26 May 2009. secondary prevention: Systematic review and meta-analysis. JAMA 297: 2. Young C, Horton R (2005) Putting clinical trials into context. Lancet 366: 842–857. 107–108. 22. Montori VM, Wilczynski NL, Morgan D, Haynes RB (2005) Optimal search 3. Moher D,TetzlaffJ,TriccoAC, SampsonM,AltmanDG(2007) strategies for retrieving systematic reviews from Medline: Analytical survey. Epidemiology and reporting characteristics of systematic reviews. PLoS Med BMJ 330: 68. 4: e78. doi:10.1371/journal.pmed.0040078. 23. Bischoff-Ferrari HA, Willett WC, Wong JB, Giovannucci E, Dietrich T, et al. 4. Dixon E, Hameed M, Sutherland F, Cook DJ, Doig C (2005) Evaluating meta- (2005) Fracture prevention with vitamin D supplementation: A meta-analysis of analyses in the general surgical literature: A critical appraisal. Ann Surg 241: randomized controlled trials. JAMA 293: 2257–2264. 450–459. 24. Hopewell S, Clarke M, Moher D, Wager E, Middleton P, et al. (2008) 5. Hemels ME, Vicente C, Sadri H, Masson MJ, Einarson TR (2004) Quality CONSORT for reporting randomised trials in journal and conference assessment of meta-analyses of RCTs of pharmacotherapy in major depressive abstracts. Lancet 371: 281–283. disorder. Curr Med Res Opin 20: 477–484. 25. Hopewell S, Clarke M, Moher D, Wager E, Middleton P, et al. (2008) 6. Jin W, Yu R, Li W, Youping L, Ya L, et al. (2008) The reporting quality of CONSORT for reporting randomized controlled trials in journal and meta-analyses improves: A random sampling study. J Clin Epidemiol 61: conference abstracts: Explanation and elaboration. PLoS Med 5: e20. 770–775. doi:10.1371/journal.pmed.0050020. 7. Moher D, Simera I, Schulz KF, Hoey J, Altman DG (2008) Helping editors, 26. Haynes RB, Mulrow CD, Huth EJ, Altman DG, Gardner MJ (1990) More peer reviewers and authors improve the clarity, completeness and transparency informative abstracts revisited. Ann Intern Med 113: 69–76. of reporting health research. BMC Med 6: 13. 27. Mulrow CD, Thacker SB, Pugh JA (1988) A proposal for more informative 8. Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, et al. (1999) Improving abstracts of review articles. Ann Intern Med 108: 613–615. the quality of reports of meta-analyses of randomised controlled trials: The 28. Froom P, Froom J (1993) Deficiencies in structured medical abstracts. J Clin QUOROM statement. Quality of Reporting of Meta-analyses. Lancet 354: Epidemiol 46: 591–594. 1896–1900. 29. Hartley J (2000) Clarifying the abstracts of systematic literature reviews. Bull 9. Green S, Higgins JPT, Alderson P, Clarke M, Mulrow CD, et al. (2008) Med Libr Assoc 88: 332–337. Chapter 1: What is a systematic review? In: Higgins JPT, Green S, editors. 30. Hartley J, Sydes M, Blurton A (1996) Obtaining information accurately and Cochrane handbook for systematic reviews of interventions version 5.0.0 quickly: Are structured abstract more efficient? J Infor Sci 22: 349–356. [updated February 2008]. The Cochrane Collaboration. Available: http:// 31. Pocock SJ, Hughes MD, Lee RJ (1987) Statistical problems in the reporting of www.cochrane-handbook.org/. Accessed 26 May 2009. clinical trials. A survey of three medical journals. N Engl J Med 317: 426–432. 10. Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, et al. (2008) GRADE: An emerging consensus on rating quality of evidence and strength of 32. Taddio A, Pain T, Fassos FF, Boon H, Ilersich AL, et al. (1994) Quality of recommendations. BMJ 336: 924–926. nonstructured and structured abstracts of original research articles in the British 11. Higgins JPT, Altman DG (2008) Chapter 8: Assessing risk of bias in included Medical Journal, the Canadian Medical Association Journal and the Journal of studies. In: Higgins JPT, Green S, eds. Cochrane handbook for systematic the American Medical Association. CMAJ 150: 1611–1615. reviews of interventions version 5.0.0 [updated February 2008]. The Cochrane 33. Harris KC, Kuramoto LK, Schulzer M, Retallack JE (2009) Effect of school- Collaboration, Available: http://www.cochrane-handbook.org/. Accessed 26 based physical activity interventions on body mass index in children: A meta- May 2009. analysis. CMAJ 180: 719–726. 12. Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group (2008) 34. James MT, Conley J, Tonelli M, Manns BJ, MacRae J, et al. (2008) Meta- Preferred reporting items for systematic reviews and meta-analyses: The analysis: Antibiotics for prophylaxis against hemodialysis catheter-related PRISMA Statement. PLoS Med 6: e1000097. 10.1371/journal.pmed.1000097. infections. Ann Intern Med 148: 596–605. 13. Atkins D, Fink K, Slutsky J (2005) Better information for better health care: 35. Counsell C (1997) Formulating questions and locating primary studies for The Evidence-based Practice Center program and the Agency for Healthcare inclusion in systematic reviews. Ann Intern Med 127: 380–387. Research and Quality. Ann Intern Med 142: 1035–1041. 36. Gotzsche PC (2000) Why we need a broad perspective on meta-analysis. It may 14. Helfand M, Balshem H (2009) Principles for developing guidance: AHRQ and be crucially important for patients. BMJ 321: 585–586. the effective health-care program. J Clin Epidemiol, In press. 37. Grossman P, Niemann L, Schmidt S, Walach H (2004) Mindfulness-based 15. Higgins JPT, Green S (2008) Cochrane handbook for systematic reviews of stress reduction and health benefits. A meta-analysis. J Psychosom Res 57: interventions version 5.0.0 [updated February 2008]. The Cochrane 35–43. Collaboration. Available: http://www.cochrane-handbook.org/. Accessed 26 38. Brunton G, Green S, Higgins JPT, Kjeldstrøm M, Jackson N, et al. (2008) May 2009. Chapter 2: Preparing a Cochrane review. In: Higgins JPT, Green S, eds. 16. Centre for Reviews and Dissemination (2009) Systematic reviews: CRD’s Cochrane handbook for systematic reviews of interventions version 5.0.0 guidance for undertaking reviews in health care. York: University of York, [updated February 2008]. The Cochrane Collaboration, Available: http:// Available: http://www.york.ac.uk/inst/crd/systematic_reviews_book.htm. Ac- www.cochrane-handbook.org/. Accessed 26 May 2009. cessed 26 May 2009. 39. Sutton AJ, Abrams KR, Jones DR, Sheldon TA, Song F (1998) Systematic 17. Altman DG, Schulz KF, Moher D, Egger M, Davidoff F, et al. (2001) The reviews of trials and other studies. Health Technol Assess 2: 1–276. revised CONSORT statement for reporting randomized trials: Explanation 40. Ioannidis JP, Rosenberg PS, Goedert JJ, O’Brien TR (2002) Commentary: and elaboration. Ann Intern Med 134: 663–694. Meta-analysis of individual participants’ data in genetic epidemiology. 18. Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, et al. (2003) Am J Epidemiol 156: 204–210. The STARD statement for reporting studies of diagnostic accuracy: 41. Stewart LA, Clarke MJ (1995) Practical methodology of meta-analyses Explanation and elaboration. Clin Chem 49: 7–18. (overviews) using updated individual patient data. Cochrane Working Group. 19. Vandenbroucke JP, von Elm E, Altman DG, Gøtzsche PC, Mulrow CD, et al. Stat Med 14: 2057–2079. (2007) Strengthening the Reporting of Observational Studies in Epidemiology 42. Chan AW, Hrobjartsson A, Haahr MT, Gøtzsche PC, Altman DG (2004) (STROBE): Explanation and elaboration. PLoS Med 4: e297. doi:10.1371/ Empirical evidence for selective reporting of outcomes in randomized trials: journal.pmed.0040297. Comparison of protocols to published articles. JAMA 291: 2457–2465. 20. Barker A, Maratos EC, Edmonds L, Lim E (2007) Recurrence rates of video- 43. Dwan K, Altman DG, Arnaiz JA, Bloom J, Chan AW, et al. (2008) Systematic assisted thoracoscopic versus open surgery in the prevention of recurrent pneumothoraces: A systematic review of randomised and non-randomised review of the empirical evidence of study publication bias and outcome trials. Lancet 370: 329–335. reporting bias. PLoS ONE 3: e3081. doi:10.1371/journal.pone.0003081. PLoS Medicine | www.plosmedicine.org 24 July 2009 | Volume 6 | Issue 7 | e1000100 44. Silagy CA, Middleton P, Hopewell S (2002) Publishing protocols of systematic 72. Major MP, Major PW, Flores-Mir C (2006) An evaluation of search and reviews: Comparing what was done to what was planned. JAMA 287: selection methods used in dental systematic reviews published in English. J Am 2831–2834. Dent Assoc 137: 1252–1257. 45. Centre for Reviews and Dissemination (2009) Research projects. York: 73. Major MP, Major PW, Flores-Mir C (2007) Benchmarking of reported search University of York, Available: http://www.crd.york.ac.uk/crdweb. Accessed and selection methods of systematic reviews by dental speciality. Evid Based 26 May 2009. Dent 8: 66–70. 46. The Joanna Briggs Institute (2009) Protocols & work in progress. Available: 74. Shah MR, Hasselblad V, Stevenson LW, Binanay C, O’Connor CM, et al. http://www.joannabriggs.edu.au/pubs/systematic_reviews_prot.php. Ac- (2005) Impact of the pulmonary artery catheter in critically ill patients: Meta- cessed 26 May 2009. analysis of randomized clinical trials. JAMA 294: 1664–1670. 47. Bagshaw SM, McAlister FA, Manns BJ, Ghali WA (2006) Acetylcysteine in the 75. Edwards P, Clarke M, DiGuiseppi C, Pratap S, Roberts I, et al. (2002) prevention of contrast-induced nephropathy: A case study of the pitfalls in the Identification of randomized controlled trials in systematic reviews: Accuracy evolution of evidence. Arch Intern Med 166: 161–166. and reliability of screening records. Stat Med 21: 1635–1640. 48. Biondi-Zoccai GG, Lotrionte M, Abbate A, Testa L, Remigi E, et al. (2006) 76. Cooper HM, Ribble RG (1989) Influences on the outcome of literature Compliance with QUOROM and quality of reporting of overlapping meta- searches for integrative research reviews. Knowledge 10: 179–201. analyses on the role of acetylcysteine in the prevention of contrast associated 77. Mistiaen P, Poot E (2006) Telephone follow-up, initiated by a hospital-based nephropathy: Case study. BMJ 332: 202–209. health professional, for postdischarge problems in patients discharged from 49. Sacks HS, Berrier J, Reitman D, Ancona-Berk VA, Chalmers TC (1987) Meta- hospital to home. Cochrane Database Syst Rev Issue 4: CD004510. analyses of randomized controlled trials. N Engl J Med 316: 450–455. doi:10.1002/14651858.CD004510.pub3. 50. Schroth RJ, Hitchon CA, Uhanova J, Noreddin A, Taback SP, et al. (2004) 78. Jones AP, Remmington T, Williamson PR, Ashby D, Smyth RL (2005) High Hepatitis B vaccination for patients with chronic renal failure. Cochrane prevalence but low impact of data extraction and reporting errors were found Database Syst Rev Issue 3: CD003775. doi:10.1002/14651858.CD003775. in Cochrane systematic reviews. J Clin Epidemiol 58: 741–742. pub2. 79. Clarke M, Hopewell S, Juszczak E, Eisinga A, Kjeldstrom M (2006) 51. Egger M, Zellweger-Zahner T, Schneider M, Junker C, Lengeler C, et al. Compression stockings for preventing deep vein thrombosis in airline (1997) Language bias in randomised controlled trials published in English and passengers. Cochrane Database Syst Rev Issue 2: CD004002. doi:10.1002/ German. Lancet 350: 326–329. 14651858.CD004002.pub2. 52. Gregoire G, Derderian F, Le Lorier J (1995) Selecting the language of the 80. Tramer MR, Reynolds DJ, Moore RA, McQuay HJ (1997) Impact of covert publications included in a meta-analysis: Is there a Tower of Babel bias? J Clin duplicate publication on meta-analysis: A case study. BMJ 315: 635–640. Epidemiol 48: 159–163. 81. von Elm E, Poglia G, Walder B, Tramer MR (2004) Different patterns of 53. Ju¨ni P, Holenstein F, Sterne J, Bartlett C, Egger M (2002) Direction and impact duplicate publication: An analysis of articles used in systematic reviews. JAMA of language bias in meta-analyses of controlled trials: Empirical study. 291: 974–980. Int J Epidemiol 31: 115–123. 82. Gotzsche PC (1989) Multiple publication of reports of drug trials. Eur J Clin 54. Moher D, Pham B, Klassen TP, Schulz KF, Berlin JA, et al. (2000) What Pharmacol 36: 429–432. contributions do languages other than English make on the results of meta- 83. Allen C, Hopewell S, Prentice A (2005) Non-steroidal anti-inflammatory drugs analyses? J Clin Epidemiol 53: 964–972. for pain in women with endometriosis. Cochrane Database Syst Rev Issue 4: 55. Pan Z, Trikalinos TA, Kavvoura FK, Lau J, Ioannidis JP (2005) Local CD004753. doi:10.1002/14651858.CD004753.pub2. literature bias in genetic epidemiology: An empirical evaluation of the Chinese 84. Glasziou P, Meats E, Heneghan C, Shepperd S (2008) What is missing from literature. PLoS Med 2: e334. doi:10.1371/journal.pmed.0020334. descriptions of treatment in trials and reviews? BMJ 336: 1472–1474. 56. Hopewell S, McDonald S, Clarke M, Egger M (2007) Grey literature in meta- 85. Tracz MJ, Sideras K, Bolona ER, Haddad RM, Kennedy CC, et al. (2006) analyses of randomized trials of health care interventions. Cochrane Database Testosterone use in men and its effects on bone health. A systematic review and Syst Rev Issue 2: MR000010. doi:10.1002/14651858.MR000010.pub3. meta-analysis of randomized placebo-controlled trials. J Clin Endocrinol Metab 57. Melander H, Ahlqvist-Rastad J, Meijer G, Beermann B (2003) Evidence 91: 2011–2016. b(i)ased medicine—Selective reporting from studies sponsored by pharmaceu- 86. Bucher HC, Hengstler P, Schindler C, Guyatt GH (2000) Percutaneous tical industry: Review of studies in new drug applications. BMJ 326: transluminal coronary angioplasty versus medical treatment for non-acute 1171–1173. coronary heart disease: Meta-analysis of randomised controlled trials. BMJ 321: 58. Sutton AJ, Duval SJ, Tweedie RL, Abrams KR, Jones DR (2000) Empirical 73–77. assessment of effect of publication bias on meta-analyses. BMJ 320: 1574–1577. 87. Gluud LL (2006) Bias in clinical intervention research. Am J Epidemiol 163: 59. Gotzsche PC (2006) Believability of relative risks and odds ratios in abstracts: 493–501. Cross sectional study. BMJ 333: 231–234. 88. Pildal J, Hro´bjartsson A, Jorgensen KJ, Hilden J, Altman DG, et al. (2007) 60. Bhandari M, Devereaux PJ, Guyatt GH, Cook DJ, Swiontkowski MF, et al. Impact of allocation concealment on conclusions drawn from meta-analyses of (2002) An observational study of orthopaedic abstracts and subsequent full-text randomized trials. Int J Epidemiol 36: 847–857. publications. J Bone Joint Surg Am 84-A: 615–621. 89. Moja LP, Telaro E, D’Amico R, Moschetti I, Coe L, et al. (2005) Assessment of 61. Rosmarakis ES, Soteriades ES, Vergidis PI, Kasiakou SK, Falagas ME (2005) methodological quality of primary studies by systematic reviews: Results of the From conference abstract to full paper: Differences between data presented in metaquality cross sectional study. BMJ 330: 1053. conferences and journals. Faseb J 19: 673–680. 90. Moher D, Jadad AR, Tugwell P (1996) Assessing the quality of randomized 62. Toma M, McAlister FA, Bialy L, Adams D, Vandermeer B, et al. (2006) controlled trials. Current issues and future directions. Int J Technol Assess Transition from meeting abstract to full-length journal article for randomized Health Care 12: 195–208. controlled trials. JAMA 295: 1281–1287. 91. Sanderson S, Tatt ID, Higgins JP (2007) Tools for assessing quality and 63. Saunders Y, Ross JR, Broadley KE, Edmonds PM, Patel S (2004) Systematic susceptibility to bias in observational studies in epidemiology: A systematic review of bisphosphonates for hypercalcaemia of malignancy. Palliat Med 18: review and annotated bibliography. Int J Epidemiol 36: 666–676. 418–431. 92. Greenland S (1994) Invited commentary: A critical look at some popular meta- 64. Shojania KG, Sampson M, Ansari MT, Ji J, Doucette S, et al. (2007) How analytic methods. Am J Epidemiol 140: 290–296. quickly do systematic reviews go out of date? A survival analysis. Ann Intern 93. Ju¨ni P, Altman DG, Egger M (2001) Systematic reviews in health care: Med 147: 224–233. Assessing the quality of controlled clinical trials. BMJ 323: 42–46. 65. Bergerhoff K, Ebrahim S, Paletta G (2004) Do we need to consider ‘in process 94. Kunz R, Oxman AD (1998) The unpredictability paradox: Review of empirical citations’ for search strategies? 12th Cochrane Colloquium; 2–6 October 2004; comparisons of randomised and non-randomised clinical trials. BMJ 317: Ottawa, Ontario, Canada. Available: http://www.cochrane.org/colloquia/ 1185–1190. abstracts/ottawa/P-039.htm. Accessed 26 May 2009. 95. Balk EM, Bonis PA, Moskowitz H, Schmid CH, Ioannidis JP, et al. (2002) 66. Zhang L, Sampson M, McGowan J (2006) Reporting of the role of expert Correlation of quality measures with estimates of treatment effect in meta- searcher in Cochrane reviews. Evid Based Libr Info Pract 1: 3–16. analyses of randomized controlled trials. JAMA 287: 2973–2982. 67. Turner EH, Matthews AM, Linardatos E, Tell RA, Rosenthal R (2008) 96. Devereaux PJ, Beattie WS, Choi PT, Badner NH, Guyatt GH, et al. (2005) Selective publication of antidepressant trials and its influence on apparent How strong is the evidence for the use of perioperative beta blockers in non- efficacy. N Engl J Med 358: 252–260. cardiac surgery? Systematic review and meta-analysis of randomised controlled 68. Alejandria MM, Lansang MA, Dans LF, Mantaring JB (2002) Intravenous trials. BMJ 331: 313–321. immunoglobulin for treating sepsis and septic shock. Cochrane Database Syst 97. Devereaux PJ, Bhandari M, Montori VM, Manns BJ, Ghali WA, et al. (2002) Rev Issue 1: CD001090. doi:10.1002/14651858.CD001090. Double blind, you are the weakest link—Good-bye! ACP J Club 136: A11. 69. Golder S, McIntosh HM, Duffy S, Glanville J (2006) Developing efficient 98. van Nieuwenhoven CA, Buskens E, van Tiel FH, Bonten MJ (2001) search strategies to identify reports of adverse effects in MEDLINE and Relationship between methodological trial quality and the effects of selective EMBASE. Health Info Libr J 23: 3–12. digestive decontamination on pneumonia and mortality in critically ill patients. 70. Sampson M, McGowan J, Cogo E, Grimshaw J, Moher D, et al. (2009) An JAMA 286: 335–340. evidence-based practice guideline for the peer review of electronic search strategies. J Clin Epidemiol, E-pub 2009 February 18. 99. Guyatt GH, Cook D, Devereaux PJ, Meade M, Straus S (2002) Therapy. 71. Flores-Mir C, Major MP, Major PW (2006) Search and selection methodology Users’ guides to the medical literature. AMA Press. pp 55–79. of systematic reviews in orthodontics (2000–2004). Am J Orthod Dentofacial 100. Sackett DL, Gent M (1979) Controversy in counting and attributing events in Orthop 130: 214–217. clinical trials. N Engl J Med 301: 1410–1412. PLoS Medicine | www.plosmedicine.org 25 July 2009 | Volume 6 | Issue 7 | e1000100 101. Montori VM, Devereaux PJ, Adhikari NK, Burns KE, Eggert CH, et al. (2005) 130. Fuccio L, Minardi ME, Zagari RM, Grilli D, Magrini N, et al. (2007) Meta- Randomized trials stopped early for benefit: A systematic review. JAMA 294: analysis: Duration of first-line proton-pump inhibitor based triple therapy for 2203–2209. Helicobacter pylori eradication. Ann Intern Med 147: 553–562. 102. Guyatt GH, Devereaux PJ (2002) Therapy and validity: The principle of 131. Egger M, Smith GD (1998) Bias in location and selection of studies. BMJ 316: intention-to-treat. In: Guyatt GH, Rennie DR, eds. Users’ guides to the 61–66. medical literature AMA Press. pp 267–273. 132. Ravnskov U (1992) Cholesterol lowering trials in coronary heart disease: 103. Berlin JA (1997) Does blinding of readers affect the results of meta-analyses? Frequency of citation and outcome. BMJ 305: 15–19. University of Pennsylvania Meta-analysis Blinding Study Group. Lancet 350: 133. Hind D, Booth A (2007) Do health technology assessments comply with 185–186. QUOROM diagram guidance? An empirical study. BMC Med Res Methodol 104. Jadad AR, Moore RA, Carroll D, Jenkinson C, Reynolds DJ, et al. (1996) 7: 49. Assessing the quality of reports of randomized clinical trials: Is blinding 134. Curioni C, Andre C (2006) Rimonabant for overweight or obesity. Cochrane necessary? Control Clin Trials 17: 1–12. Database Syst Rev Issue 4: CD006162. doi:10.1002/14651858.CD006162. 105. Pittas AG, Siegel RD, Lau J (2004) Insulin therapy for critically ill hospitalized pub2. patients: A meta-analysis of randomized controlled trials. Arch Intern Med 164: 135. DeCamp LR, Byerley JS, Doshi N, Steiner MJ (2008) Use of antiemetic agents 2005–2011. in acute gastroenteritis: A systematic review and meta-analysis. Arch Pediatr 106. Lakhdar R, Al-Mallah MH, Lanfear DE (2008) Safety and tolerability of Adolesc Med 162: 858–865. angiotensin-converting enzyme inhibitor versus the combination of angioten- 136. Pakos EE, Ioannidis JP (2004) Radiotherapy vs. nonsteroidal anti-inflammatory sin-converting enzyme inhibitor and angiotensin receptor blocker in patients drugs for the prevention of heterotopic ossification after major hip procedures: with left ventricular dysfunction: A systematic review and meta-analysis of A meta-analysis of randomized trials. Int J Radiat Oncol Biol Phys 60: randomized controlled trials. J Card Fail 14: 181–188. 888–895. 107. Bobat R, Coovadia H, Stephen C, Naidoo KL, McKerrow N, et al. (2005) 137. Skalsky K, Yahav D, Bishara J, Pitlik S, Leibovici L, et al. (2008) Treatment of Safety and efficacy of zinc supplementation for children with HIV-1 infection human brucellosis: Systematic review and meta-analysis of randomised in South Africa: A randomised double-blind placebo-controlled trial. Lancet controlled trials. BMJ 336: 701–704. 366: 1862–1867. 138. Altman DG, Cates C (2001) The need for individual trial results in reports of 108. Deeks JJ, Altman DG (2001) Effect measures for meta-analysis of trials with systematic reviews. BMJ. Rapid response. binary outcomes. In: Egger M, Smith GD, Altman DG, eds. Systematic reviews 139. Gotzsche PC, Hrobjartsson A, Maric K, Tendal B (2007) Data extraction in healthcare: Meta-analysis in context. 2nd edition. London: BMJ Publishing errors in meta-analyses that use standardized mean differences. JAMA 298: Group. 430–437. 109. Deeks JJ (2002) Issues in the selection of a summary statistic for meta-analysis of 140. Lewis S, Clarke M (2001) Forest plots: Trying to see the wood and the trees. clinical trials with binary outcomes. Stat Med 21: 1575–1600. BMJ 322: 1479–1480. 110. Engels EA, Schmid CH, Terrin N, Olkin I, Lau J (2000) Heterogeneity and 141. Papanikolaou PN, Ioannidis JP (2004) Availability of large-scale evidence on statistical significance in meta-analysis: An empirical study of 125 meta- specific harms from systematic reviews of randomized trials. Am J Med 117: analyses. Stat Med 19: 1707–1728. 582–589. 111. Tierney JF, Stewart LA, Ghersi D, Burdett S, Sydes MR (2007) Practical 142. Duffett M, Choong K, Ng V, Randolph A, Cook DJ (2007) Surfactant therapy methods for incorporating summary time-to-event data into meta-analysis. for acute respiratory failure in children: A systematic review and meta-analysis. Trials 8: 16. Crit Care 11: R66. 112. Michiels S, Piedbois P, Burdett S, Syz N, Stewart L, et al. (2005) Meta-analysis 143. Balk E, Raman G, Chung M, Ip S, Tatsioni A, et al. (2006) Effectiveness of when only the median survival times are known: A comparison with individual management strategies for renal artery stenosis: A systematic review. Ann patient data results. Int J Technol Assess Health Care 21: 119–125. Intern Med 145: 901–912. 113. Briel M, Studer M, Glass TR, Bucher HC (2004) Effects of statins on stroke 144. Palfreyman S, Nelson EA, Michaels JA (2007) Dressings for venous leg ulcers: prevention in patients with and without coronary heart disease: A meta-analysis Systematic review and meta-analysis. BMJ 335: 244. of randomized controlled trials. Am J Med 117: 596–606. 145. Ioannidis JP, Patsopoulos NA, Evangelou E (2007) Uncertainty in heteroge- 114. Jones M, Schenkel B, Just J, Fallowfield L (2004) Epoetin alfa improves quality neity estimates in meta-analyses. BMJ 335: 914–916. of life in patients with cancer: Results of metaanalysis. Cancer 101: 1720–1732. 146. Appleton KM, Hayward RC, Gunnell D, Peters TJ, Rogers PJ, et al. (2006) 115. Elbourne DR, Altman DG, Higgins JP, Curtin F, Worthington HV, et al. Effects of n-3 long-chain polyunsaturated fatty acids on depressed mood: (2002) Meta-analyses involving cross-over trials: Methodological issues. Systematic review of published trials. Am J Clin Nutr 84: 1308–1316. Int J Epidemiol 31: 140–149. 147. Kirsch I, Deacon BJ, Huedo-Medina TB, Scoboria A, Moore TJ, et al. (2008) 116. Follmann D, Elliott P, Suh I, Cutler J (1992) Variance imputation for overviews Initial severity and antidepressant benefits: A meta-analysis of data submitted to of clinical trials with continuous response. J Clin Epidemiol 45: 769–773. the Food and Drug Administration. PLoS Med 5: e45. doi:10.1371/ 117. Wiebe N, Vandermeer B, Platt RW, Klassen TP, Moher D, et al. (2006) A journal.pmed.0050045. systematic review identifies a lack of standardization in methods for handling 148. Reichenbach S, Sterchi R, Scherer M, Trelle S, Burgi E, et al. (2007) Meta- missing variance data. J Clin Epidemiol 59: 342–353. analysis: Chondroitin for osteoarthritis of the knee or hip. Ann Intern Med 146: 118. Hrobjartsson A, Gotzsche PC (2004) Placebo interventions for all clinical 580–590. conditions. Cochrane Database Syst Rev Issue 2: CD003974. doi:10.1002/ 149. Hodson EM, Craig JC, Strippoli GF, Webster AC (2008) Antiviral medications 14651858.CD003974.pub2. for preventing cytomegalovirus disease in solid organ transplant recipients. 119. Shekelle PG, Morton SC, Maglione M, Suttorp M, Tu W, et al. (2004) Cochrane Database Syst Rev Issue 2: CD003774. doi:10.1002/ Pharmacological and surgical treatment of obesity. Evid Rep Technol Assess 14651858.CD003774.pub3. (Summ). pp 1–6. 150. Thompson SG, Higgins JP (2002) How should meta-regression analyses be 120. Chan AW, Altman DG (2005) Identifying outcome reporting bias in undertaken and interpreted? Stat Med 21: 1559–1573. randomised trials on PubMed: Review of publications and survey of authors. 151. Chan AW, Krleza-Jeric K, Schmid I, Altman DG (2004) Outcome reporting BMJ 330: 753. bias in randomized trials funded by the Canadian Institutes of Health 121. Williamson PR, Gamble C (2005) Identification and impact of outcome Research. CMAJ 171: 735–740. selection bias in meta-analysis. Stat Med 24: 1547–1561. 152. Hahn S, Williamson PR, Hutton JL, Garner P, Flynn EV (2000) Assessing the 122. Williamson PR, Gamble C, Altman DG, Hutton JL (2005) Outcome selection potential for bias in meta-analysis due to selective reporting of subgroup bias in meta-analysis. Stat Methods Med Res 14: 515–524. analyses within studies. Stat Med 19: 3325–3336. 123. Ioannidis JP, Trikalinos TA (2007) The appropriateness of asymmetry tests for 153. Green LW, Glasgow RE (2006) Evaluating the relevance, generalization, and publication bias in meta-analyses: A large survey. CMAJ 176: 1091–1096. applicability of research: Issues in external validation and translation 124. Briel M, Schwartz GG, Thompson PL, de Lemos JA, Blazing MA, et al. (2006) methodology. Eval Health Prof 29: 126–153. Effects of early treatment with statins on short-term clinical outcomes in acute 154. Liberati A, D’Amico R, Pifferi, Torri V, Brazzi L (2004) Antibiotic prophylaxis coronary syndromes: A meta-analysis of randomized controlled trials. JAMA to reduce respiratory tract infections and mortality in adults receiving intensive 295: 2046–2056. care. Cochrane Database Syst Rev Issue 1: CD000022. doi:10.1002/ 125. Song F, Eastwood AJ, Gilbody S, Duley L, Sutton AJ (2000) Publication and 14651858.CD000022.pub2. related biases. Health Technol Assess 4: 1–115. 155. Gonzalez R, Zamora J, Gomez-Camarero J, Molinero LM, Banares R, et al. 126. Schmid CH, Stark PC, Berlin JA, Landais P, Lau J (2004) Meta-regression (2008) Meta-analysis: Combination endoscopic and drug therapy to prevent detected associations between heterogeneous treatment effects and study-level, variceal rebleeding in cirrhosis. Ann Intern Med 149: 109–122. but not patient-level, factors. J Clin Epidemiol 57: 683–697. 156. D’Amico R, Pifferi S, Leonetti C, Torri V, Tinazzi A, et al. (1998) Effectiveness 127. Higgins JP, Thompson SG (2004) Controlling the risk of spurious findings from of antibiotic prophylaxis in critically ill adult patients: Systematic review of meta-regression. Stat Med 23: 1663–1682. randomised controlled trials. BMJ 316: 1275–1285. 128. Thompson SG, Higgins JP (2005) Treating individuals 4: Can meta-analysis 157. Olsen O, Middleton P, Ezzo J, Gotzsche PC, Hadhazy V, et al. (2001) Quality help target interventions at individuals most likely to benefit? Lancet 365: 341–346. of Cochrane reviews: Assessment of sample from 1998. BMJ 323: 829–832. 129. Uitterhoeve RJ, Vernooy M, Litjens M, Potting K, Bensing J, et al. (2004) 158. Hopewell S, Wolfenden L, Clarke M (2008) Reporting of adverse events in Psychosocial interventions for patients with advanced cancer—A systematic systematic reviews can be improved: Survey results. J Clin Epidemiol 61: review of the literature. Br J Cancer 91: 1050–1062. 597–602. PLoS Medicine | www.plosmedicine.org 26 July 2009 | Volume 6 | Issue 7 | e1000100 Available: http://www.york.ac.uk/inst/crd/pdf/appm.pdf. Accessed 26 May 159. Cook DJ, Reeve BK, Guyatt GH, Heyland DK, Griffith LE, et al. (1996) Stress ulcer prophylaxis in critically ill patients. Resolving discordant meta-analyses. JAMA 275: 308–314. 188. Moher D, Cook DJ, Jadad AR, Tugwell P, Moher M, et al. (1999) Assessing the 160. Jadad AR, Cook DJ, Browman GP (1997) A guide to interpreting discordant quality of reports of randomised trials: Implications for the conduct of meta- systematic reviews. CMAJ 156: 1411–1416. analyses. Health Technol Assess 3: i–iv, 1–98. 161. Clarke L, Clarke M, Clarke T (2007) How useful are Cochrane reviews in 189. Devereaux PJ, Choi PT, El-Dika S, Bhandari M, Montori VM, et al. (2004) An identifying research needs? J Health Serv Res Policy 12: 101–103. observational study found that authors of randomized controlled trials frequently use concealment of randomization and blinding, despite the failure 162. [No authors listed] (2000) World Medical Association Declaration of Helsinki: to report these methods. J Clin Epidemiol 57: 1232–1236. Ethical principles for medical research involving human subjects. JAMA 284: 190. Soares HP, Daniels S, Kumar A, Clarke M, Scott C, et al. (2004) Bad reporting 3043–3045. does not mean bad methods for randomised trials: Observational study of 163. Clarke M, Hopewell S, Chalmers I (2007) Reports of clinical trials should begin randomised controlled trials performed by the Radiation Therapy Oncology and end with up-to-date systematic reviews of other relevant evidence: A status Group. BMJ 328: 22–24. report. J R Soc Med 100: 187–190. 191. Liberati A, Himel HN, Chalmers TC (1986) A quality assessment of 164. Dube C, Rostom A, Lewin G, Tsertsvadze A, Barrowman N, et al. (2007) The randomized control trials of primary treatment of breast cancer. J Clin Oncol use of aspirin for primary prevention of colorectal cancer: A systematic review 4: 942–951. prepared for the U.S. Preventive Services Task Force. Ann Intern Med 146: 192. Moher D, Jadad AR, Nichol G, Penman M, Tugwell P, et al. (1995) Assessing 365–375. the quality of randomized controlled trials: An annotated bibliography of scales 165. Critchley J, Bates I (2005) Haemoglobin colour scale for anaemia diagnosis and checklists. Control Clin Trials 16: 62–73. where there is no laboratory: A systematic review. Int J Epidemiol 34: 193. Greenland S, O’Rourke K (2001) On the bias produced by quality scores in 1425–1434. meta-analysis, and a hierarchical view of proposed solutions. Biostatistics 2: 166. Lexchin J, Bero LA, Djulbegovic B, Clark O (2003) Pharmaceutical industry 463–471. sponsorship and research outcome and quality: Systematic review. BMJ 326: 194. Ju¨ni P, Witschi A, Bloch R, Egger M (1999) The hazards of scoring the quality 1167–1170. of clinical trials for meta-analysis. JAMA 282: 1054–1060. 167. Als-Nielsen B, Chen W, Gluud C, Kjaergard LL (2003) Association of funding 195. Fleiss JL (1993) The statistical basis of meta-analysis. Stat Methods Med Res 2: and conclusions in randomized drug trials: A reflection of treatment effect or 121–145. adverse events? JAMA 290: 921–928. 196. Villar J, Mackey ME, Carroli G, Donner A (2001) Meta-analyses in systematic 168. Peppercorn J, Blood E, Winer E, Partridge A (2007) Association between reviews of randomized controlled trials in perinatal medicine: Comparison of pharmaceutical involvement and outcomes in breast cancer clinical trials. fixed and random effects models. Stat Med 20: 3635–3647. Cancer 109: 1239–1246. 197. Lau J, Ioannidis JP, Schmid CH (1998) Summing up evidence: One answer is 169. Yank V, Rennie D, Bero LA (2007) Financial ties and concordance between not always enough. Lancet 351: 123–127. results and conclusions in meta-analyses: Retrospective cohort study. BMJ 335: 198. DerSimonian R, Laird N (1986) Meta-analysis in clinical trials. Control Clin 1202–1205. Trials 7: 177–188. 170. Jorgensen AW, Hilden J, Gøtzsche PC (2006) Cochrane reviews compared with 199. Hunter JE, Schmidt FL (2000) Fixed effects vs. random effects meta-analysis industry supported meta-analyses and other meta-analyses of the same drugs: models: Implications for cumulative research knowledge. Int J Sel Assess 8: Systematic review. BMJ 333: 782. 275–292. 171. Gotzsche PC, Hrobjartsson A, Johansen HK, Haahr MT, Altman DG, et al. 200. Deeks JJ, Altman DG, Bradburn MJ (2001) Statistical methods for examining (2007) Ghost authorship in industry-initiated randomised trials. PLoS Med 4: heterogeneity and combining results from several studies in meta-analysis. In: e19. doi:10.1371/journal.pmed.0040019. Egger M, Davey Smith G, Altman DG, eds. Systematic reviews in healthcare: 172. Akbari A, Mayhew A, Al-Alawi M, Grimshaw J, Winkens R, et al. (2008) Meta-analysis in context. London: BMJ Publishing Group. pp 285–312. Interventions to improve outpatient referrals from primary care to secondary 201. Warn DE, Thompson SG, Spiegelhalter DJ (2002) Bayesian random effects care. Cochrane Database Syst Rev Issue 2: CD005471. doi:10.1002/ meta-analysis of trials with binary outcomes: Methods for the absolute risk 14651858.CD005471.pub2. difference and relative risk scales. Stat Med 21: 1601–1623. 173. Davies P, Boruch R (2001) The Campbell Collaboration. BMJ 323: 294–295. 202. Higgins JP, Thompson SG, Deeks JJ, Altman DG (2003) Measuring 174. Pawson R, Greenhalgh T, Harvey G, Walshe K (2005) Realist review—A new inconsistency in meta-analyses. BMJ 327: 557–560. method of systematic review designed for complex policy interventions. J Health 203. Higgins JP, Thompson SG (2002) Quantifying heterogeneity in a meta- Serv Res Policy 10(Suppl 1): 21–34. analysis. Stat Med 21: 1539–1558. 175. Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O, et al. (2005) 204. Huedo-Medina TB, Sanchez-Meca J, Marin-Martinez F, Botella J (2006) Storylines of research in diffusion of innovation: A meta-narrative approach to Assessing heterogeneity in meta-analysis: Q statistic or I2 index? Psychol systematic review. Soc Sci Med 61: 417–430. Methods 11: 193–206. 176. Lumley T (2002) Network meta-analysis for indirect treatment comparisons. 205. Thompson SG, Turner RM, Warn DE (2001) Multilevel models for meta- Stat Med 21: 2313–2324. analysis, and their application to absolute risk differences. Stat Methods Med 177. Salanti G, Higgins JP, Ades AE, Ioannidis JP (2008) Evaluation of networks of Res 10: 375–392. randomized trials. Stat Methods Med Res 17: 279–301. 206. Dickersin K (2005) Publication bias: Recognising the problem, understanding 178. Altman DG, Moher D (2005) [Developing guidelines for reporting healthcare its origin and scope, and preventing harm. In: Rothstein HR, Sutton AJ, research: Scientific rationale and procedures.]. Med Clin (Barc) 125(Suppl 1): Borenstein M, eds. Publication bias in meta-analysis—Prevention, assessment 8–13. and adjustments. West Sussex: John Wiley & Sons. 356 p. 179. Delaney A, Bagshaw SM, Ferland A, Manns B, Laupland KB, et al. (2005) A 207. Scherer RW, Langenberg P, von Elm E (2007) Full publication of results systematic evaluation of the quality of meta-analyses in the critical care initially presented in abstracts. Cochrane Database Syst Rev Issue 2: literature. Crit Care 9: R575–582. MR000005. doi:10.1002/14651858.MR000005.pub3. 180. Altman DG, Simera I, Hoey J, Moher D, Schulz K (2008) EQUATOR: 208. Krzyzanowska MK, Pintilie M, Tannock IF (2003) Factors associated with Reporting guidelines for health research. Lancet 371: 1149–1150. failure to publish large randomized trials presented at an oncology meeting. 181. Plint AC, Moher D, Morrison A, Schulz K, Altman DG, et al. (2006) Does the JAMA 290: 495–501. CONSORT checklist improve the quality of reports of randomised controlled 209. Hopewell S, Clarke M (2001) Methodologists and their methods. Do trials? A systematic review. Med J Aust 185: 263–267. methodologists write up their conference presentations or is it just 15 minutes 182. Simera I, Altman DG, Moher D, Schulz KF, Hoey J (2008) Guidelines for of fame? Int J Technol Assess Health Care 17: 601–603. reporting health research: The EQUATOR network’s survey of guideline 210. Ghersi D (2006) Issues in the design, conduct and reporting of clinical trials that authors. PLoS Med 5: e139. doi:10.1371/journal.pmed.0050139. impact on the quality of decision making. PhD thesis. Sydney: School of Public 183. Last JM (2001) A dictionary of epidemiology. Oxford: Oxford University Press Health, Faculty of Medicine, University of Sydney. & International Epidemiological Association. 211. von Elm E, Rollin A, Blumle A, Huwiler K, Witschi M, et al. (2008) Publication 184. Antman EM, Lau J, Kupelnick B, Mosteller F, Chalmers TC (1992) A and non-publication of clinical trials: Longitudinal study of applications comparison of results of meta-analyses of randomized control trials and submitted to a research ethics committee. Swiss Med Wkly 138: 197–203. recommendations of clinical experts. Treatments for myocardial infarction. 212. Sterne JA, Egger M (2001) Funnel plots for detecting bias in meta-analysis: JAMA 268: 240–248. Guidelines on choice of axis. J Clin Epidemiol 54: 1046–1055. 185. Oxman AD, Guyatt GH (1993) The science of reviewing research. 213. Harbord RM, Egger M, Sterne JA (2006) A modified test for small-study effects Ann N Y Acad Sci 703: 125–133; discussion 133–124. in meta-analyses of controlled trials with binary endpoints. Stat Med 25: 186. O’Connor D, Green S, Higgins JPT (2008) Chapter 5: Defining the review 3443–3457. question and developing criteria for including studies. In: Higgins JPT, Green S, 214. Peters JL, Sutton AJ, Jones DR, Abrams KR, Rushton L (2006) Comparison of eds. Cochrane handbook for systematic reviews of interventions version 5.0.0 two methods to detect publication bias in meta-analysis. JAMA 295: 676–680. [updated February 2008]. The Cochrane Collaboration, Available: http:// 215. Rothstein HR, Sutton AJ, Borenstein M (2005) Publication bias in meta- www.cochrane-handbook.org/. Accessed 26 May 2009. analysis: Prevention, assessment and adjustments. West Sussex: John Wiley & 187. McDonagh M, Whiting P, Bradley M, Cooper J, Sutton A, et al. (2000) A Sons. systematic review of public water fluoridation. Protocol changes (Appendix M). 216. Lau J, Ioannidis JP, Terrin N, Schmid CH, Olkin I (2006) The case of the NHS Centre for Reviews and Dissemination. York: University of York, misleading funnel plot. BMJ 333: 597–600. PLoS Medicine | www.plosmedicine.org 27 July 2009 | Volume 6 | Issue 7 | e1000100 217. Terrin N, Schmid CH, Lau J (2005) In an empirical evaluation of the funnel 220. Sterne JAC, Egger M, Moher D (2008) Chapter 10: Addressing reporting plot, researchers could not visually identify publication bias. J Clin Epidemiol biases. In: Higgins JPT, Green S, eds. Cochrane handbook for systematic 58: 894–901. reviews of interventions version 5.0.0 [updated February 2008]. The Cochrane 218. Egger M, Davey Smith G, Schneider M, Minder C (1997) Bias in meta-analysis Collaboration, Available: http://www.cochrane-handbook.org/. Accessed 26 detected by a simple, graphical test. BMJ 315: 629–634. May 2009. 219. Ioannidis JP, Trikalinos TA (2007) An exploratory test for an excess of significant findings. Clin Trials 4: 245–253. PLoS Medicine | www.plosmedicine.org 28 July 2009 | Volume 6 | Issue 7 | e1000100

Journal

Annals of Internal MedicineUnpaywall

Published: Aug 18, 2009

There are no references for this article.