Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

A Transparent Process for “Evidence‐Informed” Policy Making

A Transparent Process for “Evidence‐Informed” Policy Making Introduction There has been much discussion about how to incorporate research evidence into policy, especially in public health (Black ; Bowen & Zwi ; Macintyre ; Rutten ; Sutherland et al . ). Major political institutions, such as the European Commission (EC), claim to facilitate use of the best available scientific knowledge in decision‐making (European Commission ). Despite this, it is often difficult to decipher which evidence contributed to a policy, or how it contributed. For example, Sharman & Holmes ( , p. 318) doubted “that the breadth of scientific data available on the potential [greenhouse gas] emissions from biofuels production was adequately factored into the final policy decision” by the EC to set a 10% renewable content target for road transport fuels by 2020. We propose a consistent and cumulative approach to incorporating research evidence into policy, with ongoing systematic collation of evidence relating to all possible policy options, combined with expert evaluation. Such an approach now routinely informs medical practice, following a hierarchical model, in which studies are evaluated for evidence quality and synthesized in systematic reviews, then condensed into “synopses” of digestible, practical information to support clinical decisions (Sackett et al . ; Dicenso et al . ; Moja & Banzi ). Synopses are a vital element of the evidence‐based framework, because they compile evidence across a range of different possible options that a practitioner or decision‐maker has to decide between. In other areas of applied science, including environmental science, the systematic review has become an established tool for incorporating science into policy (Keene & Pullin ), but the synopsis layer is only just emerging. Global synopses of evidence relevant to conservation have been published for wild bees (Dicks et al . ) and birds (Williams et al . ). As part of the Conservation Evidence project (Conservation Evidence ), synopses are currently being prepared for bats, carnivores, amphibians, reptiles, forests, and control of UK freshwater invasive species. Here, we show how a synopsis can inform policy, using the ongoing restructuring of the European Union's (EU) Common Agricultural Policy (CAP) as an example. The process demonstrates that elements of the proposed reform are not supported by experimental evidence. It identifies areas of uncertainty and presents alternative options supported by evidence. In areas such as environment or public health, policy‐relevant studies are often scant and too heterogeneous to allow formal meta‐analysis. Furthermore, results often differ depending on the contexts in which interventions were applied and the responses measured. Policy makers want to know the likelihood of an intervention working under specific sets of conditions. Although highly valuable, a generic effect size across all studies, as presented by systematic reviews, may not always provide sufficient detail. However, once a narrative synopsis is collated, the body of evidence relating to a given problem can be assessed or evaluated rapidly using expert judgment, as shown here. This second stage of assessing evidence can be completed sufficiently quickly to respond to changing political or economic conditions and provides a clear audit trail, allowing scrutiny of the research evidence supporting policy decisions. European agriculture: a case study The CAP is the EU's biggest single expenditure, accounting for over €57 billion annually in 2010 and 2011 (European Commission ), over 40% of the total EU budget. It has two separate “pillars.” Pillar I, comprising 75% of spending, entails annual direct payments to farmers and market measures such as purchasing surplus produce. Pillar II entails multiannual rural development and voluntary environmental measures, including agrienvironment schemes designed and implemented at member state level. At present, the CAP is going through a process of reform. A new structure is being developed, due to be implemented at the start of 2015. A crucial aspect of the reform proposals (European Commission ; Defra ) is that 30% of direct payments within Pillar I should become conditional upon “compulsory greening measures” that address climate and environmental policy goals. This is intended to ensure “that all EU farmers […] deliver environmental and climate benefits as part of their everyday activities” (European Commission ). Three “compulsory greening measures” have been identified, each with ecosystem and habitat maintenance as one of its objectives (European Commission , ). These measures, incorporating the amendments voted for by the European Parliament in March 2013 (European Parliament News ), are as follows: Crop diversification (ensuring at least three crops annually) on farms with an arable area >30 ha. Maintenance of the proportion of total agricultural land area managed as permanent grassland and pasture. “Ecological Focus Areas” on 3% of arable land (possibly rising to 7% by 2018), which may comprise: hedgerows, ditches, stone walls, in field trees, ponds, land planted with nitrogen fixing crops, terraces, buffer strips, landscape features, and afforested areas or land left fallow. Independently of the CAP reform process, we compiled a synopsis of experimental evidence for the effects of 116 interventions to protect wildlife in farmland in northern temperate Europe (Dicks et al . ). The process used systematic mapping (Randall & James ), incorporating existing systematic reviews (see Supporting Information for details of the review methods). Evidence was collected from all European countries west of Russia, but not those south of France, Switzerland Austria, Hungary, and Romania. It excluded correlative studies, including only studies that experimentally tested interventions. We recognize that this excludes important evidence in this specific context, but the highest quality of quantitative evidence was considered (as described by Pullin & Knight ) and the process remains valid as a proof‐of‐concept. The synopsis did not cover evidence on nonwildlife aspects of ecosystems, such as water, cultural landscapes, or carbon storage. For 85 of the 116 interventions, we evaluated the evidence based on the combined judgment of 10 experts, using a method of formal consensus development involving iterative scoring and discussion, which took just 3 days (see Supporting Information). The 10 experts were selected to represent the important areas of knowledge required for this exercise. Their combined expertise covered farmland ecology (birds, amphibians, plants, mammals, and insects) and agricultural policy. Three group members came from policy‐ or conservation‐oriented consultancies or charities directly involved in interpreting evidence on farming and conservation for use in policy. This method of expert selection, called “purposive sampling,” is often used to form panels for expert judgment (Hasson et al . ). We adhered to the optimum group size of between 8 and 10 recommended for groups interpreting medical evidence into clinical practice guidelines (Eccles et al . ). This expert evaluation allowed assessment of the evidence for wildlife benefits in northern Europe from the potential compulsory greening measures. We evaluated the certainty of the available evidence based on the question “How much do we understand the extent to which this intervention benefits wildlife on farmland?” and compiled responses of the 10 experts to the question “Based on the evidence presented, does this intervention benefit wildlife?” We did not evaluate the relative magnitude of the effects of these interventions. Twelve interventions corresponded directly or indirectly to the proposed greening measures (Table ). Our synopsis found little experimental evidence to support benefits for wildlife arising from crop diversification, or the overarching greening measure of Ecological Focus Areas. It did find that some of the proposed management options for Ecological Focus Areas could offer real benefits. Three such interventions are supported by good evidence of a benefit to wildlife (67%–69% certainty of knowledge, 9–10/10 experts considered the evidence demonstrates a benefit to wildlife). However, current proposals do not indicate how farmers would be influenced to undertake these particular options, or how management quality could be assured. Expert evaluation of evidence (Dicks et al . ) for interventions relevant to proposed compulsory greening measures in European CAP reform. There is no expert evaluation (*) of interventions for which we captured no evidence, as there were no studies to evaluate Compulsory greening measure Relevant intervention for which compiled evidence evaluated by experts No. of studies Certainty of knowledge (% known) Proportion experts stating “yes this benefits wildlife” Crop diversification Increase crop diversity 4 9 0.0 Ecological Focus Areas Increase the proportion of seminatural habitat in the farmed landscape 5 22 0.2 Specific habitat types suggested for Ecological Focus Areas Hedges Manage hedgerows to benefit wildlife 20 50 0.7 Ditches Manage ditches to benefit wildlife 11 44 0.4 Stone walls Restore or maintain dry stone walls 0 * * In field trees Protect in‐field trees 0 * * Plant in‐field trees 0 * * Ponds Create scrapes and pools 6 30 1.0 Buffer strips Plant grass buffer strips/margins 69 69 0.9 Create uncultivated margins around intensive arable or pasture fields 49 67 1.0 Provide buffer strips alongside water courses 7 17 0.1 Land left fallow Provide (or retain) set‐aside areas in farmland 54 68 0.9 This assessment is similar to comments made about the proposed greening measures by Poláková et al . ( , p.159) in a report financed by the European Commission to look at how CAP can contribute to Europe's biodiversity goals. Poláková et al . ( ) say “Amongst the [Pillar 1] proposals, the measure that perhaps has the most potential to deliver additional environmental benefit is the ‘ecological focus area’… However, the actual magnitude of the benefits will depend on the precise requirements under this measure, which have not yet been determined.” A further eight interventions (Table ) not currently suggested as compulsory greening measures were evaluated as having a high certainty of knowledge (>60%), with unanimous agreement among our experts of a wildlife benefit. Three seem potentially appropriate as specific habitat types within Ecological Focus Areas. Two have potential cobenefits for climate change mitigation in agriculture (UK Committee on Climate Change ). Interventions to benefit wildlife on farmland not included in compulsory greening measures with certainty of knowledge >60% and unanimous expert agreement that they provide benefit to wildlife. One intervention included in Table (create uncultivated margins) also falls into this category. The evidence used is summarized in Dicks et al . ( ) Intervention Number of studies Certainty of knowledge Proportion experts stating “yes this benefits wildlife” Create skylark plots 11 77 1.0 Restore/create species‐rich semi‐natural grassland * 71 74 1.0 Mowing techniques to reduce bird/mammal mortality 8 71 1.0 Reduce fertilizer, pesticide or herbicide use generally † 47 69 1.0 Plant nectar flower mixture/wildflower strips * 105 67 1.0 Use organic rather than mineral fertilizers † 19 66 1.0 Plant wild bird seed or cover mixture * 49 63 1.0 Leave cultivated, uncropped margins or plots 20 63 1.0 *Interventions that could be incorporated into CAP Pillar I, as optional habitat types within Ecological Focus Areas. †Interventions recommended as measures to mitigate climate change, accounted for in calculations of agricultural mitigation potential by the UK Committee on Climate Change ( ). We did not identify an intervention exactly matching the greening requirement to retain permanent grassland. “Maintain species‐rich, seminatural grassland” was listed. This intervention involves specific management measures such as traditional cutting or grazing regimes and avoidance of fertilizer inputs. These are unlikely to be accomplished by the proposed compulsory greening measure, which would simply prevent conversion to cropland or temporary (rotated) grassland. There is good evidence that maintaining species‐rich grassland would provide wildlife benefits. Our synopsis found 22 studies on this intervention. The expert group classed it among interventions for which the relevant literature was not adequately covered by our reviewing method, because the initial literature search was focused on farm management rather than seminatural habitat management. This meant it was excluded from further evaluation. Even so, based on the evidence presented, it was given a score of 57% for certainty, and 8/10 experts answered that it would benefit wildlife. But this evidence would only apply to policy measures that specified types of management appropriate for seminatural species‐rich grassland, as some existing agrienvironment schemes do. Evidence was also compiled for converting existing arable land back to permanent grassland, but this was not considered equivalent because existing permanent grassland is expected to have a higher biodiversity value than reverted land for many years after reversion. One intervention in Table (“provide buffer strips along water courses”) lacks strong evidence for a wildlife benefit, but may be effective at improving water quality. The second greening measure “Retain all permanent grassland” could also benefit climate regulation (through soil carbon storage) and cultural landscapes. Given the shift in environmental policy toward ecosystem services in Europe (European Commission ), and the opportunities for using the CAP to support ecosystem services more broadly (Plieninger et al . ), there is an urgent need to synthesize evidence for the effects of land management options on all ecosystem services, rather than just wildlife. Advantages of combining evidence synopsis with expert evaluation The process of creating a narrative evidence synopsis structured around interventions, combined with expert judgment to evaluate the evidence, facilitates evidence‐informed decision‐making. While science is one of several drivers or value sets that operate in policy making (Sharman & Holmes ), it seems imperative to move toward a transparent means of incorporating science into decisions where and when required. In our example, we compiled only direct, experimental evidence where interventions were implemented and their effects monitored (see Supporting Information). This follows the approach used for presenting evidence to doctors at the point of care in medicine. Predictive modeling and correlative studies were excluded, as they are in medicine. For environmental policy, these study types can provide crucial evidence for decisions when direct experimental evidence is sparse, as is always likely at large spatial scales because of the cost and logistical difficulty of performing landscape‐scale experiments. In the longer term, synopses should be developed that cover all types of evidence relevant to particular policy areas, including, for example, predictive modeling, correlative evidence, economic evidence, and qualitative research. Kept separately, different types of evidence could then be weighted in expert evaluations for given policy questions, following the kind of evidence typology approach suggested by Petticrew & Roberts ( ). For environmental questions, experimental evidence would usually be given the highest weight at a given spatial scale, because it shows what actually happens if you intervene, rather than what you expect to happen based on observed correlation. These are not always the same. For example, many studies have shown that biodiversity in farmland is positively correlated with the proportion of seminatural habitat (Tscharntke et al . ). Yet, the experimental evidence we found for increasing the proportion of seminatural habitat shows mixed effects on biodiversity, rather than a consistently positive effect. The evidence comprises studies monitoring landscape‐level effects of the Swiss Ecological Compensation Areas (ECAs) scheme, which obliged farmers to manage at least 7% of their agricultural land as ECAs from 1998 onwards. This scheme is very similar to the Ecological Focus Areas originally proposed in the CAP compulsory greening measures. The reason for the lack of clear benefits in Switzerland could be because at larger scales, mobile organisms such as birds redistribute in the landscape rather than showing genuine population change in response to agrienvironment schemes or because the implementation of ECAs on Swiss farms was not ideal. Both possibilities deserve further research. The process demonstrated here has three advantages over current efforts to incorporate evidence into policy decisions. First, it provides an evidence audit trail, allowing both contemporaneous and retrospective scrutiny of policy decisions. The individual items of evidence that feed into summary statements for each intervention are easy to locate. The underlying review process is explicit about the search protocol and criteria for including or excluding papers. This is important yet often not the case in policy‐related documents. For example, neither the European Commission's own impact assessment (European Commission ) nor its commissioned research (Poláková et al . ) on the biodiversity effects of the CAP reform proposals contain clear statements of how, or with what criteria, evidence was gathered. Similarly, reviews underlying large‐scale scientific assessments such as those on climate change (Pachauri & Reisinger ), or ecosystems (Hassan et al . ), are carried out by working groups of scientists who cover the literature in their areas of expertise using unspecified methods of search and appraisal. These assessment processes are given authority by the number and breadth of expert reviewers who subsequently comment on the text. It is difficult for people outside the process to scrutinize the way evidence was used, to check for bias in interpretation of the literature or to repeat the process independently. Second, previously synthesized evidence allows rapid responses to proposed policy change. By reviewing evidence for all potential interventions, the slow ongoing process of review is separated from a responsive policy development phase. The design of the expert evaluation process can be tailored to specific policy questions or contexts. In this case study, the compulsory greening measures were intended to be applied uniformly across Europe, so a crude assessment of benefit to wildlife across species and scales was appropriate, although our evaluation only applies to northern Europe. A synopsis of evidence for farmland wildlife conservation that also covered southern Europe, or the entire world (encompassing similar bioclimatic conditions outside Europe), would have been more appropriate to the policy, but was not available at this time. It would be straightforward to ask different questions of the expert group, to evaluate, for example, effects on particular taxa, in specific regions or contexts, or to evaluate the relative magnitude of wildlife benefits or disbenefits, from the same evidence base. In this way, existing synopses that are global in scope (such as Dicks et al . ; Williams et al . ) can be used to address policy questions relating to taxa‐specific conservation anywhere in the world. The publication of synopses allows new evidence to be contributed as it arises and the whole reevaluated cumulatively, without having to start again from scratch when a new assessment is required. Coverage of all possible interventions or policy options is necessary from the outset to make this process efficient. Developing a complete list is an important early stage in the synopsis writing process (see Supporting Information). It can also be achieved through collaborative “solutions scanning” exercises involving diverse stakeholders, as demonstrated for marine conservation by Jacquet et al . ( ). It is important to revisit this step, perhaps every few years, to allow new policy ideas or interventions to be identified and added. Since interventions already tested in the literature are added during the writing of a synopsis, it is likely that newly suggested interventions will only be covered by recent literature. The third advantage of the process is that it identifies areas of uncertainty and knowledge gaps, and clarifies whether uncertainty is within individual studies, from differences between studies or from the scale of existing evidence. Where uncertainty derives from study differences, the message of research evidence can easily be misrepresented through selective presentation of a subset of relevant evidence (Sarewitz ). Thorough, consistent, and openly accessible synopses of evidence should make selective presentation obvious. In this case, the existing evidence is strongly biased toward a small number of northern European countries where the effects of agricultural interventions have been intensively studied. For the 20 interventions discussed here, studies came from 19 countries altogether, but for each individual intervention, an average of 76% (ranging from 33% to 100%) of the studies providing evidence came from three countries—the UK, Germany, and Switzerland. This bias is made very apparent by the process because the study locations are clearly stated in summary paragraphs and key messages. The expert evaluation exercise should incorporate such biases into certainty scores by generating lower than 100% certainty when the evidence is only partially relevant, due, for example, to its location. In the evaluation reported here, the maximum certainty score was 77%, for the effectiveness of skylark plots in northern Europe (Table ). This intervention had been tested in three northern European countries—UK, Switzerland, and Denmark. The evidence told a consistent story for the target species skylark Alauda arvensis and there was some evidence of benefits for other nontarget species groups such as plants and invertebrates. An expert evaluation of the same evidence for a country or region not biogeographically represented by these studies would be expected to produce lower certainty values because the experts would interpret the evidence as not entirely relevant. It is possible that the final scores from the expert evaluation would have been different if a different set of 10 people were involved. Even with our private scoring process, scores may be influenced to an unknown extent by prior knowledge and opinions, combined with psychological and social factors operating during the discussion. The influence of group composition on recommendations made by groups developing evidence‐based clinical guidelines for medical practice is a subject of active research (Hutchings & Raine ; Gardner et al . ; Hopthrow et al . ). Similar research on the processes by which guidance or evaluations are developed from evidence to support decisions in environmental policy is greatly needed. Current best practice for developing clinical practice guidelines is that groups should be multidisciplinary, with key stakeholders represented (Eccles et al . ). In the expert group used here, the key stakeholders were from organizations spanning the boundary between science and policy (called “boundary organizations” by Cook et al . ), rather than farmer groups or policy makers themselves. It would be enlightening to repeat the process with different stakeholder groups represented. This model of collating evidence in synopses to feed into decisions is the norm in medical practice (Dicenso et al . ). We show how it is equally applicable in environmental policy and advocate its application by policy makers, across all policy areas that aim to be informed by evidence. The current assessment suggests that Ecological Focus Areas should be retained as the main compulsory greening measure for biodiversity objectives of the CAP, but with detailed prescriptions of management set at national, or preferably regional levels. For northern and temperate Europe, these prescriptions should include uncultivated margins, grass buffer strips, and land left fallow, as well as interventions marked with an asterisk in Table . However, a final policy conclusion should incorporate further synopses, considering correlative evidence and the wider ecosystem service objectives of greening the CAP. Acknowledgments We thank Arcadia, the UK Research Councils’ Rural Economy and Land Use Programme (Project: RES‐240‐25‐006), the Natural Environment Research Council (Project: NE/K001191/1), the Swedish Research Council Formas, Will Peach, Clunie Keenleyside, Alison McKnight, Juliana Dänhardt, Katy James, Annelie Jönsson, David Showler, Susan Turpie, David Williams, Joscelyne Ashpole, and Tara Proud. We also thank three anonymous reviewers for helpful comments that improved the article. http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Conservation Letters Wiley

A Transparent Process for “Evidence‐Informed” Policy Making

Download PDF
7 pages

Loading next page...
 
/lp/wiley/a-transparent-process-for-evidence-informed-policy-making-OwaEKIlDu4

References (46)

Publisher
Wiley
Copyright
"Copyright © 2014 Wiley Periodicals, Inc."
eISSN
1755-263X
DOI
10.1111/conl.12046
Publisher site
See Article on Publisher Site

Abstract

Introduction There has been much discussion about how to incorporate research evidence into policy, especially in public health (Black ; Bowen & Zwi ; Macintyre ; Rutten ; Sutherland et al . ). Major political institutions, such as the European Commission (EC), claim to facilitate use of the best available scientific knowledge in decision‐making (European Commission ). Despite this, it is often difficult to decipher which evidence contributed to a policy, or how it contributed. For example, Sharman & Holmes ( , p. 318) doubted “that the breadth of scientific data available on the potential [greenhouse gas] emissions from biofuels production was adequately factored into the final policy decision” by the EC to set a 10% renewable content target for road transport fuels by 2020. We propose a consistent and cumulative approach to incorporating research evidence into policy, with ongoing systematic collation of evidence relating to all possible policy options, combined with expert evaluation. Such an approach now routinely informs medical practice, following a hierarchical model, in which studies are evaluated for evidence quality and synthesized in systematic reviews, then condensed into “synopses” of digestible, practical information to support clinical decisions (Sackett et al . ; Dicenso et al . ; Moja & Banzi ). Synopses are a vital element of the evidence‐based framework, because they compile evidence across a range of different possible options that a practitioner or decision‐maker has to decide between. In other areas of applied science, including environmental science, the systematic review has become an established tool for incorporating science into policy (Keene & Pullin ), but the synopsis layer is only just emerging. Global synopses of evidence relevant to conservation have been published for wild bees (Dicks et al . ) and birds (Williams et al . ). As part of the Conservation Evidence project (Conservation Evidence ), synopses are currently being prepared for bats, carnivores, amphibians, reptiles, forests, and control of UK freshwater invasive species. Here, we show how a synopsis can inform policy, using the ongoing restructuring of the European Union's (EU) Common Agricultural Policy (CAP) as an example. The process demonstrates that elements of the proposed reform are not supported by experimental evidence. It identifies areas of uncertainty and presents alternative options supported by evidence. In areas such as environment or public health, policy‐relevant studies are often scant and too heterogeneous to allow formal meta‐analysis. Furthermore, results often differ depending on the contexts in which interventions were applied and the responses measured. Policy makers want to know the likelihood of an intervention working under specific sets of conditions. Although highly valuable, a generic effect size across all studies, as presented by systematic reviews, may not always provide sufficient detail. However, once a narrative synopsis is collated, the body of evidence relating to a given problem can be assessed or evaluated rapidly using expert judgment, as shown here. This second stage of assessing evidence can be completed sufficiently quickly to respond to changing political or economic conditions and provides a clear audit trail, allowing scrutiny of the research evidence supporting policy decisions. European agriculture: a case study The CAP is the EU's biggest single expenditure, accounting for over €57 billion annually in 2010 and 2011 (European Commission ), over 40% of the total EU budget. It has two separate “pillars.” Pillar I, comprising 75% of spending, entails annual direct payments to farmers and market measures such as purchasing surplus produce. Pillar II entails multiannual rural development and voluntary environmental measures, including agrienvironment schemes designed and implemented at member state level. At present, the CAP is going through a process of reform. A new structure is being developed, due to be implemented at the start of 2015. A crucial aspect of the reform proposals (European Commission ; Defra ) is that 30% of direct payments within Pillar I should become conditional upon “compulsory greening measures” that address climate and environmental policy goals. This is intended to ensure “that all EU farmers […] deliver environmental and climate benefits as part of their everyday activities” (European Commission ). Three “compulsory greening measures” have been identified, each with ecosystem and habitat maintenance as one of its objectives (European Commission , ). These measures, incorporating the amendments voted for by the European Parliament in March 2013 (European Parliament News ), are as follows: Crop diversification (ensuring at least three crops annually) on farms with an arable area >30 ha. Maintenance of the proportion of total agricultural land area managed as permanent grassland and pasture. “Ecological Focus Areas” on 3% of arable land (possibly rising to 7% by 2018), which may comprise: hedgerows, ditches, stone walls, in field trees, ponds, land planted with nitrogen fixing crops, terraces, buffer strips, landscape features, and afforested areas or land left fallow. Independently of the CAP reform process, we compiled a synopsis of experimental evidence for the effects of 116 interventions to protect wildlife in farmland in northern temperate Europe (Dicks et al . ). The process used systematic mapping (Randall & James ), incorporating existing systematic reviews (see Supporting Information for details of the review methods). Evidence was collected from all European countries west of Russia, but not those south of France, Switzerland Austria, Hungary, and Romania. It excluded correlative studies, including only studies that experimentally tested interventions. We recognize that this excludes important evidence in this specific context, but the highest quality of quantitative evidence was considered (as described by Pullin & Knight ) and the process remains valid as a proof‐of‐concept. The synopsis did not cover evidence on nonwildlife aspects of ecosystems, such as water, cultural landscapes, or carbon storage. For 85 of the 116 interventions, we evaluated the evidence based on the combined judgment of 10 experts, using a method of formal consensus development involving iterative scoring and discussion, which took just 3 days (see Supporting Information). The 10 experts were selected to represent the important areas of knowledge required for this exercise. Their combined expertise covered farmland ecology (birds, amphibians, plants, mammals, and insects) and agricultural policy. Three group members came from policy‐ or conservation‐oriented consultancies or charities directly involved in interpreting evidence on farming and conservation for use in policy. This method of expert selection, called “purposive sampling,” is often used to form panels for expert judgment (Hasson et al . ). We adhered to the optimum group size of between 8 and 10 recommended for groups interpreting medical evidence into clinical practice guidelines (Eccles et al . ). This expert evaluation allowed assessment of the evidence for wildlife benefits in northern Europe from the potential compulsory greening measures. We evaluated the certainty of the available evidence based on the question “How much do we understand the extent to which this intervention benefits wildlife on farmland?” and compiled responses of the 10 experts to the question “Based on the evidence presented, does this intervention benefit wildlife?” We did not evaluate the relative magnitude of the effects of these interventions. Twelve interventions corresponded directly or indirectly to the proposed greening measures (Table ). Our synopsis found little experimental evidence to support benefits for wildlife arising from crop diversification, or the overarching greening measure of Ecological Focus Areas. It did find that some of the proposed management options for Ecological Focus Areas could offer real benefits. Three such interventions are supported by good evidence of a benefit to wildlife (67%–69% certainty of knowledge, 9–10/10 experts considered the evidence demonstrates a benefit to wildlife). However, current proposals do not indicate how farmers would be influenced to undertake these particular options, or how management quality could be assured. Expert evaluation of evidence (Dicks et al . ) for interventions relevant to proposed compulsory greening measures in European CAP reform. There is no expert evaluation (*) of interventions for which we captured no evidence, as there were no studies to evaluate Compulsory greening measure Relevant intervention for which compiled evidence evaluated by experts No. of studies Certainty of knowledge (% known) Proportion experts stating “yes this benefits wildlife” Crop diversification Increase crop diversity 4 9 0.0 Ecological Focus Areas Increase the proportion of seminatural habitat in the farmed landscape 5 22 0.2 Specific habitat types suggested for Ecological Focus Areas Hedges Manage hedgerows to benefit wildlife 20 50 0.7 Ditches Manage ditches to benefit wildlife 11 44 0.4 Stone walls Restore or maintain dry stone walls 0 * * In field trees Protect in‐field trees 0 * * Plant in‐field trees 0 * * Ponds Create scrapes and pools 6 30 1.0 Buffer strips Plant grass buffer strips/margins 69 69 0.9 Create uncultivated margins around intensive arable or pasture fields 49 67 1.0 Provide buffer strips alongside water courses 7 17 0.1 Land left fallow Provide (or retain) set‐aside areas in farmland 54 68 0.9 This assessment is similar to comments made about the proposed greening measures by Poláková et al . ( , p.159) in a report financed by the European Commission to look at how CAP can contribute to Europe's biodiversity goals. Poláková et al . ( ) say “Amongst the [Pillar 1] proposals, the measure that perhaps has the most potential to deliver additional environmental benefit is the ‘ecological focus area’… However, the actual magnitude of the benefits will depend on the precise requirements under this measure, which have not yet been determined.” A further eight interventions (Table ) not currently suggested as compulsory greening measures were evaluated as having a high certainty of knowledge (>60%), with unanimous agreement among our experts of a wildlife benefit. Three seem potentially appropriate as specific habitat types within Ecological Focus Areas. Two have potential cobenefits for climate change mitigation in agriculture (UK Committee on Climate Change ). Interventions to benefit wildlife on farmland not included in compulsory greening measures with certainty of knowledge >60% and unanimous expert agreement that they provide benefit to wildlife. One intervention included in Table (create uncultivated margins) also falls into this category. The evidence used is summarized in Dicks et al . ( ) Intervention Number of studies Certainty of knowledge Proportion experts stating “yes this benefits wildlife” Create skylark plots 11 77 1.0 Restore/create species‐rich semi‐natural grassland * 71 74 1.0 Mowing techniques to reduce bird/mammal mortality 8 71 1.0 Reduce fertilizer, pesticide or herbicide use generally † 47 69 1.0 Plant nectar flower mixture/wildflower strips * 105 67 1.0 Use organic rather than mineral fertilizers † 19 66 1.0 Plant wild bird seed or cover mixture * 49 63 1.0 Leave cultivated, uncropped margins or plots 20 63 1.0 *Interventions that could be incorporated into CAP Pillar I, as optional habitat types within Ecological Focus Areas. †Interventions recommended as measures to mitigate climate change, accounted for in calculations of agricultural mitigation potential by the UK Committee on Climate Change ( ). We did not identify an intervention exactly matching the greening requirement to retain permanent grassland. “Maintain species‐rich, seminatural grassland” was listed. This intervention involves specific management measures such as traditional cutting or grazing regimes and avoidance of fertilizer inputs. These are unlikely to be accomplished by the proposed compulsory greening measure, which would simply prevent conversion to cropland or temporary (rotated) grassland. There is good evidence that maintaining species‐rich grassland would provide wildlife benefits. Our synopsis found 22 studies on this intervention. The expert group classed it among interventions for which the relevant literature was not adequately covered by our reviewing method, because the initial literature search was focused on farm management rather than seminatural habitat management. This meant it was excluded from further evaluation. Even so, based on the evidence presented, it was given a score of 57% for certainty, and 8/10 experts answered that it would benefit wildlife. But this evidence would only apply to policy measures that specified types of management appropriate for seminatural species‐rich grassland, as some existing agrienvironment schemes do. Evidence was also compiled for converting existing arable land back to permanent grassland, but this was not considered equivalent because existing permanent grassland is expected to have a higher biodiversity value than reverted land for many years after reversion. One intervention in Table (“provide buffer strips along water courses”) lacks strong evidence for a wildlife benefit, but may be effective at improving water quality. The second greening measure “Retain all permanent grassland” could also benefit climate regulation (through soil carbon storage) and cultural landscapes. Given the shift in environmental policy toward ecosystem services in Europe (European Commission ), and the opportunities for using the CAP to support ecosystem services more broadly (Plieninger et al . ), there is an urgent need to synthesize evidence for the effects of land management options on all ecosystem services, rather than just wildlife. Advantages of combining evidence synopsis with expert evaluation The process of creating a narrative evidence synopsis structured around interventions, combined with expert judgment to evaluate the evidence, facilitates evidence‐informed decision‐making. While science is one of several drivers or value sets that operate in policy making (Sharman & Holmes ), it seems imperative to move toward a transparent means of incorporating science into decisions where and when required. In our example, we compiled only direct, experimental evidence where interventions were implemented and their effects monitored (see Supporting Information). This follows the approach used for presenting evidence to doctors at the point of care in medicine. Predictive modeling and correlative studies were excluded, as they are in medicine. For environmental policy, these study types can provide crucial evidence for decisions when direct experimental evidence is sparse, as is always likely at large spatial scales because of the cost and logistical difficulty of performing landscape‐scale experiments. In the longer term, synopses should be developed that cover all types of evidence relevant to particular policy areas, including, for example, predictive modeling, correlative evidence, economic evidence, and qualitative research. Kept separately, different types of evidence could then be weighted in expert evaluations for given policy questions, following the kind of evidence typology approach suggested by Petticrew & Roberts ( ). For environmental questions, experimental evidence would usually be given the highest weight at a given spatial scale, because it shows what actually happens if you intervene, rather than what you expect to happen based on observed correlation. These are not always the same. For example, many studies have shown that biodiversity in farmland is positively correlated with the proportion of seminatural habitat (Tscharntke et al . ). Yet, the experimental evidence we found for increasing the proportion of seminatural habitat shows mixed effects on biodiversity, rather than a consistently positive effect. The evidence comprises studies monitoring landscape‐level effects of the Swiss Ecological Compensation Areas (ECAs) scheme, which obliged farmers to manage at least 7% of their agricultural land as ECAs from 1998 onwards. This scheme is very similar to the Ecological Focus Areas originally proposed in the CAP compulsory greening measures. The reason for the lack of clear benefits in Switzerland could be because at larger scales, mobile organisms such as birds redistribute in the landscape rather than showing genuine population change in response to agrienvironment schemes or because the implementation of ECAs on Swiss farms was not ideal. Both possibilities deserve further research. The process demonstrated here has three advantages over current efforts to incorporate evidence into policy decisions. First, it provides an evidence audit trail, allowing both contemporaneous and retrospective scrutiny of policy decisions. The individual items of evidence that feed into summary statements for each intervention are easy to locate. The underlying review process is explicit about the search protocol and criteria for including or excluding papers. This is important yet often not the case in policy‐related documents. For example, neither the European Commission's own impact assessment (European Commission ) nor its commissioned research (Poláková et al . ) on the biodiversity effects of the CAP reform proposals contain clear statements of how, or with what criteria, evidence was gathered. Similarly, reviews underlying large‐scale scientific assessments such as those on climate change (Pachauri & Reisinger ), or ecosystems (Hassan et al . ), are carried out by working groups of scientists who cover the literature in their areas of expertise using unspecified methods of search and appraisal. These assessment processes are given authority by the number and breadth of expert reviewers who subsequently comment on the text. It is difficult for people outside the process to scrutinize the way evidence was used, to check for bias in interpretation of the literature or to repeat the process independently. Second, previously synthesized evidence allows rapid responses to proposed policy change. By reviewing evidence for all potential interventions, the slow ongoing process of review is separated from a responsive policy development phase. The design of the expert evaluation process can be tailored to specific policy questions or contexts. In this case study, the compulsory greening measures were intended to be applied uniformly across Europe, so a crude assessment of benefit to wildlife across species and scales was appropriate, although our evaluation only applies to northern Europe. A synopsis of evidence for farmland wildlife conservation that also covered southern Europe, or the entire world (encompassing similar bioclimatic conditions outside Europe), would have been more appropriate to the policy, but was not available at this time. It would be straightforward to ask different questions of the expert group, to evaluate, for example, effects on particular taxa, in specific regions or contexts, or to evaluate the relative magnitude of wildlife benefits or disbenefits, from the same evidence base. In this way, existing synopses that are global in scope (such as Dicks et al . ; Williams et al . ) can be used to address policy questions relating to taxa‐specific conservation anywhere in the world. The publication of synopses allows new evidence to be contributed as it arises and the whole reevaluated cumulatively, without having to start again from scratch when a new assessment is required. Coverage of all possible interventions or policy options is necessary from the outset to make this process efficient. Developing a complete list is an important early stage in the synopsis writing process (see Supporting Information). It can also be achieved through collaborative “solutions scanning” exercises involving diverse stakeholders, as demonstrated for marine conservation by Jacquet et al . ( ). It is important to revisit this step, perhaps every few years, to allow new policy ideas or interventions to be identified and added. Since interventions already tested in the literature are added during the writing of a synopsis, it is likely that newly suggested interventions will only be covered by recent literature. The third advantage of the process is that it identifies areas of uncertainty and knowledge gaps, and clarifies whether uncertainty is within individual studies, from differences between studies or from the scale of existing evidence. Where uncertainty derives from study differences, the message of research evidence can easily be misrepresented through selective presentation of a subset of relevant evidence (Sarewitz ). Thorough, consistent, and openly accessible synopses of evidence should make selective presentation obvious. In this case, the existing evidence is strongly biased toward a small number of northern European countries where the effects of agricultural interventions have been intensively studied. For the 20 interventions discussed here, studies came from 19 countries altogether, but for each individual intervention, an average of 76% (ranging from 33% to 100%) of the studies providing evidence came from three countries—the UK, Germany, and Switzerland. This bias is made very apparent by the process because the study locations are clearly stated in summary paragraphs and key messages. The expert evaluation exercise should incorporate such biases into certainty scores by generating lower than 100% certainty when the evidence is only partially relevant, due, for example, to its location. In the evaluation reported here, the maximum certainty score was 77%, for the effectiveness of skylark plots in northern Europe (Table ). This intervention had been tested in three northern European countries—UK, Switzerland, and Denmark. The evidence told a consistent story for the target species skylark Alauda arvensis and there was some evidence of benefits for other nontarget species groups such as plants and invertebrates. An expert evaluation of the same evidence for a country or region not biogeographically represented by these studies would be expected to produce lower certainty values because the experts would interpret the evidence as not entirely relevant. It is possible that the final scores from the expert evaluation would have been different if a different set of 10 people were involved. Even with our private scoring process, scores may be influenced to an unknown extent by prior knowledge and opinions, combined with psychological and social factors operating during the discussion. The influence of group composition on recommendations made by groups developing evidence‐based clinical guidelines for medical practice is a subject of active research (Hutchings & Raine ; Gardner et al . ; Hopthrow et al . ). Similar research on the processes by which guidance or evaluations are developed from evidence to support decisions in environmental policy is greatly needed. Current best practice for developing clinical practice guidelines is that groups should be multidisciplinary, with key stakeholders represented (Eccles et al . ). In the expert group used here, the key stakeholders were from organizations spanning the boundary between science and policy (called “boundary organizations” by Cook et al . ), rather than farmer groups or policy makers themselves. It would be enlightening to repeat the process with different stakeholder groups represented. This model of collating evidence in synopses to feed into decisions is the norm in medical practice (Dicenso et al . ). We show how it is equally applicable in environmental policy and advocate its application by policy makers, across all policy areas that aim to be informed by evidence. The current assessment suggests that Ecological Focus Areas should be retained as the main compulsory greening measure for biodiversity objectives of the CAP, but with detailed prescriptions of management set at national, or preferably regional levels. For northern and temperate Europe, these prescriptions should include uncultivated margins, grass buffer strips, and land left fallow, as well as interventions marked with an asterisk in Table . However, a final policy conclusion should incorporate further synopses, considering correlative evidence and the wider ecosystem service objectives of greening the CAP. Acknowledgments We thank Arcadia, the UK Research Councils’ Rural Economy and Land Use Programme (Project: RES‐240‐25‐006), the Natural Environment Research Council (Project: NE/K001191/1), the Swedish Research Council Formas, Will Peach, Clunie Keenleyside, Alison McKnight, Juliana Dänhardt, Katy James, Annelie Jönsson, David Showler, Susan Turpie, David Williams, Joscelyne Ashpole, and Tara Proud. We also thank three anonymous reviewers for helpful comments that improved the article.

Journal

Conservation Letters – Wiley

Published: Mar 1, 2014

Keywords: ; ; ; ;

There are no references for this article.