Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

The effects of communicating uncertainty on public trust in facts and numbers

The effects of communicating uncertainty on public trust in facts and numbers The effects of communicating uncertainty on public trust in facts and numbers a,b,c,1 a,b,d,1 a,b Anne Marthe van der Bles , Sander van der Linden , Alexandra L. J. Freeman , a,b and David J. Spiegelhalter  a b Winton Centre for Risk and Evidence Communication, University of Cambridge, Cambridge CB3 0WA, United Kingdom; Department of Pure Mathematics and Mathematical Statistics, University of Cambridge, Cambridge CB3 0WA, United Kingdom; Department of Social Psychology, University of Groningen, 19712 TS Groningen, The Netherlands; and Cambridge Social Decision-Making Lab, Department of Psychology, University of Cambridge, Cambridge CB2 3RQ, United Kingdom Edited by Arild Underdal, University of Oslo, Oslo, Norway, and approved February 20, 2020 (received for review August 7, 2019) Uncertainty is inherent to our knowledge about the state of the the general sense of honesty evoked [by uncertainty] .. . this did not world yet often not communicated alongside scientific facts and seem to offset concerns about the agency’s competence” (p. 491). In numbers. In the “posttruth” era where facts are increasingly con- fact, partly for these reasons, Fischhoff (1) notes that scientists may tested, a common assumption is that communicating uncertainty be reluctant to discuss the uncertainties of their work. This hesita- will reduce public trust. However, a lack of systematic research tion spans across domains: For example, journalists find it difficult makes it difficult to evaluate such claims. We conducted five exper- to communicate scientific uncertainty and regularly choose to ig- iments—including one preregistered replication with a national nore it altogether (10, 11). Physicians are reluctant to communicate sample and one field experiment on the BBC News website (total uncertainty about evidence to patients (12), fearing that the com- n = 5,780)—to examine whether communicating epistemic uncer- plexity of uncertainty may overwhelm and confuse patients (13, 14). tainty about facts across different topics (e.g., global warming, im- Osman et al. (15) even go as far as to argue explicitly that “the drive migration), formats (verbal vs. numeric), and magnitudes (high vs. to increase transparency on uncertainty of the scientific process low) influences public trust. Results show that whereas people do specifically does more harm than good” (p. 131). perceive greater uncertainty when it is communicated, we observed At the same time, many organizations that produce and only a small decrease in trust in numbers and trustworthiness of the communicate evidence to the public, such as the European Food source, and mostly for verbal uncertainty communication. These Safety Authority, have committed themselves to openness and results could help reassure all communicators of facts and science transparency about their (scientific) work, which includes com- that they can be more open and transparent about the limits of municating uncertainties around evidence (16–19). These at- human knowledge. tempts have not gone without criticism and discussion about the potential impacts on public opinion (15, 20). What exactly do we communication uncertainty trust posttruth contested | | | | know about the effects of communicating uncertainty around facts, numbers, and science to the public? ur knowledge is inherently uncertain. The process by which Owe gather information about the state of the world is char- Significance acterized by assumptions, limitations, extrapolations, and gener- alizations, which brings imprecisions and uncertainties to the facts, Does openly communicating uncertainty around facts and numbers, and scientific hypotheses that express our understanding numbers necessarily undermine audiences’ trust in the facts, or of the world around us. However, despite the fact that scientists the communicators? Despite concerns among scientists, ex- and other producers of knowledge are usually well-aware of the perts, and journalists, this has not been studied extensively. In uncertainties around their findings, these are often not commu- four experiments and one field experiment on the BBC News nicated clearly to the public and other key stakeholders (1). This website, words and numerical ranges were used to communi- lack of transparency could potentially compromise important de- cate uncertainty in news article-like texts. The texts included cisions people make based on scientific or statistical evidence, contested topics such as climate change and immigration sta- from personal medical decisions to government policies. tistics. While people’s prior beliefs about topics influenced their Recent societal developments do not seem to encourage more trust in the facts, they did not influence how people responded openness about uncertainty: It has been suggested that we are to the uncertainty being communicated. Communicating un- living in a “posttruth” era in which facts, evidence, and experts certainty numerically only exerted a minor effect on trust. are deeply mistrusted (2). Cross-national survey studies suggest Knowing this should allow academics and science communi- that in many countries, trust in institutions and governments is in cators to be more transparent about the limits of human decline (3–5). Although the underlying causes of changes in trust knowledge. are likely to be complex and varied, it has been suggested that Author contributions: A.M.v.d.B., S.v.d.L., A.L.J.F., and D.J.S. designed research; one way to potentially repair and restore public trust in science, A.M.v.d.B., S.v.d.L., and A.L.J.F. performed research; A.M.v.d.B. and S.v.d.L. analyzed data; evidence, and official statistics is to be more open and trans- and A.M.v.d.B., S.v.d.L., A.L.J.F., and D.J.S. wrote the paper. parent about scientific uncertainty (2). However, it is often as- The authors declare no competing interest. sumed that communicating uncertainty transparently will invite This article is a PNAS Direct Submission. criticism, can signal incompetence, or even decrease public trust in This open access article is distributed under Creative Commons Attribution License 4.0 science (1, 6–8). In fact, as summarized by the National Acade- (CC BY). mies of Sciences, Engineering, and Medicine report on science Data deposition: The datasets collected and analyzed in this paper are available on the communication, “as a rule, people dislike uncertainty [...] people Open Science Framework (https://doi.org/10.17605/OSF.IO/MT6S7). may attribute uncertainty to poor science [.. . and] in some cases, 1 To whom correspondence may be addressed. Email: a.m.van.der.bles@rug.nl or sander. communicating uncertainty can diminish perceived scientific au- vanderlinden@psychol.cam.ac.uk. thority” (ref. 7, pp. 27–28). For example, research by Johnson and This article contains supporting information online at https://www.pnas.org/lookup/suppl/ doi:10.1073/pnas.1913678117/-/DCSupplemental. Slovic (9) found that for some respondents, uncertainty “evoked doubt about agency trustworthiness” (p. 490), and that “despite First published March 23, 2020. 7672–7683 | PNAS | April 7, 2020 | vol. 117 | no. 14 www.pnas.org/cgi/doi/10.1073/pnas.1913678117 Prior research has distinguished between two kinds of un- statistics. From a purely statistical point of view, variability certainty: epistemic uncertainty about the past and present state around a central estimate signals that the estimate is more un- of the world that arises because of what we do not know but certain; thus, when uncertainty is taken into account, a logical could know in theory (e.g., uncertainty due to limitations of the conclusion would be that the central estimate is less informative sample or methodology) vs. uncertainty about the future that and reliable. However, communicating uncertainty when it exists arises because we cannot know (i.e., randomness, chance; we can also be seen as part of an organization’s goal to be open and cannot know for certain what will happen tomorrow) (21). Al- transparent, which could foster perceived trustworthiness (40). though uncertainty about the future is a widely acknowledged Given limited and mixed prior findings, we set out to systemat- aspect of forecasts and predictions, epistemic uncertainty about ically examine how communicating uncertainty about a range of the past and present is equally important yet often overlooked in facts influences public trust in both numbers and their sources in communication. It is the uncertainty around the decrease in four large and diverse online experiments (combined n = 4,249) unemployment in the United Kingdom (e.g., estimated at a de- and one field experiment on the BBC News website (n = 1,531). crease of 119,000 people from January to March 2019 compared We were particularly interested in comparing the effects of to a year earlier, with a 95% CI of ±96,000) (22); or uncertainty communicating both verbal and numerical uncertainty around a around the number of people who attended US President Trump’s (contested) numeric estimate on perceived uncertainty (cogni- inauguration, which he famously claimed “had the biggest audience tion) and perceived trustworthiness.* in the history of inaugural speeches” (23) (attendance is estimated In experiment 1, 1,122 participants read a short text about one to have been between 300,000 and 600,000) (24). Psychological of three topics, which contained either no uncertainty (just a research suggests that people intuitively distinguish between these point estimate), uncertainty communicated as a numerical range two kinds of uncertainty (25, 26). (in addition to the point estimate), or uncertainty communicated How people react to aleatoric uncertainty about the future has as a verbal statement (also in addition to the point estimate). been relatively well studied. A large literature indicates that The three topics were as follows: the number of unemployed people are generally averse to uncertainty when making decisions people in the United Kingdom, the number of tigers currently about the future—a psychological tendency known as ambiguity left in India, and the increase in the global average surface aversion (27). Moreover, the relatively large body of research on temperature between 1880 and 2010. We selected these topics to the interpretation of verbal expressions of uncertainty such as represent various “types” of numbers (large vs. small) as well as “likely” or “unlikely” shows that there is considerable variability for variation in their level of “contestedness.” That is, we between people in how they interpret these words, creating expected there to be different levels of variation in prior atti- problems for effective communication (28–31). It has therefore tudes toward the topics: Whereas opinions on climate change are been suggested that communicating uncertainty numerically might divided in the United Kingdom (41), we anticipated less division be preferable; yet several studies found that numerical uncertainty in opinion around the conservation of endangered animals. might suffer from its own interpretation issues (32–34). To illus- In the short text about unemployment, for example, partici- trate, recent research has found that motivated cognition can have pants read that recently an official report had been published, an impact on probabilistic reasoning: Prior beliefs about climate which stated that between April and June 2017, the number of change or gun laws in the United States influenced people’sin- unemployed people in the United Kingdom was an estimated terpretation of the distribution underlying ambiguous numerical 1,484,000. Participants then either received no further informa- ranges about these issues (35). tion (control condition), or a numerical range (numerical un- However, much less research has specifically focused on epistemic certainty condition: “minimum 1,413,000 to maximum 1,555,000”), uncertainty. In fact, a recent review concluded that empirical evi- or an equivalent verbal statement (verbal uncertainty condition: dence about the psychological effects—positive or negative—of “The report states that there is some uncertainty around this es- communicating epistemic uncertainty about facts and numbers is timate, it could be somewhat higher or lower”). The source of the limited and scattered with mixed findings (21). For example, numbers was an “official report,” which was not further specified in research by Johnson and Slovic (9, 33, 36) suggests that com- our first experiment in order to avoid source bias. municating uncertainty via numerical ranges signaled honesty After reading the short text, participants first indicated how and competency for some of their participants, but dishonesty the information made them feel on a feeling thermometer, then and incompetency for others. Other studies, for example, in the were asked to recall the number they had just read about, and context of cancer risk or scientific evidence about climate change subsequently were asked a series of questions about how un- and genetically modified organisms, found that communicating certain and reliable they perceived the number to be, and how uncertainty around estimates did not seem to affect people’s trustworthy they thought the writers of the report were. Our scientific beliefs or credibility judgments (8, 37, 38). measures of trust were modeled after prior studies from Johnson Given these mixed findings, the present research program and Slovic (9). To ensure that our findings are systematic and aims to address this gap in the literature and is one of the first to robust, follow-up experiments—including a preregistered repli- examine the effects of communicating epistemic uncertainty cation—varied the magnitude of the uncertainty in the message, about facts, numbers, and evidence on public trust. We examine the style and format of how the uncertainty was communicated, the effects of communicating uncertainty around numbers— as well as the sampling platform. including contested numbers—that are communicated routinely in the media, such as the unemployment rate and migration Results statistics. Specifically, we draw on a recent theoretical review The analyses revealed no meaningful differences between topics (21), which suggests that research on communicating epistemic in how people reacted to the uncertainty communication. For uncertainty should consider its effects on three key conceptual ease of interpretation, we therefore present the results collapsed dimensions, namely, 1) cognition (how people perceive and un- across topics (but see SI Appendix for further details). The results derstand uncertainty), 2) emotion (how people feel about the showed that our manipulation was effective in that participants uncertainty), and 3) trustworthiness (the extent to which people perceived the numbers to be more uncertain when uncertainty trust the information). Following recent work on source credi- was communicated either numerically or verbally (Fig. 1A). An bility in science communication (39), we further consider and conceptually distinguish two key forms of trust: people’s trust in the numbers themselves and people’s trust in the “source” of *Effects on people’s emotions were of secondary interest and are reported in detail in the these numbers, for example, in the organization producing the SI Appendix. van der Bles et al. PNAS | April 7, 2020 | vol. 117 | no. 14 | 7673 PSYCHOLOGICAL AND COGNITIVE SCIENCES 2 analysis of variance (ANOVA) testing the effect of uncertainty communication format [F = 60.96, P < 0.001; η = 0.10]. (2,1119) communication (control vs. numerical vs. verbal) on perceived un- Verbal uncertainty communication reduced people’s trust in certainty of the number showed a significant main effect [F = (2, 1119) numbers compared to the control condition (M = 3.51 vs. 4.52, 138.67, P < 0.001; η = 20]. All reported post hoc paired comparisons M = −1.01, 95% CI [−1.23; −0.79], d = 0.75) and compared to diff used Tukey’s honestly significant difference. People who were pre- numerical uncertainty communication (M = 3.51 vs. 4.31, M = diff sented with uncertainty as a numeric range perceived the numbers to −0.80, 95% CI [−1.03; −0.57], d = 0.60). Importantly, there was be significantly more uncertain compared to people in the control no significant difference in trust in numbers between the nu- merical uncertainty communication and control conditions (M = condition [M = 4.78 vs. 4.14, M = 0.64, 95% confidence interval diff 4.31 vs. 4.52, M = −0.21, 95% CI [−0.43; 0.01], d = 0.17). (CI) [0.40; 0.88], d = 0.45]. Participants who were presented with diff Finally, we asked participants how trustworthy they thought uncertainty as a verbal statement perceived the numbers to be “the writers of the report” were, as a measure of trust in the significantly more uncertain than both those in the numerical (M = source. Again, we found that the verbal uncertainty communi- 5.82 vs. 4.78, M = 1.04, 95% CI [0.80; 1.28], d = 0.79) and the diff cation led to a small significant decrease in people’s trust in the control conditions (M = 5.82 vs. 4.14, M = 1.68, 95% CI [1.44; diff source, whereas the numerical uncertainty communication did 1.92], d = 1.17). not (Fig. 1C). The ANOVA showed a main effect of uncertainty We also asked participants to indicate how reliable and how trustworthy they thought the numbers were; given their correlation communication format [F = 11.03, P < 0.001; η = 0.02]. (2, 1119) (r = 0.88), scores on these two questions were combined to form a Communicating uncertainty verbally reduced participant’s trust measure of “trust in numbers.” The results showed that, although in the source, compared to the control condition (M = 4.19 vs. 4.55, M = −0.36, 95% CI [−0.57; −0.15], d = 0.28) and nu- our verbal phrase of uncertainty communication decreased trust in diff merical uncertainty communication (M = 4.19 vs. 4.58, M = numbers, our numerical uncertainty communication did not diff −0.39, 95% CI [−0.61; −0.17], d = 0.31). Again, there was no (Fig. 1B). The ANOVA revealed a main effect of uncertainty significant decrease for numerical uncertainty communication compared to control (M = 4.58 vs. 4.55, M = 0.03, 95% CI diff [−0.18; 0.24], d = 0.02). Perceived uncertainty Experiment 1 thus showed that while people did perceive uncertainty about numbers both when it was communicated numerically and verbally, only the verbal communication re- duced people’s trust in the numbers and the source. In addition, the results showed no significant effect of uncertainty commu- nication on people’s affect or mood; please see SI Appendix for the full results. Because we found no substantial differences between topics in people’s responses to uncertainty, experiment 2 only used the UK employment number to study whether the magnitude of the uncertainty itself can influence the psycho- 1 logical effects of communicating uncertainty. Control Numerical Verbal Manipulating the Magnitude of Uncertainty. The goal of experi- ment 2 was twofold: first, to replicate the results from experi- Trust in number ment 1 (for unemployment) and second, to examine whether 7 the magnitude of the uncertainty affected people’s trust in numbers and trust in the source. This experiment followed a 1 (control condition: no uncertainty) + 2 (numerical vs. verbal communication) × 3 (lower vs. original vs. higher uncertainty) between-subject design. For numerical uncertainty communi- cation, we presented the original 95% CI, which therefore acted as a replication of experiment 1; lower uncertainty using a 2 range half the size (99.99% CI); or higher uncertainty using a range twice as large (68% CI) as the original CI. For verbal uncertainty communication, wepresented thesamebaseline Control Numerical Verbal phrase as in experiment 1 for the “original” magnitude (“...it could be somewhat higher or lower”); less uncertainty using the Trust in source phrase “slightly higher or lower”; or more uncertainty using the phrase “a lot higher or lower.” The verbal phrases were chosen to mirror the magnitude of the numerical uncertainty. First, we analyzed the results of the “original uncertainty” levels and the control condition in experiment 2: a direct repli- cation of experiment 1. For perceived uncertainty and trust in the number, we replicated the results of the first experiment. The analyses, all reported in detail in the SI Appendix, showed that participants perceived the number to be significantly more un- certain when numerical uncertainty was communicated (compared to control) and when verbal uncertainty was communicated Control Numerical Verbal (compared to both control and numerical uncertainty). Similarly, just as in experiment 1, participants reported less trust in the Fig. 1. The results of experiment 1: Means per condition for perceived number when verbal uncertainty was communicated compared to uncertainty (A), trust in numbers (B), and trust in the source (C). The error both control and when numerical uncertainty was communicated; bars represent 95% CIs around the means, and jitter represents the distri- bution of the underlying data. with no significant decrease in trust for numerical uncertainty 7674 | www.pnas.org/cgi/doi/10.1073/pnas.1913678117 van der Bles et al. Score Mean score Mean score [−0.83; −0.45], d = 0.48). In addition, across formats, post hoc Perceived uncertainty comparisons showed that a lower magnitude of uncertainty led to higher trust in numbers compared to the original range or phrase (M = 4.06 vs. 3.75, M = 0.31, 95% CI [0.03; 0.58], d = 0.23). diff Both comparisons were not significantly different from higher magnitude of uncertainty (Fig. 2B). For trust in the source, communication format (numerical or verbal) appeared to make no difference, but the magnitude of the uncertainty did (Fig. 2C). An ANOVA of format and magnitude Numerical Verbal Numerical Verbal Numerical Verbal showed no main effect of format [F = 0.96, P = 0.33], a (1, 741) Low uncertainty Actual uncertainty High uncertainty significant main effect of magnitude [F = 4.30, P = 0.01; (2, 741) Trust in number η = 0.01], but no significant interaction [F = 0.37, P = 0.69]. (2, 741) Regardless of format, lower magnitudes of uncertainty led to higher levels of trust in the source compared to the original range or phrase (M = 4.34 vs. 4.01, M = 0.33, 95% CI [0.06; 5 diff 0.60], d = 0.26), but neither were significantly different from higher magnitude of uncertainty (M = 4.23, SD = 1.31; Fig. 2C). The results of experiment 2 thus suggest that magnitude, communicated without further context, did not have a strong Numerical Verbal Numerical Verbal Numerical Verbal impact on people’s reactions to uncertainty communication: It did not influence perceptions of uncertainty, and only lower Low uncertainty Actual uncertainty High uncertainty magnitudes of uncertainty were related to slightly higher levels of Trust in source perceived reliability of the number and trustworthiness of the source. Without further context, participants might not have been able to interpret the numerical ranges as being relatively small or large magnitudes of uncertainty, although the same is 4 not true of the verbal conditions. Although preliminary, what we 3 can conclude from these results is that, in the absence of further 2 context, it appears that whether and how uncertainty is com- 1 municated is more important in determining how people re- Numerical Verbal Numerical Verbal Numerical Verbal spond than the magnitude of the uncertainty in question. Low uncertainty Actual uncertainty High uncertainty Varying the Format of Uncertainty Communication. Following these Fig. 2. The results of experiment 2: Means per condition for perceived findings, we set out to systematically test the effects of additional uncertainty (A), trust in numbers (B), and trust in the source (C). The error numeric and verbal uncertainty communication formats in ex- bars represent 95% CIs around the means, and jitter represents the distri- periment 3, and to move toward a more realistic and better con- bution of the underlying data. textualized communication scenario. This experiment had eight conditions, which are presented in Table 1. The choice of formats was influenced by the formats the UK Office for National Sta- (compared to control). However, in contrast to experiment 1 where tistics uses to communicate uncertainty around unemployment we found that verbal uncertainty communication reduced trust in the source, experiment 2 showed no significant effect of uncertainty numbers, which was again the context we used for this experiment for consistency. To improve the ecological validity of the experi- communication: Both numerical and verbal uncertainty communi- ment, the manipulation was written as a traditional news media cated did not decrease people’s trust in the source, compared to article and included two unemployment figures. Uncertainty was control (please see SI Appendix,Fig.S2 A–C). Next, we examined the effects of the magnitude of uncertainty. communicated in the same format around both figures. Results are presented in Fig. 3. The level of perceived un- Somewhat surprisingly, we found that the magnitude of the communicated uncertainty did not affect people’s perceptions of certainty around the numbers differed between formats [Fig. 3A; the uncertainty of the numbers (Fig. 2A). A two-way ANOVA of one-way ANOVA effect of format: F = 14.43, P < 0.001; (7, 1192) format (numeric vs. verbal) and magnitude (lower vs. original vs. η = 0.08]. For all formats in which uncertainty was being com- higher) showed a significant main effect of format [F = municated, except one, participants perceived the numbers to be (1, 741) 67.93, P < 0.001; η = 0.08], but no significant main effect of more uncertain compared to the control condition (post hoc paired comparisons:p values = 0.022 to <0.001; M = 0.53 to diff magnitude [F = 2.92, P = 0.055; η = 0.01] nor a significant (2, 741) 1.10; d values = 0.37 to 0.72). The exception was the condition in interaction [F = 1.86, P = 0.16; η = 0.01]. Regardless of (2, 741) which only the word “estimated” had been added (M = −0.07, diff magnitude, and as in experiment 1, verbal uncertainty communi- P = 1.00). People in this condition did not perceive the numbers to cation led to higher levels of perceived uncertainty than numerical be more uncertain compared to those in the control condition, uncertainty communication (M = 5.50 vs. 4.68, M = 0.83, 95% diff indicating that only using the word “estimated” seems insufficient CI [0.63; 1.03], d = 0.60). to communicate the existence of uncertainty around a number. However, the results did show a small effect of magnitude on People’s trust in numbers similarly differed between formats people’s trust in numbers (Fig. 2B). An ANOVA of format and [Fig. 3B; F = 5.97, P < 0.001; η = 0.03]. Whereas for most (7, 1192) magnitude showed a main effect of format [F = 43.44, P < (1, 741) formats, people’s trust in the numbers was significantly reduced 0.001; η = 0.06], and main effect of magnitude [F = 3.63, (2, 741) compared to control (post hoc paired comparisons: p values = P = 0.03; η = 0.01], but no significant interaction [F = 1.44, (2, 741) 0.035 to 0.011; M = −0.47 to −0.53; d values = 0.35 to 0.37), diff this was not the case when uncertainty was communicated with P = 0.24]. Regardless of magnitude, verbal uncertainty commu- nication decreased trust in numbers compared to numerical the word “estimated” (M = 0.13, 95% CI [−0.33; 0.59], P = diff communication (M = 3.56 vs. 4.20, M = −0.64, 95% CI 0.98), with the implicit uncertainty statement (M = −0.37, diff diff van der Bles et al. PNAS | April 7, 2020 | vol. 117 | no. 14 | 7675 Score Mean score Mean score PSYCHOLOGICAL AND COGNITIVE SCIENCES Table 1. Overview of the conditions and manipulation texts of experiment 3 and 4 Format Experiment 3 Experiment 4 Control “Official figures from the first quarter of 2018 show “Migration figures: EU migration still (no uncertainty) that UK unemployment fell by 116,000 compared adding to UK population. Official figures from with the same period last year. [...]” last year show that there were 101,000 more people coming to the UK from the EU than leaving in 2017. This is the lowest EU net migration figure since 2013, but it means that EU migrants are still adding to the UK population. [...]” Numerical range with ...by 116,000 (range between 17,000 and 215,000)... . . .101,000 (range between 68,000 and point estimate 132,000)... Numerical range ...by between 17,000 and 215,000... without point estimate Numerical point ...by 116,000 (±99,000)... . . .101,000 (±33,000)... estimate ±2SEs Verbal explicit ...by 116,000 compared with the same period last year, .. .101,000 more people coming to the UK from uncertainty statement although there is some uncertainty around this the EU than leaving in 2017. The report states figure: It could be somewhat higher or lower. [...] there is uncertainty around the exact figure— it could be higher or lower. [.. .] Verbal implicit ...by 116,000 compared with the same period last year, uncertainty statement although there is a range around this figure: could be somewhat higher or lower. [...] Verbal uncertainty word ...by an estimated 116,000... . . .around 101,000... Mixed numerical and verbal phrase ...by an estimated 116,000 (±99,000)... 95% CI [−0.83; 0.08], P = 0.20), or with a numerical range with of the civil servants responsible for the statistics, nor for jour- point estimate (M = −0.10, 95% CI [−0.56; 0.36], P = 1.00). nalists who write such articles. diff In this experiment, we assessed trust in the source by asking Uncertainty Around Contested Numbers. Experiments 2 and 3 were people to what extent they thought that the civil servants who were responsible for the unemployment figures were trustworthy. conducted in the context of UK unemployment numbers, which Results are shown in Fig. 3C. We found that different formats are generally considered not highly contested and thus might be did make a small difference to people’s trust in the source [one- less likely to result in changes in trust-related perceptions. We way ANOVA: F = 2.15, P = 0.04; η = 0.01]. However, therefore conducted experiment 4 in the context of UK migration, (7, 1192) that difference was not between the conditions in which un- which is a more contested issue on which public opinion is divided certainty was communicated and the control: Across all formats, (42). Experiment 4 was preregistered on aspredicted.org (https:// trust in the source did not differ significantly from the control aspredicted.org/blind.php?x=d3xu67) and also conducted on a condition (range M = 3.94 to 4.48 vs. M = 4.24, SD = control control national sample of the UK population (Methods). Participants 1.55). The difference was between specific formats: Compared to were first asked about their attitudes toward migration, before people to whom uncertainty was communicated through the word being randomly selected to read one of five versions of a fictitious “estimated,” people to whom uncertainty was communicated in newspaper article about migration statistics, which are presented the numeric +/− format or mixed format (“estimated +/−”)per- in Table 1. ceived the source to be significantly less trustworthy (M = 4.48 vs. The results of experiment 4 showed that, similar to experiment 3.94, M = 0.54, 95% CI [0.02; 1.06], d = 0.40 and M = 4.48 vs. diff 3, participants perceived the number to be more uncertain for all 3.94, M = 0.54, 95% CI [0.03; 1.05], d = 0.39, respectively). diff communication formats compared to the control condition, except To examine the boundary conditions of the effects on trust, we for just using the word “around” before the number [Fig. 4A;one- also asked people to indicate how trustworthy they thought jour- way ANOVA F = 22.11, P < 0.001; η = 0.08]. Post hoc (4, 1045) nalists who write news articles like the ones they had read were paired comparisons showed significant differences between the and how reliable they thought government statistics in general are; control condition vs. communicating uncertainty as a numeric these judgements did not differ significantly for different uncer- range (M = 4.47 vs. 5.27, M = −0.80, 95% CI [−1.22; −0.39], 2 diff tainty communication formats [F = 1.60, P = 0.13; η = 0.01, (7, 1192) d = 0.50), using “+/−” (M = 4.47 vs. 5.24, M = −0.77, 95% CI diff and F = 1.60, P = 0.13; η = 0.01, respectively]. (7, 1192) [−1.19; −0.35], d = 0.50), and as an explicit verbal statement (M = In conclusion, then, the results of experiment 3 showed that 4.47 vs. 5.52, M = −1.06, 95% CI [−1.48; −0.64], d = 0.68). diff whereas participants perceived uncertainty when uncertainty was As Fig. 4B shows, communication formats did affect partici- communicated in most numeric and verbal formats, not all for- pants’ trust in numbers [one-way ANOVA F = 7.29, P < (4, 1044) mats affected people’s trust in the numbers. Communicating 0.001; η = 0.03], but this overall effect was qualified by a sig- uncertainty via a numerical range with point estimate or an im- nificant decrease in trust for the explicit verbal statement when plicit verbal statement did not significantly decrease trust in compared to control (M = 3.28 vs. 3.90, M = −0.62, 95% CI numbers compared to the control condition. Adding the word diff [−1.01; −0.23], d = 0.42). The numerical formats and the verbal “estimated” also did not decrease trust, but this format apparently “around” condition did not significantly reduce trust in numbers failed to communicate uncertainty to people. However, impor- compared to control. tantly, just as in experiment 2, none of the uncertainty commu- Furthermore, we found no effect of uncertainty communi- nication formats decreased trust in the source compared to not cation on participants’ trust in the source, which in this exper- communicating uncertainty: There was no impact of uncertainty communication when we asked people about the trustworthiness iment was assessed as perceived trustworthiness of “the civil 7676 | www.pnas.org/cgi/doi/10.1073/pnas.1913678117 van der Bles et al. Third, consistent with experiment 3, participants’ trust in Perceived uncertainty journalists and in government statistics in general were not sig- nificantly affected by communicating uncertainty in these dif- ferent formats (one-way ANOVAs: F = 0.85, P = 0.50, 6 (4, 1044) 5 and F = 1.73, P = 0.14, respectively). These additional (4, 1045) findings suggest that communicating uncertainty verbally has an impact on the perceived reliability of the number itself and con- clusions based on a number, but does not seem to impact judge- ments about the source(s) of numbers (civil servants or journalists), Control Numerical Numerical Numerical Verbal Verbal Verbal Mixed +/− Range Range Cue Explicit Implicit Numerical/ nor generalize to governmental statistics more broadly. Estimate No Estimate Verbal Given the contested nature of immigration statistics, we also Trust in number explored the extent to which people’s prior attitudes toward immigration affected the results. We split our sample into two groups (based on the median = 4.33): people with negative at- titudes toward immigration (mean, 1.00 to 4.00) and people with positive attitudes (mean, 4.33 to 7.00). Two-way ANOVAs (format × immigration attitude: negative vs. positive) revealed main effects of immigration attitudes on perceived uncertainty, trust in numbers, and trust in the source, but no interaction ef- Control Numerical Numerical Numerical Verbal Verbal Verbal Mixed fects. People with positive attitudes toward immigration perceived +/− Range Range Cue Explicit Implicit Numerical/ less uncertainty around the numbers [F = 6.15, P = 0.01; Estimate No Estimate Verbal (1, 1040) η = 0.01], reported more trust in the numbers [F = 33.39, (1, 1039) Trust in source p P < 0.001; η = 0.03], and more trust in the source [F = (1, 1040) 45.22, P < 0.001; η = 0.04] than people with negative attitudes toward immigration. However, we found no significant interaction effects between attitudes and communication format. To assess the robustness of these results, we also conducted a series of Control Numerical Numerical Numerical Verbal Verbal Verbal Mixed +/− Range Range Cue Explicit Implicit Numerical/ Perceived uncertainty Estimate No Estimate Verbal Fig. 3. The results of experiment 3: Means per condition for perceived uncertainty (A), trust in numbers (B), and trust in the source (C). The error bars represent 95% CIs around the means, and jitter represents the distri- bution of the underlying data. servants responsible for the migration statistics” [F = (4, 1045) 1.19, P = 0.31]. Control Numerical Numerical Verbal Verbal In summary, we found that whereas both numeric formats and +/− Range Explicit Cue the explicit verbal statement did communicate uncertainty Trust in number around the net migration number, only the explicit verbal state- ment decreased perceived reliability of the number, and no 7 format decreased participants’ perceptions of trustworthiness of the source. This pattern of results is broadly consistent with our preregistered hypotheses based on our previous three studies (see SI Appendix for more details). We also asked people to what extent they thought that 1) the conclusions based on the number, 2) the news article they just read, 3) journalists who write news articles such as this one, and Control Numerical Numerical Verbal Verbal 4) government statistics in general were trustworthy. +/− Range Explicit Cue This revealed that format did affect trust in the conclusions Trust in source [one-way ANOVA of format: F = 3.53, P = 0.007; (4, 1045) η = 0.01]. This was due to the explicit verbal statement of un- 7 certainty leading to lower trust in the conclusions than the word “around” (post hoc paired comparisons: M = 3.51 vs. 3.96, M = −0.45, 95% CI [−0.84; −0.05], d = 0.30). All uncertainty diff communication formats did not differ significantly from the control condition (M = 3.90, SD = 1.46). Second, there appeared to be a small effect of format on trust in the news article itself (one-way ANOVA of format: F = (4, 1042) Control Numerical Numerical Verbal Verbal +/− Range Explicit Cue 2.43, P = 0.046; η = 0.01), but post hoc paired comparisons did not show significant differences between formats—it was again Fig. 4. The results of experiment 4: Means per condition for perceived mainly driven by a decrease for the explicit verbal statement uncertainty (A), trust in numbers (B), and trust in the source (C). The error compared to the control condition (M = 3.66 vs. 4.04, M = diff bars represent 95% CIs around the means, and the jitter represents the −0.38, 95% CI [−0.77; 0.01], d = 0.26). distribution of the underlying data. van der Bles et al. PNAS | April 7, 2020 | vol. 117 | no. 14 | 7677 Score Mean score Mean Score Score Mean score Mean score PSYCHOLOGICAL AND COGNITIVE SCIENCES hierarchical linear regressions with a continuous interaction trustworthy it was, and how trustworthy and competent the term. These analyses produced the same findings and are statisticians responsible for producing the figure were, and how reported in the SI Appendix. Overall, how people responded to trustworthy they thought the journalist responsible for producing uncertainty communication was not affected by their prior at- the article was. The results are presented in Fig. 7. titudes toward immigration. The results of this field experiment showed that, in line with the laboratory experiments, people perceived the number to be Internal Metaanalysis. To consolidate all of our main findings and more uncertain when numerical uncertainty had been commu- to shed further light on the psychological effects of communi- nicated, compared to no uncertainty communication in the cating uncertainty, we conducted a random-effects metaanalysis across all four studies for each of our key dependent variables. To ensure that the results were comparable, we only included the formats that were consistently tested across all four experiments. For ease of interpretation, we contrast “no uncertainty” (control condition) vs. “uncertainty” communication, differentiating only between “verbal” (explicit verbal statement) vs. “numeric” (numeric range with point estimate) uncertainty communication as separate subgroups. Results are presented in Fig. 5. Overall, the communication of uncertainty in itself had a large effect on perceived uncertainty (d = 0.65; 95% CI [0.42; 0.87]), with the effect of verbal uncertainty (d = 0.88; 95% CI [0.62; 1.14]) being over twice that of numeric uncertainty (d = 0.43; 95% CI [0.33; 0.52]). Importantly, the communication of un- certainty did lead to a significant overall decrease in perceived reliability of the numbers (d = −0.34; 95% CI [−0.16; −0.53]). Although relatively small and nonsignificant across some of the studies, the weighted effect of providing numeric uncertainty on trust in numbers was also negative and significant (d = −0.15; 95% CI [−0.05; −0.24]). However, much of the overall effect seems to stem from verbal uncertainty, as the negative effect of verbal uncertainty on trust in numbers was much more sub- stantial (d = −0.55; 95% CI [−0.35; −0.74]). Last, although the weighted effect of the communication of uncertainty across studies did also significantly and negatively influence perceived trustworthiness of the source (d = −0.12; 95% CI [−0.03; −0.22]), the size of the effect is similarly small and seems to be driven by verbal uncertainty (d = −0.21; 95% CI [−0.12; −0.31]) rather than numeric uncertainty (d = −0.03; 95% CI [−0.03; 0.06]). Field Experiment on the BBC News Website. Finally, we assessed to what extent our findings would generalize beyond the context of an online laboratory experiment to a real-world setting. We there- fore engaged in a unique experiment on the live BBC News website to test the effects of communicating uncertainty in an online news article about the United Kingdom’s labor market statistics, which are released monthly by the Office for National Statistics. After a pilot experiment using a BBC News article about the UK labor market in September 2019, which is reported in the SI Appendix, we conducted an experiment with an online news ar- ticle about the UK labor market on October 15, 2019. Readers of the live BBC News website were randomly shown one of three versions of the news article (Fig. 6). The first figure mentioned in this article was the unemployment rate, which “.. . unexpectedly rose to 3.9% in the June-to-August period from 3.8%, after the number of people in work unexpectedly fell by 56,000, official figures showed.” Readers were either shown this target figure without any uncertainty mentioned, as is common in all news reporting (including the BBC); with a verbal uncertainty cue (“.. . rose to an estimated 3.9%”), as is sometimes used in BBC News reporting; or with a numeric range and verbal cue [“.. . rose to an estimated 3.9% (between 3.7% and 4.1%)”], which is un- common in news reporting. All other figures mentioned in the article were reported without uncertainty. After the first para- graph of the news article, which contained the figure of interest, readers were invited to take part in a short study about this ar- ticle. As our survey had to be brief, we only included our key measures: After asking participants to rate their current emo- tional state (affect), we asked them how certain or uncertain they Fig. 5. Random-effects metaanalysis. Perceived uncertainty (A), trust in thought the unemployment rate figure in the story was, how numbers (B), and trust in the source (C). 7678 | www.pnas.org/cgi/doi/10.1073/pnas.1913678117 van der Bles et al. trustworthiness of the journalist [F = 0.86, P = 0.42]. (2, 1526) Participants’ judgments of the competence and trustworthiness of the statisticians were highly correlated (r = 0.80, P < 0.001), and on the high end of the scale (M = 5.44, SD = 1.41, and M = 5.28, SD = 1.55, respectively, out of seven); participants’ rating of the trustworthiness of the journalist was slightly lower (M = 4.61, SD = 1.54). These results suggest that communicating uncertainty to the participants of this field study, did not affect their (already positive) views of the trustworthiness and competence of the people involved in producing and reporting unemployment figures. Discussion Centuries of human thinking about uncertainty among many leaders, journalists, scientists, and policymakers boil down to a simple and powerful intuition: “No one likes uncertainty” (1, 6, 7, 27). It is therefore often assumed that communicating un- certainty transparently will decrease public trust in science (1, 7). In this program of research, we set out to investigate whether such claims have any empirical basis. We did this by communicating epistemic uncertainty around basic facts and numbers and by systematically varying 1) the topic, 2) the magnitude of the un- certainty, and 3) the format and context through which uncertainty was communicated. We assessed the effects of uncertainty on relevant outcome measures, including cognition and trust. Perceived uncertainty Fig. 6. Image of the BBC News article that was used in experiment 5 (nu- Control Numerical Verbal merical condition: including a numeric range). Reprinted with permission from BBC News. B Trust in number control condition. An ANOVA showed a significant main effect of uncertainty communication on perceived uncertainty [F = (2, 1526) 4.67, P = 0.01; η = 0.006]. Participants who read the version of the news article with a numeric range around the unemployment rate figure perceived the figure to be more uncertain than people in the control condition (M = 3.56 vs. 3.31, M = 0.25, 95% CI [0.06; diff 0.44], d = 0.19). Participants who read the version of the news ar- ticle with the verbal cue scored in between the numerical and control conditions, not significantly different from either (M = 3.41, Control Numerical Verbal SD = 1.39). This finding suggests that participants did seem to have Trust in source noticed the uncertainty that was communicated. Uncertainty communication, however, did not affect partici- pants’ trust in the number [F = 1.20, P = 0.30], nor trust in (2, 1526) the source, in this case, the statisticians responsible for producing the figures [F = 1.24, P = 0.29]. These findings comple- (2, 1525) ment the results from our laboratory experiments, which showed 4 that a verbal cue such as “estimated” did not seem to commu- nicate uncertainty to people and did not affect their trust in numbers or the source (as found in experiments 3 and 4). In this field experiment, we again found communicating uncertainty as a Control Numerical Verbal numeric range did not affect people’s trust in the source, and it also did not affect trust in the number. Fig. 7. The results of field experiment 5: Means per condition for perceived In addition, the results showed no significant effects of un- uncertainty (A), trust in numbers (B), and trust in the source (C). The error certainty communication on affect [F = 0.44, P = 0.65], (2, 1519) bars represent 95% CIs around the means, and the jitter represents the competence of the source [F = 0.61, P = 0.54], and distribution of the underlying data. (2, 1525) van der Bles et al. PNAS | April 7, 2020 | vol. 117 | no. 14 | 7679 Score Score Score PSYCHOLOGICAL AND COGNITIVE SCIENCES Overall, we found little evidence to suggest that communi- reliability either, which is an important finding in itself and cating numerical uncertainty about measurable facts and num- warrants further research. bers backfires or elicits psychological reactance. Across five high- Accordingly, based on these results, we therefore recommend powered studies and an internal metaanalysis, we show that that the communication of uncertainty around basic facts and people do recognize and perceive uncertainty when communi- numbers in the media is best conveyed through numerical ranges cated around point estimates, both verbally and numerically with a point estimate. This format in particular did not seem to (except when only words such as “estimated” or “about” are used significantly influence (i.e., reduce) perceived trust and reliability to imply uncertainty). In addition, uncertainty did not seem to in either the number or the source of uncertainty. In addition, we influence their affective reaction (SI Appendix), and although the draw attention to the fact that using the word “estimate” or in- provision of uncertainty in general did slightly decrease people’s creasing the magnitude of the confidence interval did not seem trust in and perceived reliability of the numbers, this effect to alter people’s perception of uncertainty, which points to the emerged for explicit verbal uncertainty in particular. need to better contextualize the degree of uncertainty for people. Our research offers an important bridge between producers of Last, it is notable that we find little evidence for the motivated statistics, communicators, and their audiences. For example, cognition of uncertainty (35). For example, even around more statisticians or scientists could argue that because most numeric contested topics, such as global warming and immigration, al- estimates are never completely certain, presenting uncertainty though main effects were observed for people’s prior attitude around the number offers more precise information and should toward the issue, there was no significant interaction with the therefore foster more trustworthiness, not less. However, if a communication of uncertainty. At the very least, this suggests general audience had not considered that there might be any that motivated interpretations of uncertainty do not always oc- uncertainty around a number in the first place (e.g., around cur. At the same time, we must acknowledge several limitations unemployment), then from a purely normative point of view of our program of research. people’s reaction to uncertainty in our studies is not entirely First, we recognize that people are known to struggle with inappropriate: By providing clear variability around estimates, it psychological uncertainty about the future (44, 45), perhaps more is reasonable for people to adjust their level of trust in the so than uncertainty about measurable facts and numbers, an area numbers themselves. In a similar vein, one might argue that it is previously neglected, and thus the focus of the current work. The difficult for people to appraise the trustworthiness of a number context of our research was also limited, culturally, to the United without having access to the methodology through which the Kingdom, and more contested examples for this population (e.g., estimate is derived. However, from a social scientific standpoint, around the United Kingdom’s political exit from the European we recognize that people are frequently exposed to numbers in Union) may have elicited different results. Moreover, while we the news without necessarily having access to additional in- conceptually replicated our results across multiple studies and formation, for example, about the quality of the underlying evi- platforms—including a preregistered national sample—we did not dence (or indirect uncertainty). So how do people actually arrive investigate uncertainty around more emotionally charged topics in at a judgment as to what numbers are reliable and trustworthy in this study, such as uncertainty about personal health outcomes the face of uncertainty? Although we did not set out explicitly to (e.g., cancer), nor manipulated contestedness as an experimental investigate the mechanism by which people adjust their judg- factor. Indeed, there may be other circumstances (not examined ments in response to uncertainty, an exploratory mediation here) where a significant degree of uncertainty could elicit strong analysis on the nationally representative sample (experiment 4) emotional reactions. Finally, we attempted to improve the eco- clearly suggests that the main effect of uncertainty communica- logical and external validity of our manipulations by engaging in a tion (uncertainty vs. no uncertainty) on trustworthiness is fully real-world experiment on the live BBC News website. Although mediated by people’s perception of the uncertainty (see SI Ap- findings corroborated what we observed in controlled laboratory pendix for mediation analyses). In other words, this suggests that settings, the BBC study necessarily relied on a somewhat skewed the more uncertain people perceive the numbers to be, the less and self-selected sample. In addition, although we generally relied reliable and trustworthy they find them. The current results help on large and diverse samples, and our main effects were suffi- inform theoretical predictions about how people might respond ciently powered, we may not have had sufficient power to detect to direct uncertainty about numbers, and we encourage future very small effects in all post hoc comparisons. Sensitivity analyses research to further investigate potential mechanisms as well as showed, however, that given the sample sizes of experiments 3 and how people might respond to indirect uncertainty, such as addi- 4 (and assuming α = 0.05 and power of 0.80), we should have been tional information about the quality of the underlying evidence. able to detect small effects in these studies (f = 0.101, d = 0.20; In sum, prior research has investigated whether the provision and f = 0.107, d = 0.21, respectively). The smallest effects of in- of uncertainty can help signal transparency and honesty on be- terest reported in our paper are broadly beyond those thresholds half of the communicator, or—in contrast—whether communi- (e.g., d = 0.26 to 0.72). cating uncertainty decreases trust and signals incompetence (9, Nonetheless, even considering all of these boundary conditions, 15, 17, 36). By and large, our findings illustrate that the provision our results help inform and challenge strongly held—and often of numerical uncertainty—in particular as a numeric range— nonempirical—assumptions across domains about how the public does not substantially decrease trust in either the numbers or the will react to the communication of uncertainty about basic science, source of the message. Verbal quantifiers of uncertainty, how- facts, and numbers (1, 7). A key challenge to maintaining public ever, do seem to decrease both perceived reliability of the trust in science is for communicators to be honest and transparent numbers as well as the perceived trustworthiness of the source. about the limitations of our current state of knowledge. The high These findings were robust across topics (both contested and degree of consistency in our results, across topics, magnitudes of noncontested), mode of communication, and magnitude of un- uncertainty, and communication formats suggest that people “can certainty. More generally, the strong negative effects of verbal handle the truth.” However, if we want to effectively convey un- uncertainty appear consistent with prior findings that people are certainty about pressing issues, such as rising sea levels, the averse to more ambiguous statements (27, 43). As such, we hy- number of tigers left in India, the state of the economy, or how pothesize that the communication of numerical uncertainty may many people turn out to presidential elections; natural scientists, offer a degree of precision that reduces people’s tendency to statisticians, and social scientists should work together to evaluate view the admission of uncertainty as a sign of incompetence (1, 9, how to best present scientific uncertainty in an open and trans- 36). On the other hand, across all studies, the communication parent manner. As such, our findings can provide valuable guid- of uncertainty never significantly increased perceived trust or ance to scientists, communicators, practitioners, and policymakers 7680 | www.pnas.org/cgi/doi/10.1073/pnas.1913678117 van der Bles et al. the verbal condition (somewhat higher or lower). These numbers are based on alike, who are all united by a common interest in how to effectively reports by the UK Office for National Statistics (49), the International Union for communicate the truth in a so-called posttruth world. Conservation of Nature Red list (50), and the Intergovernmental Panel on Cli- mate Change (51), respectively. Materials and Methods Measures. After reading the text, participants first reported how the in- The survey experiments were completed in a web browser and took ∼12 min formation made them feel on a standard feeling thermometer from 0 = to complete. For experiments 1 to 3, we recruited participants on the plat- negative/unhappy to 10 = positive/happy (results are reported in the SI form Prolific. Prolific has been found to be similar to Amazon Mechanical Appendix). They were then asked to recall what number was reported in the Turk in terms of data quality, and better suited to recruit UK-based partic- text (open question), and whether they remembered any uncertainty being ipants (46, 47). Participants were paid £1.20 for their participation and were implied around this number (yes, no, don’t know, don’t remember). These not allowed to participate in more than one experiment. For experiment 4, questions served as manipulation checks and to increase the salience of the we used Qualtrics Panels to recruit a sample that was nationally represen- target number. The open text responses showed that most participants were tative of the United Kingdom population in terms of gender, age, and re- able to either correctly recall the target number (in experiment 2, where gion in which the participants lived. For experiment 5, a field study, we we coded all responses: 54.4%), or give a sensible estimate of the target collaborated with BBC News and recruited visitors of the BBC News website number (experiment 2: 30.2%), indicating that generally participants un- and app. This survey took ∼2 min to complete. Ethical approval for this re- derstood what we meant by “this number” in the questions that followed. search was granted by the Cambridge Psychology Research Ethics Committee Next, our key dependent variables were assessed: perceived uncertainty of (experiments 1, 2, and 5) and the Department of Psychology Ethics Com- the number (average of 2 items, “To what extent do you think that this mittee (experiments 3 and 4) of the University of Cambridge. All participants number is certain or uncertain?”:1 = very certain to 7 = very uncertain; gave informed consent before participation and received detailed debrief- “How much uncertainty do you think there is about this number?”: 10-point ing information afterward. SI Appendix includes tables with an overview of slider: not at all uncertain to very uncertain; r = 0.63), trust in the number the characteristics of the participants for each experiment (SI Appendix, (modeled after ref. 9; average of 2 items, “To what extent do you think this Table S1) and per condition in each experiment (SI Appendix, Tables S2–S6), number is reliable [trustworthy]?”: 7-point scale from 1 = not at all to 7 = which show that the experimental groups were balanced in terms of par- very reliable [trustworthy]; r = 0.88), and trust in the source (“To what extent ticipants’ age, gender, education level, and numeracy. do you think the writers of this report are trustworthy?”: 7-point scale from 1 = not at all to 7 = very trustworthy). In addition, we also asked people to Experiment 1. how uncertain the number made them feel (10-point slider, 1 = not at all to Sample and design. In experiment 1, we used a between-subjects design to test 10 = very uncertain; results reported in SI Appendix). After these dependent three forms of uncertainty communication (numeric vs. verbal vs. control) about variables, a series of unrelated variables were assessed (for more in- three topics (tigers vs. unemployment vs. climate change). Based on a priori formation, see SI Appendix). The experiment finished with questions about power calculations, which indicated we would need 1,075 people for 90% demographic information and a detailed debrief. power to detect a small (interaction) effect (f = 0.12) when α was set at 0.05, we decided to recruit 1,125 participants (125 per cell of the design; we did this for Experiment 2. experiment 2 and 3 as well). Three of these participants indicated to be below Sample and design. Experiment 2 followed on from experiment 1. Instead of 18 y of age and were excluded from further analyses. The sample thus con- varying the topics, however, we were interested in the effect of different sisted of n = 1,122 people [769 women (68.5%); average age, 37.72; SD, 12.12; magnitudes of uncertainty. This experiment therefore consisted of a control range, 18 to 72]. Compared to the UK population, this sample was relatively condition (no uncertainty communicated) plus a 2 (numeric vs. verbal un- highly educated. Organisation for Economic Co-operation and Development certainty communication) × 3 (lower vs. original vs. higher magnitude of data show that 18.8% of the 24- to 65-y-olds in the United Kingdom attained uncertainty) factorial design; so a total of seven conditions, to which par- primary and middle school education, 35.4% upper secondary education ticipants were randomly allocated. Based on a priori power calculations, (General Certificate of Secondary Education [GCSE] and A-levels), and 45.7% which indicated we would need 752 participants for 90% power to detect a attainted tertiary education (bachelor’s, master’s, PhD, etc.) (48). In our sample, small (f = 0.13) interaction effect between format and magnitude when α 1.6% indicated to have no educational qualifications, 38.5% indicated to have was set at 0.05, we recruited 877 participants from Prolific (∼125 per cell for attained upper secondary education, and 59.6% indicated to have attained the seven-cell design). The sample consisted of 582 women and 292 men tertiary education. On average, political orientation of the sample was slightly (average age, 34.68; SD, 12.02; range, 18 to 80). Similar to experiment 1, leaning toward liberal (M = 3.49, SD = 1.42, on a scale from 1 = very liberal to this sample was relatively highly educated compared to the UK population: 7 = very conservative). 1.1% indicated to have no educational qualifications, 39% indicated to Treatment and procedure. After agreeing to participate, participants were have attained upper secondary education, and 59.8% indicated to have asked several questions about their beliefs related to the three topics: about attained tertiary education. On average, political orientation of the sample the conservation of endangered animals, about the present state of the was slightly liberal (M = 3.40; SD = 1.42). country and economy, and about climate change (for more information on all Treatment and procedure. All participants read the same text as in study 1 about measures, see SI Appendix). After this, participants were randomly allocated unemployment, with either no uncertainty communicated, or uncertainty to be presented with one of nine texts. For example, the text about un- communicated numerically or verbally. The different magnitudes were either employment read as follows: the original magnitude that was communicated in experiment 1, which was a numerical range of 1,413,000 to 1,555,000 (95% CI around the point estimate Recently, an official report came out with new information about the of 1,484,000 unemployed people) or the sentence stating that the number unemployment rate in the United Kingdom. This report stated that be- could be “somewhat higher or lower.” However, in addition, lower un- tween April and June 2017, government statistics showed that an esti- certainty was communicated as a range of minimum 1,448,500 to maximum mated 1,484,000 people in the UK were unemployed. 1,519,500 (a 68% CI around the point estimate, which is a range that is half as Participants in the control condition only read about this central estimate, wide as the original) or through the wording “slightly higher or lower” (verbal without any information about uncertainty. For participants in the numeric condition). Higher uncertainty was communicated as a range of minimum 1,342,000 to maximum 1,626,000 (99.99% CI, which is a range that is twice as uncertainty communication condition, the exact same sentence finished with a wide as the original) or through the wording “a lot higher or lower.” Before numeric range: “.. .unemployed (minimum 1,413,000 to maximum 1,555,000).” reading this text, participants were asked some questions about their beliefs For participants in the verbal uncertainty communication condition, an extra about the state of the country and economy, and afterward, they were asked sentence was added to the text: “The report states that there is some un- the same exact questions as in study 1 (SI Appendix). certainty around this estimate, it could be somewhat higher or lower.” The control text about tigers reported that “an official report stated that in 2015 an estimated 2,226 tigers were left in India.” In the numeric uncertainty commu- Experiment 3. nication condition, a range of minimum 1,945 to maximum 2,491 was added, Sample and design. In experiment 3, we aimed to test various other numeric and in the verbal uncertainty communication condition, the exact same sen- and verbal uncertainty communication formats, still in the context of un- tence was used as in the unemployment condition. The text about climate employment for consistency using a relatively high magnitude of uncertainty. change reported that “an official report stated that between 1880 and 2012, This study had eight conditions, and we recruited n = 1,200 participants from the earth’s average global surface temperature has increased by an estimated Prolific, based on power calculations that indicated this would give us 90% 0.85°C.” In the numeric uncertainty condition, a range of minimum 0.65 to power to detect a small (f = 0.125) effect when α was set at 0.05. The sample maximum 1.06 was added, and once again the exact same sentence was used in consisted of 806 women and 388 men (average age, 36.65; SD, 11.98; range, van der Bles et al. PNAS | April 7, 2020 | vol. 117 | no. 14 | 7681 PSYCHOLOGICAL AND COGNITIVE SCIENCES 18 to 85). Just as in experiments 1 and 2, the sample was relatively highly Measures. Before reading the text, participants answered demographic ques- educated compared to the UK population (no educational qualifications, tions and questions about their beliefs about the state of society and the 1.3%; upper secondary, 34.7%; tertiary education, 63.7%) and on average economy and their attitudes toward migration [with three items from the European Social Survey (53): “Would you say it is generally good for the UK’s slightly leaning toward a liberal political orientation (M = 3.40; SD = 1.38). economy that people come to live here from other countries?”:1 = very bad After answering questions about their beliefs about the state of the country for the economy to 7 = very good for the economy; “Would you say that the and economy, people read a version of the following text, which was designed UK’s cultural life is generally undermined or enriched by people coming to live to read more like a short news article to increase ecological validity: here from other countries?”:1 = cultural life undermined to 7 = cultural life UK unemployment drops enriched; “Is the UK made a worse or a better place to live by people coming to live here from other countries?”:1 = worse place to live to 7 = better place Official figures from the first quarter of 2018 show that UK unemploy- to live; α = 0.91]. After reading the text, participants answered the same ment fell by 116,000 compared with the same period last year. questions as in experiments 1 and 2. Perceived trustworthiness of the source This puts the total number of people who are unemployed at 1.42 was assessed with the item, “To what extent do you think the civil servants million. who are responsible for these migration figures are trustworthy?” on a scale from 1 = not at all to 7 = very trustworthy. In addition, we asked people to The number of those in work increased and wage growth improved over what extent they thought the conclusions based on the number; the news the same period. However, weak incomes have been a problem for a article they just read; and journalists who write articles such as the one they decade. “It will take a long period of wages rising above the rate of read were trustworthy, and to what extent government statistics in general inflation for people to feel significantly better off,” one economics were reliable (on scales from 1 = not at all to 7 = very trustworthy [reliable]). commentator is quoted as saying. Experiment 5: Field Experiment with BBC News. This version served as the control condition. We tested three numeric Sample and design. For this field experiment, we worked with BBC News and uncertainty communication formats, three verbal formats, and a mixed the BBC’s Head of Statistics. After gaining experience with the process of numeric/verbal format (Table 1). running a field experiment in this context during a Pilot Study in September Measures. After reading the text, participants were asked the same questions 2019 (SI Appendix), we conducted the experiment on October 15, 2019, as in experiments 1 and 2, except now specified for each number (the fall in using BBC News Online’s coverage of the October Labor Market Release unemployment and the total number of unemployed people) for the recall from the UK Office for National Statistics. After the labor market figures question, their perception of uncertainty around the numbers (α = 0.80) and were released, we worked with the relevant journalists and the Head of trust in the numbers (α = 0.92). For the analyses, answers were averaged Statistics to select a target figure to communicate uncertainty before the across both numbers given that there were no meaningful or significant news article was published on the website. The journalists were responsible differences between the two. Trust in the source was assessed with the item for the content of the news article. The target figure we selected was the UK “To what extent do you think the civil servants who are responsible for these unemployment rate, which was the first figure mentioned in the news story unemployment figures are trustworthy?” on a scale from 1 = not at all to 7 = (Fig. 6). The field experiment had three conditions: visitors of the website very trustworthy. In addition, we asked people to what extent they thought were randomly shown a version of the news article in which the target journalists who write articles such as the one they read were trustworthy, figure was presented without any uncertainty (“... rose to 3.9%”); with a and to what extent they thought government statistics in general were re- verbal uncertainty cue (“.. . rose to an estimated 3.9%”); or with a numeric liable (on scales from 1 = not at all to 7 = very trustworthy [reliable]). range and verbal cue [“... rose to an estimated 3.9% (between 3.7 and 4.1%)”]. At the bottom of the first paragraph of the news article, readers Experiment 4. were invited to “Click here to take part in a short study about this article run Sample and design. Experiment 4 was preregistered at aspredicted.org (https:// by the University of Cambridge.” aspredicted.org/blind.php?x=d3xu67). We recruited 1,050 adults who lived in BBC News website visitors were able to participate in the study for about the United Kingdom to participate in this study via Qualtrics Panels, based 24 h. During that time, 2,462 people clicked on the survey link, which took on power calculations that indicated we would need 995 participants to people to the starting page of the online survey with information about the have 90% power to detect a small (f = 0.125) effect when α was set at 0.05. study and informed consent. The survey was completed by 1,700 people (18 This sample was nationally representative of the general UK population in of whom completed the dependent variables but not demographics): 549 terms of age, gender, and geography quotas (51% female; mean age, 45.34 y; people in the control condition, 557 in the numeric condition, and 594 in the SD, 16.47; age range, 18 to 86). In this sample, 8.9% of the participants had verbal condition. A technical issue that was created when the journalistic no educational qualifications, 44.8% had attained upper secondary educa- team updated the story after its first release resulted in participants in both tion, and 46.1% had tertiary education. On average, the sample was again experimental conditions also being shown the control condition version of slightly leaning liberal (M = 3.74; SD = 1.51). the story, without any uncertainty mentioned, between 10:00 AM and 10:49 Treatment and procedure. We aimed to test whether we would find the same AM UK time. We therefore had to exclude all participants in the experi- results when communicating uncertainty around a more contentious topic, so mental conditions who participated between in that time frame, which were we presented people with a text about migration statistics based on a BBC 69 participants in the numerical and 94 in the verbal condition. We also News article of these Office of National Statistics figures (52): excluded five participants who reported to be below 18 y of age, and one outlier who reported being 114 y old (which was extremely unlikely). The Migration figures: EU migration still adding to UK population final sample consisted of 1,531 people: 520 participants in the control con- Official figures from last year show that there were 101,000 more people dition, 463 in the numerical condition, and 470 in the verbal condition. We coming to the UK from the EU than leaving in 2017. This is the lowest EU had no control over the exact number of people that would participate in net migration figure since 2013, but it means that EU migrants are still this field study, so we conducted a sensitivity analysis to compute the effect adding to the UK population. size that we should be able to detect with 1,531 participants in three groups, α = 0.05 and 90% power: which is a small effect, f = 0.09. There were 1,131 Net migration is the difference between the number of people coming men (73.9%) and 344 women (22.5%) who participated, with an average to live in the UK for at least 12 months and those emigrating. The 2017 age of 44.82 (SD, 15.29; range, 18 to 86). The sample was relatively highly overall net migration figure (both from the EU and non-EU countries) is educated: 30.5% indicated to have obtained a higher degree (MSc, PhD, or also down, from record highs in 2015 and early 2016. equivalent), 43.6% a bachelor’s degree, 22.5% school (GCSE, A-level, or equivalent), and 1% indicated to have not completed formal education. However, “The figures show that the government remains a long way Measures. After reading information about the study and providing informed off from meeting its objective to cut overall net migration, EU and non- consent, participants first answered a question about their current affec- EU, to the tens of thousands,” one Home Affairs correspondent is tive state, “How does the information you just read make you feel?”: quoted as saying. on a feeling thermometer from 0 = negative/unhappy to 10 = positive/ This experiment had five conditions (Table 1): Besides the control condi- happy, and subsequently a comprehension question (SI Appendix). Next, we tion (above, no uncertainty), uncertainty was communicated numerically assessed perceived uncertainty (“How certain or uncertain do you think the with a range after the point estimate, or via “+/− two standard errors”;or unemployment rate figure in the story is?”: on a scale from 1 = very certain verbally using the word “around” before the estimate of 101,000, or with an to 7 = very uncertain), trust in the number (“How trustworthy do you think explicit verbal statement. the unemployment rate figure in the story is?”:1 = not at all trustworthy to 7682 | www.pnas.org/cgi/doi/10.1073/pnas.1913678117 van der Bles et al. 7 = very trustworthy), trust in the source (“How trustworthy do you think ACKNOWLEDGMENTS. Funding was provided for these studies by the Nuffield Foundation (Grant OSP/43227). The Foundation has funded this the statisticians responsible for producing the figure are?”:1 = not at all project, but the views expressed are those of the authors and not necessarily trustworthy to 7 = very trustworthy), competence of the source (“How the Foundation. Prof. Sir David Spiegelhalter and Dr. Alexandra Freeman competent do you think the statisticians responsible for producing the fig- and the Winton Centre for Risk and Evidence Communication are supported ure are?”:1 = not at all competent to 7 = very competent), and trust in the by a donation from the David and Claudia Harding Foundation. We also journalistic source (“How trustworthy do you think the journalist responsible thank Robert Cuffe, the BBC Business team, the BBC Home Affairs team, and for writing the story is?”:1 = not at all trustworthy to 7 = very trustworthy). the BBC Digital team for their help and support in conducting the field experiment, but the views expressed are those of the authors and not The questionnaire finished with asking participants for their age, gender, necessarily those of the BBC. We also thank Vivien Chopurian for her and the highest level of education they had completed. assistance with study preparations and analyses, and a diverse advisory panel (John Aston, Robert Cuffe, Ed Humpherson, Onora O’Neill, Amy Sippitt, and Data Availability. The datasets collected and analyzed in the reported studies Elke Weber) who supported the directions of this research. This research are available on the Open Science Framework, https://osf.io/mt6s7/ (DOI:10.17605/ could not have been undertaken without the assistance of all the partici- OSF.IO/MT6S7). pants, whom we also thank. 1. B. Fischhoff, Communicating uncertainty: Fulfilling the duty to inform. Issues Sci. 28. D. A. Clark, Verbal uncertainty expressions: A critical review of two decades of re- Technol. 28,63–70 (2012). search. Curr. Psychol. 9, 203–235 (1990). 2. S. van der Linden, R. E. Löfstedt, Eds., Risk and Uncertainty in a Post-Truth Society 29. M. J. Druzdzel, Verbal Uncertainty Expressions: Literature Review (Carnegie Mellon (Routledge, 2019). University, Pittsburgh, PA, 1989). 3. Edelman, 2018 Edelman Trust Barometer. https://www.edelman.com/research/2018- 30. D. V. Budescu, T. S. Wallsten, Consistency in interpretation of probabilistic phrases. edelman-trust-barometer. Accessed 5 March 2020. Organ. Behav. Hum. Decis. Process. 36, 391–405 (1985). 4. Edelman, 2017 Edelman Trust Barometer. https://www.edelman.com/research/2017- 31. D. V. Budescu, H.-H. Por, S. B. Broomell, Effective communication of uncertainty in the edelman-trust-barometer. Accessed 5 March 2020. IPCC reports. Clim. Change 113, 181–200 (2012). 5. Pew Research Center, Beyond distrust: How Americans view their government. https:// 32. S. C. Jenkins, A. J. L. Harris, R. M. Lark, Understanding “unlikely (20% likelihood)” or www.people-press.org/2015/11/23/beyond-distrust-how-americans-view-their- “20% likelihood (unlikely)” outcomes: The robustness of the extremity effect. J. Behav. government/. Accessed 5 March 2020. Decis. Making 31, 572–586 (2018). 6. D. Ariely, Predictably Irrational: The Hidden Forces that Shape Our Decisions 33. B. B. Johnson, P. Slovic, Lay views on uncertainty in environmental health risk as- (HarperCollins, 2008). sessment. J. Risk Res. 1, 261–279 (1998). 7. National Academies of Sciences, Engineering, and Medicine, Communicating Science 34. N. F. Dieckmann, E. Peters, R. Gregory, At home on the range? Lay interpretations of Effectively: A Research Agenda (National Academies Press, 2017). numerical uncertainty ranges. Risk Anal. 35, 1281–1295 (2015). 8. A. Gustafson, R. E. Rice, The effects of uncertainty frames in three science commu- 35. N. F. Dieckmann, R. Gregory, E. Peters, R. Hartman, Seeing what you want to see: How nication topics. Sci. Commun. 41, 679–706 (2019). imprecise uncertainty ranges Enhance motivated reasoning. Risk Anal. 37, 471–486 9. B. B. Johnson, P. Slovic, Presenting uncertainty in health risk assessment: Initial studies (2017). of its effects on risk perception and trust. Risk Anal. 15, 485–494 (1995). 36. B. B. Johnson, Further notes on public response to uncertainty in risks and science. 10. S. Dunwoody, F. Hendriks, L. Massarani, H. P. Peters, “How journalists deal with sci- Risk Anal. 23, 781–789 (2003). entific uncertainty and what that means for the audience” in 15th International 37. P. K. J. Han et al., Communication of uncertainty regarding individualized cancer risk Public Communication of Science and Technology Conference (PCST 2018) (Public estimates: Effects and influential factors. Med. Decis. Making 31, 354–366 (2011). Communication of Science and Technology, 2018), pp. 3–6. 38. I. M. Lipkus, W. M. P. Klein, B. K. Rimer, Communicating breast cancer risks to women 11. M. Lehmkuhl, H. P. Peters, Constructing (un-)certainty: An exploration of journalistic using different formats. Cancer Epidemiol. Biomarkers Prev. 10, 895–898 (2001). decision-making in the reporting of neuroscience. Public Underst. Sci. 25,909–926 (2016). 39. S. T. Fiske, C. Dupree, Gaining trust as well as respect in communicating to motivated au- 12. C. H. Braddock, 3rd, K. A. Edwards, N. M. Hasenberg, T. L. Laidley, W. Levinson, In- diences about science topics. Proc. Natl. Acad. Sci. U.S.A. 111 (suppl. 4), 13593–13597 (2014). formed decision making in outpatient practice: Time to get back to basics. JAMA 282, 40. O. O’Neill, Reith lectures 2002: A question of trust, lecture 4: Trust and transparency. 2313–2320 (1999). http://downloads.bbc.co.uk/rmhttp/radio4/transcripts/20020427_reith.pdf. Accessed 5 13. M. C. Politi, P. K. J. Han, N. F. Col, Communicating the uncertainty of harms and March 2020. benefits of medical interventions. Med. Decis. Making 27, 681–695 (2007). 41. S. Fischer, R. Fitzegerald, W. Poortinga, “Climate change” in British Social Attitudes: 14. M. C. Politi, M. A. Clark, H. Ombao, D. Dizon, G. Elwyn, Communicating uncertainty The 35th Report, D. Philips, J. Curtice, M. Philips, J. Perry, Eds. (The National Centre for can lead to less decision satisfaction: A necessary cost of involving patients in shared Social Research, 2018), pp. 146–171. decision making? Health Expect. 14,84–91 (2011). 42. S. Blinder, L. Richards, Briefing: UK public opinion towards immigration: Overall at- 15. M. Osman, A. J. Heath, R. Löfstedt, The problems of increasing transparency on un- titudes and level of concern. https://migrationobservatory.ox.ac.uk/resources/briefings/ certainty. Public Underst. Sci. 27, 131–138 (2018). uk-public-opinion-toward-immigration-overall-attitudes-and-level-of-concern/. Accessed 16. C. F. Manski, Communicating uncertainty in policy analysis. Proc. Natl. Acad. Sci. U.S.A. 5 March 2020. 116, 7634–7641 (2019). 43. C. Fox, A. Tversky, Ambiguity aversion and comparative ignorance. Q. J. Econ. 110, 17. R. E. Lofstedt, M. McLoughlin, M. Osman, Uncertainty analysis: Results from an em- 585–603 (1995). pirical pilot study. A research note. J. Risk Res. 9877,1–11 (2017). 44. Y. Bar-Anan, T. D. Wilson, D. T. Gilbert, The feeling of uncertainty intensifies affective 18. A. Hart et al., Guidance on communication of uncertainty in scientific assessments. reactions. Emotion 9, 123–127 (2009). EFSA J. 17, e05520 (2019). 45. M. A. Hillen, C. M. Gutheil, T. D. Strout, E. M. A. Smets, P. K. J. Han, Tolerance of 19. B. Fischhoff, A. L. Davis, Communicating scientific uncertainty. Proc. Natl. Acad. Sci. uncertainty: Conceptual analysis, integrative model, and implications for healthcare. U.S.A. 111 (suppl. 4), 13664–13671 (2014). Soc. Sci. Med. 180,62–75 (2017). 20. R. E. Lofstedt, F. Bouder, Evidence-based uncertainty analysis: What should we now 46. E. Peer, L. Brandimarte, S. Samat, A. Acquisti, Beyond the Turk: Alternative platforms do in Europe? A view point. J. Risk Res. 9877,1–20 (2017). for crowdsourcing behavioral research. J. Exp. Soc. Psychol. 70, 153–163 (2017). 21. A. M. van der Bles et al., Communicating uncertainty about facts, numbers and sci- 47. S. Palan, C. Schitter, Prolific.ac—a subject pool for online experiments. J. Behav. Exp. ence. R. Soc. Open Sci. 6, 181870 (2019). Finance 17,22–27 (2018). 22. Office for National Statistics, Labour market overview, UK: May 2019. https://www.ons. 48. OECD, Education at a Glance 2018: OECD Indicators (OECD Publishing, 2018). gov.uk/employmentandlabourmarket/peopleinwork/employmentandemployeetypes/ 49. Office for National Statistics, UK labour market: August 2017. https://www.ons.gov. bulletins/uklabourmarket/may2019. Accessed 5 March 2020. uk/employmentandlabourmarket/peopleinwork/employmentandemployeetypes/ 23. H. Cockburn, Donald Trump again claims to have largest presidential inauguration bulletins/uklabourmarket/august2017/. Accessed 5 March 2020. audience in history. Independent, 26 January 2017. https://www.independent.co.uk/ 50. J. Goodrich et al., Panthera tigris. The IUCN Red List of Threatened Species. https://dx. news/world/americas/donald-trump-claims-presidential-inuauguration-audience-history- doi.org/10.2305/IUCN.UK.2015-2.RLTS.T15955A50659951.en. Accessed 5 March 2020. us-president-white-house-barack-a7547141.html. Accessed 5 March 2020. 51. IPCC, “Summary for policymakers” in Climate Change 2013: The Physical Science Basis. 24. R. Meyer, How will we know Trump’s inaugural crowd size? The Atlantic, 20 January Contribution of Working Group I to the Fifth Assessment Report of the Intergov- 2017. https://www.theatlantic.com/technology/archive/2017/01/how-will-we-know-trumps- ernmental Panel on Climate Change, T. F. Stocker et al., Eds. (IPCC, 2013), pp. 33–36. inaugural-crowd-size/513938/. Accessed 5 March 2020. 52. Office for National Statistics, Migration statistics quarterly report: July 2018 (re- 25. C. R. Fox, G. Ulkumen, “Distinguishing two dimensions of uncertainty” in Perspectives scheduled from May 2018). https://www.ons.gov.uk/peoplepopulationandcommunity/ on Thinking, Judging, and Decision Making (Universitetsforlaget, 2011), pp. 21–35. populationandmigration/internationalmigration/bulletins/migrationstatisticsquarterlyreport/ 26. G. Ülkümen, C. R. Fox, B. F. Malle, Two dimensions of subjective uncertainty: Clues july2018revisedfrommaycoveringtheperiodtodecember2017. Accessed 5 March 2020. from natural language. J. Exp. Psychol. Gen. 145, 1280–1297 (2016). 53. European Social Survey, ESS Round 8 source questionnaire. https://www. 27. G. Keren, L. E. M. Gerritsen, On the robustness and possible accounts of ambiguity europeansocialsurvey.org/docs/round8/fieldwork/source/ESS8_source_questionnaires. aversion. Acta Psychol. (Amst.) 103, 149–172 (1999). pdf. Accessed 5 March 2020. van der Bles et al. PNAS | April 7, 2020 | vol. 117 | no. 14 | 7683 PSYCHOLOGICAL AND COGNITIVE SCIENCES http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Proceedings of the National Academy of Sciences of the United States of America Pubmed Central

The effects of communicating uncertainty on public trust in facts and numbers

Proceedings of the National Academy of Sciences of the United States of America , Volume 117 (14) – Mar 23, 2020

Loading next page...
 
/lp/pubmed-central/the-effects-of-communicating-uncertainty-on-public-trust-in-facts-and-irsBd0NlgL

References

References for this paper are not available at this time. We will be adding them shortly, thank you for your patience.

Publisher
Pubmed Central
Copyright
Copyright © 2020 the Author(s). Published by PNAS.
ISSN
0027-8424
eISSN
1091-6490
DOI
10.1073/pnas.1913678117
Publisher site
See Article on Publisher Site

Abstract

The effects of communicating uncertainty on public trust in facts and numbers a,b,c,1 a,b,d,1 a,b Anne Marthe van der Bles , Sander van der Linden , Alexandra L. J. Freeman , a,b and David J. Spiegelhalter  a b Winton Centre for Risk and Evidence Communication, University of Cambridge, Cambridge CB3 0WA, United Kingdom; Department of Pure Mathematics and Mathematical Statistics, University of Cambridge, Cambridge CB3 0WA, United Kingdom; Department of Social Psychology, University of Groningen, 19712 TS Groningen, The Netherlands; and Cambridge Social Decision-Making Lab, Department of Psychology, University of Cambridge, Cambridge CB2 3RQ, United Kingdom Edited by Arild Underdal, University of Oslo, Oslo, Norway, and approved February 20, 2020 (received for review August 7, 2019) Uncertainty is inherent to our knowledge about the state of the the general sense of honesty evoked [by uncertainty] .. . this did not world yet often not communicated alongside scientific facts and seem to offset concerns about the agency’s competence” (p. 491). In numbers. In the “posttruth” era where facts are increasingly con- fact, partly for these reasons, Fischhoff (1) notes that scientists may tested, a common assumption is that communicating uncertainty be reluctant to discuss the uncertainties of their work. This hesita- will reduce public trust. However, a lack of systematic research tion spans across domains: For example, journalists find it difficult makes it difficult to evaluate such claims. We conducted five exper- to communicate scientific uncertainty and regularly choose to ig- iments—including one preregistered replication with a national nore it altogether (10, 11). Physicians are reluctant to communicate sample and one field experiment on the BBC News website (total uncertainty about evidence to patients (12), fearing that the com- n = 5,780)—to examine whether communicating epistemic uncer- plexity of uncertainty may overwhelm and confuse patients (13, 14). tainty about facts across different topics (e.g., global warming, im- Osman et al. (15) even go as far as to argue explicitly that “the drive migration), formats (verbal vs. numeric), and magnitudes (high vs. to increase transparency on uncertainty of the scientific process low) influences public trust. Results show that whereas people do specifically does more harm than good” (p. 131). perceive greater uncertainty when it is communicated, we observed At the same time, many organizations that produce and only a small decrease in trust in numbers and trustworthiness of the communicate evidence to the public, such as the European Food source, and mostly for verbal uncertainty communication. These Safety Authority, have committed themselves to openness and results could help reassure all communicators of facts and science transparency about their (scientific) work, which includes com- that they can be more open and transparent about the limits of municating uncertainties around evidence (16–19). These at- human knowledge. tempts have not gone without criticism and discussion about the potential impacts on public opinion (15, 20). What exactly do we communication uncertainty trust posttruth contested | | | | know about the effects of communicating uncertainty around facts, numbers, and science to the public? ur knowledge is inherently uncertain. The process by which Owe gather information about the state of the world is char- Significance acterized by assumptions, limitations, extrapolations, and gener- alizations, which brings imprecisions and uncertainties to the facts, Does openly communicating uncertainty around facts and numbers, and scientific hypotheses that express our understanding numbers necessarily undermine audiences’ trust in the facts, or of the world around us. However, despite the fact that scientists the communicators? Despite concerns among scientists, ex- and other producers of knowledge are usually well-aware of the perts, and journalists, this has not been studied extensively. In uncertainties around their findings, these are often not commu- four experiments and one field experiment on the BBC News nicated clearly to the public and other key stakeholders (1). This website, words and numerical ranges were used to communi- lack of transparency could potentially compromise important de- cate uncertainty in news article-like texts. The texts included cisions people make based on scientific or statistical evidence, contested topics such as climate change and immigration sta- from personal medical decisions to government policies. tistics. While people’s prior beliefs about topics influenced their Recent societal developments do not seem to encourage more trust in the facts, they did not influence how people responded openness about uncertainty: It has been suggested that we are to the uncertainty being communicated. Communicating un- living in a “posttruth” era in which facts, evidence, and experts certainty numerically only exerted a minor effect on trust. are deeply mistrusted (2). Cross-national survey studies suggest Knowing this should allow academics and science communi- that in many countries, trust in institutions and governments is in cators to be more transparent about the limits of human decline (3–5). Although the underlying causes of changes in trust knowledge. are likely to be complex and varied, it has been suggested that Author contributions: A.M.v.d.B., S.v.d.L., A.L.J.F., and D.J.S. designed research; one way to potentially repair and restore public trust in science, A.M.v.d.B., S.v.d.L., and A.L.J.F. performed research; A.M.v.d.B. and S.v.d.L. analyzed data; evidence, and official statistics is to be more open and trans- and A.M.v.d.B., S.v.d.L., A.L.J.F., and D.J.S. wrote the paper. parent about scientific uncertainty (2). However, it is often as- The authors declare no competing interest. sumed that communicating uncertainty transparently will invite This article is a PNAS Direct Submission. criticism, can signal incompetence, or even decrease public trust in This open access article is distributed under Creative Commons Attribution License 4.0 science (1, 6–8). In fact, as summarized by the National Acade- (CC BY). mies of Sciences, Engineering, and Medicine report on science Data deposition: The datasets collected and analyzed in this paper are available on the communication, “as a rule, people dislike uncertainty [...] people Open Science Framework (https://doi.org/10.17605/OSF.IO/MT6S7). may attribute uncertainty to poor science [.. . and] in some cases, 1 To whom correspondence may be addressed. Email: a.m.van.der.bles@rug.nl or sander. communicating uncertainty can diminish perceived scientific au- vanderlinden@psychol.cam.ac.uk. thority” (ref. 7, pp. 27–28). For example, research by Johnson and This article contains supporting information online at https://www.pnas.org/lookup/suppl/ doi:10.1073/pnas.1913678117/-/DCSupplemental. Slovic (9) found that for some respondents, uncertainty “evoked doubt about agency trustworthiness” (p. 490), and that “despite First published March 23, 2020. 7672–7683 | PNAS | April 7, 2020 | vol. 117 | no. 14 www.pnas.org/cgi/doi/10.1073/pnas.1913678117 Prior research has distinguished between two kinds of un- statistics. From a purely statistical point of view, variability certainty: epistemic uncertainty about the past and present state around a central estimate signals that the estimate is more un- of the world that arises because of what we do not know but certain; thus, when uncertainty is taken into account, a logical could know in theory (e.g., uncertainty due to limitations of the conclusion would be that the central estimate is less informative sample or methodology) vs. uncertainty about the future that and reliable. However, communicating uncertainty when it exists arises because we cannot know (i.e., randomness, chance; we can also be seen as part of an organization’s goal to be open and cannot know for certain what will happen tomorrow) (21). Al- transparent, which could foster perceived trustworthiness (40). though uncertainty about the future is a widely acknowledged Given limited and mixed prior findings, we set out to systemat- aspect of forecasts and predictions, epistemic uncertainty about ically examine how communicating uncertainty about a range of the past and present is equally important yet often overlooked in facts influences public trust in both numbers and their sources in communication. It is the uncertainty around the decrease in four large and diverse online experiments (combined n = 4,249) unemployment in the United Kingdom (e.g., estimated at a de- and one field experiment on the BBC News website (n = 1,531). crease of 119,000 people from January to March 2019 compared We were particularly interested in comparing the effects of to a year earlier, with a 95% CI of ±96,000) (22); or uncertainty communicating both verbal and numerical uncertainty around a around the number of people who attended US President Trump’s (contested) numeric estimate on perceived uncertainty (cogni- inauguration, which he famously claimed “had the biggest audience tion) and perceived trustworthiness.* in the history of inaugural speeches” (23) (attendance is estimated In experiment 1, 1,122 participants read a short text about one to have been between 300,000 and 600,000) (24). Psychological of three topics, which contained either no uncertainty (just a research suggests that people intuitively distinguish between these point estimate), uncertainty communicated as a numerical range two kinds of uncertainty (25, 26). (in addition to the point estimate), or uncertainty communicated How people react to aleatoric uncertainty about the future has as a verbal statement (also in addition to the point estimate). been relatively well studied. A large literature indicates that The three topics were as follows: the number of unemployed people are generally averse to uncertainty when making decisions people in the United Kingdom, the number of tigers currently about the future—a psychological tendency known as ambiguity left in India, and the increase in the global average surface aversion (27). Moreover, the relatively large body of research on temperature between 1880 and 2010. We selected these topics to the interpretation of verbal expressions of uncertainty such as represent various “types” of numbers (large vs. small) as well as “likely” or “unlikely” shows that there is considerable variability for variation in their level of “contestedness.” That is, we between people in how they interpret these words, creating expected there to be different levels of variation in prior atti- problems for effective communication (28–31). It has therefore tudes toward the topics: Whereas opinions on climate change are been suggested that communicating uncertainty numerically might divided in the United Kingdom (41), we anticipated less division be preferable; yet several studies found that numerical uncertainty in opinion around the conservation of endangered animals. might suffer from its own interpretation issues (32–34). To illus- In the short text about unemployment, for example, partici- trate, recent research has found that motivated cognition can have pants read that recently an official report had been published, an impact on probabilistic reasoning: Prior beliefs about climate which stated that between April and June 2017, the number of change or gun laws in the United States influenced people’sin- unemployed people in the United Kingdom was an estimated terpretation of the distribution underlying ambiguous numerical 1,484,000. Participants then either received no further informa- ranges about these issues (35). tion (control condition), or a numerical range (numerical un- However, much less research has specifically focused on epistemic certainty condition: “minimum 1,413,000 to maximum 1,555,000”), uncertainty. In fact, a recent review concluded that empirical evi- or an equivalent verbal statement (verbal uncertainty condition: dence about the psychological effects—positive or negative—of “The report states that there is some uncertainty around this es- communicating epistemic uncertainty about facts and numbers is timate, it could be somewhat higher or lower”). The source of the limited and scattered with mixed findings (21). For example, numbers was an “official report,” which was not further specified in research by Johnson and Slovic (9, 33, 36) suggests that com- our first experiment in order to avoid source bias. municating uncertainty via numerical ranges signaled honesty After reading the short text, participants first indicated how and competency for some of their participants, but dishonesty the information made them feel on a feeling thermometer, then and incompetency for others. Other studies, for example, in the were asked to recall the number they had just read about, and context of cancer risk or scientific evidence about climate change subsequently were asked a series of questions about how un- and genetically modified organisms, found that communicating certain and reliable they perceived the number to be, and how uncertainty around estimates did not seem to affect people’s trustworthy they thought the writers of the report were. Our scientific beliefs or credibility judgments (8, 37, 38). measures of trust were modeled after prior studies from Johnson Given these mixed findings, the present research program and Slovic (9). To ensure that our findings are systematic and aims to address this gap in the literature and is one of the first to robust, follow-up experiments—including a preregistered repli- examine the effects of communicating epistemic uncertainty cation—varied the magnitude of the uncertainty in the message, about facts, numbers, and evidence on public trust. We examine the style and format of how the uncertainty was communicated, the effects of communicating uncertainty around numbers— as well as the sampling platform. including contested numbers—that are communicated routinely in the media, such as the unemployment rate and migration Results statistics. Specifically, we draw on a recent theoretical review The analyses revealed no meaningful differences between topics (21), which suggests that research on communicating epistemic in how people reacted to the uncertainty communication. For uncertainty should consider its effects on three key conceptual ease of interpretation, we therefore present the results collapsed dimensions, namely, 1) cognition (how people perceive and un- across topics (but see SI Appendix for further details). The results derstand uncertainty), 2) emotion (how people feel about the showed that our manipulation was effective in that participants uncertainty), and 3) trustworthiness (the extent to which people perceived the numbers to be more uncertain when uncertainty trust the information). Following recent work on source credi- was communicated either numerically or verbally (Fig. 1A). An bility in science communication (39), we further consider and conceptually distinguish two key forms of trust: people’s trust in the numbers themselves and people’s trust in the “source” of *Effects on people’s emotions were of secondary interest and are reported in detail in the these numbers, for example, in the organization producing the SI Appendix. van der Bles et al. PNAS | April 7, 2020 | vol. 117 | no. 14 | 7673 PSYCHOLOGICAL AND COGNITIVE SCIENCES 2 analysis of variance (ANOVA) testing the effect of uncertainty communication format [F = 60.96, P < 0.001; η = 0.10]. (2,1119) communication (control vs. numerical vs. verbal) on perceived un- Verbal uncertainty communication reduced people’s trust in certainty of the number showed a significant main effect [F = (2, 1119) numbers compared to the control condition (M = 3.51 vs. 4.52, 138.67, P < 0.001; η = 20]. All reported post hoc paired comparisons M = −1.01, 95% CI [−1.23; −0.79], d = 0.75) and compared to diff used Tukey’s honestly significant difference. People who were pre- numerical uncertainty communication (M = 3.51 vs. 4.31, M = diff sented with uncertainty as a numeric range perceived the numbers to −0.80, 95% CI [−1.03; −0.57], d = 0.60). Importantly, there was be significantly more uncertain compared to people in the control no significant difference in trust in numbers between the nu- merical uncertainty communication and control conditions (M = condition [M = 4.78 vs. 4.14, M = 0.64, 95% confidence interval diff 4.31 vs. 4.52, M = −0.21, 95% CI [−0.43; 0.01], d = 0.17). (CI) [0.40; 0.88], d = 0.45]. Participants who were presented with diff Finally, we asked participants how trustworthy they thought uncertainty as a verbal statement perceived the numbers to be “the writers of the report” were, as a measure of trust in the significantly more uncertain than both those in the numerical (M = source. Again, we found that the verbal uncertainty communi- 5.82 vs. 4.78, M = 1.04, 95% CI [0.80; 1.28], d = 0.79) and the diff cation led to a small significant decrease in people’s trust in the control conditions (M = 5.82 vs. 4.14, M = 1.68, 95% CI [1.44; diff source, whereas the numerical uncertainty communication did 1.92], d = 1.17). not (Fig. 1C). The ANOVA showed a main effect of uncertainty We also asked participants to indicate how reliable and how trustworthy they thought the numbers were; given their correlation communication format [F = 11.03, P < 0.001; η = 0.02]. (2, 1119) (r = 0.88), scores on these two questions were combined to form a Communicating uncertainty verbally reduced participant’s trust measure of “trust in numbers.” The results showed that, although in the source, compared to the control condition (M = 4.19 vs. 4.55, M = −0.36, 95% CI [−0.57; −0.15], d = 0.28) and nu- our verbal phrase of uncertainty communication decreased trust in diff merical uncertainty communication (M = 4.19 vs. 4.58, M = numbers, our numerical uncertainty communication did not diff −0.39, 95% CI [−0.61; −0.17], d = 0.31). Again, there was no (Fig. 1B). The ANOVA revealed a main effect of uncertainty significant decrease for numerical uncertainty communication compared to control (M = 4.58 vs. 4.55, M = 0.03, 95% CI diff [−0.18; 0.24], d = 0.02). Perceived uncertainty Experiment 1 thus showed that while people did perceive uncertainty about numbers both when it was communicated numerically and verbally, only the verbal communication re- duced people’s trust in the numbers and the source. In addition, the results showed no significant effect of uncertainty commu- nication on people’s affect or mood; please see SI Appendix for the full results. Because we found no substantial differences between topics in people’s responses to uncertainty, experiment 2 only used the UK employment number to study whether the magnitude of the uncertainty itself can influence the psycho- 1 logical effects of communicating uncertainty. Control Numerical Verbal Manipulating the Magnitude of Uncertainty. The goal of experi- ment 2 was twofold: first, to replicate the results from experi- Trust in number ment 1 (for unemployment) and second, to examine whether 7 the magnitude of the uncertainty affected people’s trust in numbers and trust in the source. This experiment followed a 1 (control condition: no uncertainty) + 2 (numerical vs. verbal communication) × 3 (lower vs. original vs. higher uncertainty) between-subject design. For numerical uncertainty communi- cation, we presented the original 95% CI, which therefore acted as a replication of experiment 1; lower uncertainty using a 2 range half the size (99.99% CI); or higher uncertainty using a range twice as large (68% CI) as the original CI. For verbal uncertainty communication, wepresented thesamebaseline Control Numerical Verbal phrase as in experiment 1 for the “original” magnitude (“...it could be somewhat higher or lower”); less uncertainty using the Trust in source phrase “slightly higher or lower”; or more uncertainty using the phrase “a lot higher or lower.” The verbal phrases were chosen to mirror the magnitude of the numerical uncertainty. First, we analyzed the results of the “original uncertainty” levels and the control condition in experiment 2: a direct repli- cation of experiment 1. For perceived uncertainty and trust in the number, we replicated the results of the first experiment. The analyses, all reported in detail in the SI Appendix, showed that participants perceived the number to be significantly more un- certain when numerical uncertainty was communicated (compared to control) and when verbal uncertainty was communicated Control Numerical Verbal (compared to both control and numerical uncertainty). Similarly, just as in experiment 1, participants reported less trust in the Fig. 1. The results of experiment 1: Means per condition for perceived number when verbal uncertainty was communicated compared to uncertainty (A), trust in numbers (B), and trust in the source (C). The error both control and when numerical uncertainty was communicated; bars represent 95% CIs around the means, and jitter represents the distri- bution of the underlying data. with no significant decrease in trust for numerical uncertainty 7674 | www.pnas.org/cgi/doi/10.1073/pnas.1913678117 van der Bles et al. Score Mean score Mean score [−0.83; −0.45], d = 0.48). In addition, across formats, post hoc Perceived uncertainty comparisons showed that a lower magnitude of uncertainty led to higher trust in numbers compared to the original range or phrase (M = 4.06 vs. 3.75, M = 0.31, 95% CI [0.03; 0.58], d = 0.23). diff Both comparisons were not significantly different from higher magnitude of uncertainty (Fig. 2B). For trust in the source, communication format (numerical or verbal) appeared to make no difference, but the magnitude of the uncertainty did (Fig. 2C). An ANOVA of format and magnitude Numerical Verbal Numerical Verbal Numerical Verbal showed no main effect of format [F = 0.96, P = 0.33], a (1, 741) Low uncertainty Actual uncertainty High uncertainty significant main effect of magnitude [F = 4.30, P = 0.01; (2, 741) Trust in number η = 0.01], but no significant interaction [F = 0.37, P = 0.69]. (2, 741) Regardless of format, lower magnitudes of uncertainty led to higher levels of trust in the source compared to the original range or phrase (M = 4.34 vs. 4.01, M = 0.33, 95% CI [0.06; 5 diff 0.60], d = 0.26), but neither were significantly different from higher magnitude of uncertainty (M = 4.23, SD = 1.31; Fig. 2C). The results of experiment 2 thus suggest that magnitude, communicated without further context, did not have a strong Numerical Verbal Numerical Verbal Numerical Verbal impact on people’s reactions to uncertainty communication: It did not influence perceptions of uncertainty, and only lower Low uncertainty Actual uncertainty High uncertainty magnitudes of uncertainty were related to slightly higher levels of Trust in source perceived reliability of the number and trustworthiness of the source. Without further context, participants might not have been able to interpret the numerical ranges as being relatively small or large magnitudes of uncertainty, although the same is 4 not true of the verbal conditions. Although preliminary, what we 3 can conclude from these results is that, in the absence of further 2 context, it appears that whether and how uncertainty is com- 1 municated is more important in determining how people re- Numerical Verbal Numerical Verbal Numerical Verbal spond than the magnitude of the uncertainty in question. Low uncertainty Actual uncertainty High uncertainty Varying the Format of Uncertainty Communication. Following these Fig. 2. The results of experiment 2: Means per condition for perceived findings, we set out to systematically test the effects of additional uncertainty (A), trust in numbers (B), and trust in the source (C). The error numeric and verbal uncertainty communication formats in ex- bars represent 95% CIs around the means, and jitter represents the distri- periment 3, and to move toward a more realistic and better con- bution of the underlying data. textualized communication scenario. This experiment had eight conditions, which are presented in Table 1. The choice of formats was influenced by the formats the UK Office for National Sta- (compared to control). However, in contrast to experiment 1 where tistics uses to communicate uncertainty around unemployment we found that verbal uncertainty communication reduced trust in the source, experiment 2 showed no significant effect of uncertainty numbers, which was again the context we used for this experiment for consistency. To improve the ecological validity of the experi- communication: Both numerical and verbal uncertainty communi- ment, the manipulation was written as a traditional news media cated did not decrease people’s trust in the source, compared to article and included two unemployment figures. Uncertainty was control (please see SI Appendix,Fig.S2 A–C). Next, we examined the effects of the magnitude of uncertainty. communicated in the same format around both figures. Results are presented in Fig. 3. The level of perceived un- Somewhat surprisingly, we found that the magnitude of the communicated uncertainty did not affect people’s perceptions of certainty around the numbers differed between formats [Fig. 3A; the uncertainty of the numbers (Fig. 2A). A two-way ANOVA of one-way ANOVA effect of format: F = 14.43, P < 0.001; (7, 1192) format (numeric vs. verbal) and magnitude (lower vs. original vs. η = 0.08]. For all formats in which uncertainty was being com- higher) showed a significant main effect of format [F = municated, except one, participants perceived the numbers to be (1, 741) 67.93, P < 0.001; η = 0.08], but no significant main effect of more uncertain compared to the control condition (post hoc paired comparisons:p values = 0.022 to <0.001; M = 0.53 to diff magnitude [F = 2.92, P = 0.055; η = 0.01] nor a significant (2, 741) 1.10; d values = 0.37 to 0.72). The exception was the condition in interaction [F = 1.86, P = 0.16; η = 0.01]. Regardless of (2, 741) which only the word “estimated” had been added (M = −0.07, diff magnitude, and as in experiment 1, verbal uncertainty communi- P = 1.00). People in this condition did not perceive the numbers to cation led to higher levels of perceived uncertainty than numerical be more uncertain compared to those in the control condition, uncertainty communication (M = 5.50 vs. 4.68, M = 0.83, 95% diff indicating that only using the word “estimated” seems insufficient CI [0.63; 1.03], d = 0.60). to communicate the existence of uncertainty around a number. However, the results did show a small effect of magnitude on People’s trust in numbers similarly differed between formats people’s trust in numbers (Fig. 2B). An ANOVA of format and [Fig. 3B; F = 5.97, P < 0.001; η = 0.03]. Whereas for most (7, 1192) magnitude showed a main effect of format [F = 43.44, P < (1, 741) formats, people’s trust in the numbers was significantly reduced 0.001; η = 0.06], and main effect of magnitude [F = 3.63, (2, 741) compared to control (post hoc paired comparisons: p values = P = 0.03; η = 0.01], but no significant interaction [F = 1.44, (2, 741) 0.035 to 0.011; M = −0.47 to −0.53; d values = 0.35 to 0.37), diff this was not the case when uncertainty was communicated with P = 0.24]. Regardless of magnitude, verbal uncertainty commu- nication decreased trust in numbers compared to numerical the word “estimated” (M = 0.13, 95% CI [−0.33; 0.59], P = diff communication (M = 3.56 vs. 4.20, M = −0.64, 95% CI 0.98), with the implicit uncertainty statement (M = −0.37, diff diff van der Bles et al. PNAS | April 7, 2020 | vol. 117 | no. 14 | 7675 Score Mean score Mean score PSYCHOLOGICAL AND COGNITIVE SCIENCES Table 1. Overview of the conditions and manipulation texts of experiment 3 and 4 Format Experiment 3 Experiment 4 Control “Official figures from the first quarter of 2018 show “Migration figures: EU migration still (no uncertainty) that UK unemployment fell by 116,000 compared adding to UK population. Official figures from with the same period last year. [...]” last year show that there were 101,000 more people coming to the UK from the EU than leaving in 2017. This is the lowest EU net migration figure since 2013, but it means that EU migrants are still adding to the UK population. [...]” Numerical range with ...by 116,000 (range between 17,000 and 215,000)... . . .101,000 (range between 68,000 and point estimate 132,000)... Numerical range ...by between 17,000 and 215,000... without point estimate Numerical point ...by 116,000 (±99,000)... . . .101,000 (±33,000)... estimate ±2SEs Verbal explicit ...by 116,000 compared with the same period last year, .. .101,000 more people coming to the UK from uncertainty statement although there is some uncertainty around this the EU than leaving in 2017. The report states figure: It could be somewhat higher or lower. [...] there is uncertainty around the exact figure— it could be higher or lower. [.. .] Verbal implicit ...by 116,000 compared with the same period last year, uncertainty statement although there is a range around this figure: could be somewhat higher or lower. [...] Verbal uncertainty word ...by an estimated 116,000... . . .around 101,000... Mixed numerical and verbal phrase ...by an estimated 116,000 (±99,000)... 95% CI [−0.83; 0.08], P = 0.20), or with a numerical range with of the civil servants responsible for the statistics, nor for jour- point estimate (M = −0.10, 95% CI [−0.56; 0.36], P = 1.00). nalists who write such articles. diff In this experiment, we assessed trust in the source by asking Uncertainty Around Contested Numbers. Experiments 2 and 3 were people to what extent they thought that the civil servants who were responsible for the unemployment figures were trustworthy. conducted in the context of UK unemployment numbers, which Results are shown in Fig. 3C. We found that different formats are generally considered not highly contested and thus might be did make a small difference to people’s trust in the source [one- less likely to result in changes in trust-related perceptions. We way ANOVA: F = 2.15, P = 0.04; η = 0.01]. However, therefore conducted experiment 4 in the context of UK migration, (7, 1192) that difference was not between the conditions in which un- which is a more contested issue on which public opinion is divided certainty was communicated and the control: Across all formats, (42). Experiment 4 was preregistered on aspredicted.org (https:// trust in the source did not differ significantly from the control aspredicted.org/blind.php?x=d3xu67) and also conducted on a condition (range M = 3.94 to 4.48 vs. M = 4.24, SD = control control national sample of the UK population (Methods). Participants 1.55). The difference was between specific formats: Compared to were first asked about their attitudes toward migration, before people to whom uncertainty was communicated through the word being randomly selected to read one of five versions of a fictitious “estimated,” people to whom uncertainty was communicated in newspaper article about migration statistics, which are presented the numeric +/− format or mixed format (“estimated +/−”)per- in Table 1. ceived the source to be significantly less trustworthy (M = 4.48 vs. The results of experiment 4 showed that, similar to experiment 3.94, M = 0.54, 95% CI [0.02; 1.06], d = 0.40 and M = 4.48 vs. diff 3, participants perceived the number to be more uncertain for all 3.94, M = 0.54, 95% CI [0.03; 1.05], d = 0.39, respectively). diff communication formats compared to the control condition, except To examine the boundary conditions of the effects on trust, we for just using the word “around” before the number [Fig. 4A;one- also asked people to indicate how trustworthy they thought jour- way ANOVA F = 22.11, P < 0.001; η = 0.08]. Post hoc (4, 1045) nalists who write news articles like the ones they had read were paired comparisons showed significant differences between the and how reliable they thought government statistics in general are; control condition vs. communicating uncertainty as a numeric these judgements did not differ significantly for different uncer- range (M = 4.47 vs. 5.27, M = −0.80, 95% CI [−1.22; −0.39], 2 diff tainty communication formats [F = 1.60, P = 0.13; η = 0.01, (7, 1192) d = 0.50), using “+/−” (M = 4.47 vs. 5.24, M = −0.77, 95% CI diff and F = 1.60, P = 0.13; η = 0.01, respectively]. (7, 1192) [−1.19; −0.35], d = 0.50), and as an explicit verbal statement (M = In conclusion, then, the results of experiment 3 showed that 4.47 vs. 5.52, M = −1.06, 95% CI [−1.48; −0.64], d = 0.68). diff whereas participants perceived uncertainty when uncertainty was As Fig. 4B shows, communication formats did affect partici- communicated in most numeric and verbal formats, not all for- pants’ trust in numbers [one-way ANOVA F = 7.29, P < (4, 1044) mats affected people’s trust in the numbers. Communicating 0.001; η = 0.03], but this overall effect was qualified by a sig- uncertainty via a numerical range with point estimate or an im- nificant decrease in trust for the explicit verbal statement when plicit verbal statement did not significantly decrease trust in compared to control (M = 3.28 vs. 3.90, M = −0.62, 95% CI numbers compared to the control condition. Adding the word diff [−1.01; −0.23], d = 0.42). The numerical formats and the verbal “estimated” also did not decrease trust, but this format apparently “around” condition did not significantly reduce trust in numbers failed to communicate uncertainty to people. However, impor- compared to control. tantly, just as in experiment 2, none of the uncertainty commu- Furthermore, we found no effect of uncertainty communi- nication formats decreased trust in the source compared to not cation on participants’ trust in the source, which in this exper- communicating uncertainty: There was no impact of uncertainty communication when we asked people about the trustworthiness iment was assessed as perceived trustworthiness of “the civil 7676 | www.pnas.org/cgi/doi/10.1073/pnas.1913678117 van der Bles et al. Third, consistent with experiment 3, participants’ trust in Perceived uncertainty journalists and in government statistics in general were not sig- nificantly affected by communicating uncertainty in these dif- ferent formats (one-way ANOVAs: F = 0.85, P = 0.50, 6 (4, 1044) 5 and F = 1.73, P = 0.14, respectively). These additional (4, 1045) findings suggest that communicating uncertainty verbally has an impact on the perceived reliability of the number itself and con- clusions based on a number, but does not seem to impact judge- ments about the source(s) of numbers (civil servants or journalists), Control Numerical Numerical Numerical Verbal Verbal Verbal Mixed +/− Range Range Cue Explicit Implicit Numerical/ nor generalize to governmental statistics more broadly. Estimate No Estimate Verbal Given the contested nature of immigration statistics, we also Trust in number explored the extent to which people’s prior attitudes toward immigration affected the results. We split our sample into two groups (based on the median = 4.33): people with negative at- titudes toward immigration (mean, 1.00 to 4.00) and people with positive attitudes (mean, 4.33 to 7.00). Two-way ANOVAs (format × immigration attitude: negative vs. positive) revealed main effects of immigration attitudes on perceived uncertainty, trust in numbers, and trust in the source, but no interaction ef- Control Numerical Numerical Numerical Verbal Verbal Verbal Mixed fects. People with positive attitudes toward immigration perceived +/− Range Range Cue Explicit Implicit Numerical/ less uncertainty around the numbers [F = 6.15, P = 0.01; Estimate No Estimate Verbal (1, 1040) η = 0.01], reported more trust in the numbers [F = 33.39, (1, 1039) Trust in source p P < 0.001; η = 0.03], and more trust in the source [F = (1, 1040) 45.22, P < 0.001; η = 0.04] than people with negative attitudes toward immigration. However, we found no significant interaction effects between attitudes and communication format. To assess the robustness of these results, we also conducted a series of Control Numerical Numerical Numerical Verbal Verbal Verbal Mixed +/− Range Range Cue Explicit Implicit Numerical/ Perceived uncertainty Estimate No Estimate Verbal Fig. 3. The results of experiment 3: Means per condition for perceived uncertainty (A), trust in numbers (B), and trust in the source (C). The error bars represent 95% CIs around the means, and jitter represents the distri- bution of the underlying data. servants responsible for the migration statistics” [F = (4, 1045) 1.19, P = 0.31]. Control Numerical Numerical Verbal Verbal In summary, we found that whereas both numeric formats and +/− Range Explicit Cue the explicit verbal statement did communicate uncertainty Trust in number around the net migration number, only the explicit verbal state- ment decreased perceived reliability of the number, and no 7 format decreased participants’ perceptions of trustworthiness of the source. This pattern of results is broadly consistent with our preregistered hypotheses based on our previous three studies (see SI Appendix for more details). We also asked people to what extent they thought that 1) the conclusions based on the number, 2) the news article they just read, 3) journalists who write news articles such as this one, and Control Numerical Numerical Verbal Verbal 4) government statistics in general were trustworthy. +/− Range Explicit Cue This revealed that format did affect trust in the conclusions Trust in source [one-way ANOVA of format: F = 3.53, P = 0.007; (4, 1045) η = 0.01]. This was due to the explicit verbal statement of un- 7 certainty leading to lower trust in the conclusions than the word “around” (post hoc paired comparisons: M = 3.51 vs. 3.96, M = −0.45, 95% CI [−0.84; −0.05], d = 0.30). All uncertainty diff communication formats did not differ significantly from the control condition (M = 3.90, SD = 1.46). Second, there appeared to be a small effect of format on trust in the news article itself (one-way ANOVA of format: F = (4, 1042) Control Numerical Numerical Verbal Verbal +/− Range Explicit Cue 2.43, P = 0.046; η = 0.01), but post hoc paired comparisons did not show significant differences between formats—it was again Fig. 4. The results of experiment 4: Means per condition for perceived mainly driven by a decrease for the explicit verbal statement uncertainty (A), trust in numbers (B), and trust in the source (C). The error compared to the control condition (M = 3.66 vs. 4.04, M = diff bars represent 95% CIs around the means, and the jitter represents the −0.38, 95% CI [−0.77; 0.01], d = 0.26). distribution of the underlying data. van der Bles et al. PNAS | April 7, 2020 | vol. 117 | no. 14 | 7677 Score Mean score Mean Score Score Mean score Mean score PSYCHOLOGICAL AND COGNITIVE SCIENCES hierarchical linear regressions with a continuous interaction trustworthy it was, and how trustworthy and competent the term. These analyses produced the same findings and are statisticians responsible for producing the figure were, and how reported in the SI Appendix. Overall, how people responded to trustworthy they thought the journalist responsible for producing uncertainty communication was not affected by their prior at- the article was. The results are presented in Fig. 7. titudes toward immigration. The results of this field experiment showed that, in line with the laboratory experiments, people perceived the number to be Internal Metaanalysis. To consolidate all of our main findings and more uncertain when numerical uncertainty had been commu- to shed further light on the psychological effects of communi- nicated, compared to no uncertainty communication in the cating uncertainty, we conducted a random-effects metaanalysis across all four studies for each of our key dependent variables. To ensure that the results were comparable, we only included the formats that were consistently tested across all four experiments. For ease of interpretation, we contrast “no uncertainty” (control condition) vs. “uncertainty” communication, differentiating only between “verbal” (explicit verbal statement) vs. “numeric” (numeric range with point estimate) uncertainty communication as separate subgroups. Results are presented in Fig. 5. Overall, the communication of uncertainty in itself had a large effect on perceived uncertainty (d = 0.65; 95% CI [0.42; 0.87]), with the effect of verbal uncertainty (d = 0.88; 95% CI [0.62; 1.14]) being over twice that of numeric uncertainty (d = 0.43; 95% CI [0.33; 0.52]). Importantly, the communication of un- certainty did lead to a significant overall decrease in perceived reliability of the numbers (d = −0.34; 95% CI [−0.16; −0.53]). Although relatively small and nonsignificant across some of the studies, the weighted effect of providing numeric uncertainty on trust in numbers was also negative and significant (d = −0.15; 95% CI [−0.05; −0.24]). However, much of the overall effect seems to stem from verbal uncertainty, as the negative effect of verbal uncertainty on trust in numbers was much more sub- stantial (d = −0.55; 95% CI [−0.35; −0.74]). Last, although the weighted effect of the communication of uncertainty across studies did also significantly and negatively influence perceived trustworthiness of the source (d = −0.12; 95% CI [−0.03; −0.22]), the size of the effect is similarly small and seems to be driven by verbal uncertainty (d = −0.21; 95% CI [−0.12; −0.31]) rather than numeric uncertainty (d = −0.03; 95% CI [−0.03; 0.06]). Field Experiment on the BBC News Website. Finally, we assessed to what extent our findings would generalize beyond the context of an online laboratory experiment to a real-world setting. We there- fore engaged in a unique experiment on the live BBC News website to test the effects of communicating uncertainty in an online news article about the United Kingdom’s labor market statistics, which are released monthly by the Office for National Statistics. After a pilot experiment using a BBC News article about the UK labor market in September 2019, which is reported in the SI Appendix, we conducted an experiment with an online news ar- ticle about the UK labor market on October 15, 2019. Readers of the live BBC News website were randomly shown one of three versions of the news article (Fig. 6). The first figure mentioned in this article was the unemployment rate, which “.. . unexpectedly rose to 3.9% in the June-to-August period from 3.8%, after the number of people in work unexpectedly fell by 56,000, official figures showed.” Readers were either shown this target figure without any uncertainty mentioned, as is common in all news reporting (including the BBC); with a verbal uncertainty cue (“.. . rose to an estimated 3.9%”), as is sometimes used in BBC News reporting; or with a numeric range and verbal cue [“.. . rose to an estimated 3.9% (between 3.7% and 4.1%)”], which is un- common in news reporting. All other figures mentioned in the article were reported without uncertainty. After the first para- graph of the news article, which contained the figure of interest, readers were invited to take part in a short study about this ar- ticle. As our survey had to be brief, we only included our key measures: After asking participants to rate their current emo- tional state (affect), we asked them how certain or uncertain they Fig. 5. Random-effects metaanalysis. Perceived uncertainty (A), trust in thought the unemployment rate figure in the story was, how numbers (B), and trust in the source (C). 7678 | www.pnas.org/cgi/doi/10.1073/pnas.1913678117 van der Bles et al. trustworthiness of the journalist [F = 0.86, P = 0.42]. (2, 1526) Participants’ judgments of the competence and trustworthiness of the statisticians were highly correlated (r = 0.80, P < 0.001), and on the high end of the scale (M = 5.44, SD = 1.41, and M = 5.28, SD = 1.55, respectively, out of seven); participants’ rating of the trustworthiness of the journalist was slightly lower (M = 4.61, SD = 1.54). These results suggest that communicating uncertainty to the participants of this field study, did not affect their (already positive) views of the trustworthiness and competence of the people involved in producing and reporting unemployment figures. Discussion Centuries of human thinking about uncertainty among many leaders, journalists, scientists, and policymakers boil down to a simple and powerful intuition: “No one likes uncertainty” (1, 6, 7, 27). It is therefore often assumed that communicating un- certainty transparently will decrease public trust in science (1, 7). In this program of research, we set out to investigate whether such claims have any empirical basis. We did this by communicating epistemic uncertainty around basic facts and numbers and by systematically varying 1) the topic, 2) the magnitude of the un- certainty, and 3) the format and context through which uncertainty was communicated. We assessed the effects of uncertainty on relevant outcome measures, including cognition and trust. Perceived uncertainty Fig. 6. Image of the BBC News article that was used in experiment 5 (nu- Control Numerical Verbal merical condition: including a numeric range). Reprinted with permission from BBC News. B Trust in number control condition. An ANOVA showed a significant main effect of uncertainty communication on perceived uncertainty [F = (2, 1526) 4.67, P = 0.01; η = 0.006]. Participants who read the version of the news article with a numeric range around the unemployment rate figure perceived the figure to be more uncertain than people in the control condition (M = 3.56 vs. 3.31, M = 0.25, 95% CI [0.06; diff 0.44], d = 0.19). Participants who read the version of the news ar- ticle with the verbal cue scored in between the numerical and control conditions, not significantly different from either (M = 3.41, Control Numerical Verbal SD = 1.39). This finding suggests that participants did seem to have Trust in source noticed the uncertainty that was communicated. Uncertainty communication, however, did not affect partici- pants’ trust in the number [F = 1.20, P = 0.30], nor trust in (2, 1526) the source, in this case, the statisticians responsible for producing the figures [F = 1.24, P = 0.29]. These findings comple- (2, 1525) ment the results from our laboratory experiments, which showed 4 that a verbal cue such as “estimated” did not seem to commu- nicate uncertainty to people and did not affect their trust in numbers or the source (as found in experiments 3 and 4). In this field experiment, we again found communicating uncertainty as a Control Numerical Verbal numeric range did not affect people’s trust in the source, and it also did not affect trust in the number. Fig. 7. The results of field experiment 5: Means per condition for perceived In addition, the results showed no significant effects of un- uncertainty (A), trust in numbers (B), and trust in the source (C). The error certainty communication on affect [F = 0.44, P = 0.65], (2, 1519) bars represent 95% CIs around the means, and the jitter represents the competence of the source [F = 0.61, P = 0.54], and distribution of the underlying data. (2, 1525) van der Bles et al. PNAS | April 7, 2020 | vol. 117 | no. 14 | 7679 Score Score Score PSYCHOLOGICAL AND COGNITIVE SCIENCES Overall, we found little evidence to suggest that communi- reliability either, which is an important finding in itself and cating numerical uncertainty about measurable facts and num- warrants further research. bers backfires or elicits psychological reactance. Across five high- Accordingly, based on these results, we therefore recommend powered studies and an internal metaanalysis, we show that that the communication of uncertainty around basic facts and people do recognize and perceive uncertainty when communi- numbers in the media is best conveyed through numerical ranges cated around point estimates, both verbally and numerically with a point estimate. This format in particular did not seem to (except when only words such as “estimated” or “about” are used significantly influence (i.e., reduce) perceived trust and reliability to imply uncertainty). In addition, uncertainty did not seem to in either the number or the source of uncertainty. In addition, we influence their affective reaction (SI Appendix), and although the draw attention to the fact that using the word “estimate” or in- provision of uncertainty in general did slightly decrease people’s creasing the magnitude of the confidence interval did not seem trust in and perceived reliability of the numbers, this effect to alter people’s perception of uncertainty, which points to the emerged for explicit verbal uncertainty in particular. need to better contextualize the degree of uncertainty for people. Our research offers an important bridge between producers of Last, it is notable that we find little evidence for the motivated statistics, communicators, and their audiences. For example, cognition of uncertainty (35). For example, even around more statisticians or scientists could argue that because most numeric contested topics, such as global warming and immigration, al- estimates are never completely certain, presenting uncertainty though main effects were observed for people’s prior attitude around the number offers more precise information and should toward the issue, there was no significant interaction with the therefore foster more trustworthiness, not less. However, if a communication of uncertainty. At the very least, this suggests general audience had not considered that there might be any that motivated interpretations of uncertainty do not always oc- uncertainty around a number in the first place (e.g., around cur. At the same time, we must acknowledge several limitations unemployment), then from a purely normative point of view of our program of research. people’s reaction to uncertainty in our studies is not entirely First, we recognize that people are known to struggle with inappropriate: By providing clear variability around estimates, it psychological uncertainty about the future (44, 45), perhaps more is reasonable for people to adjust their level of trust in the so than uncertainty about measurable facts and numbers, an area numbers themselves. In a similar vein, one might argue that it is previously neglected, and thus the focus of the current work. The difficult for people to appraise the trustworthiness of a number context of our research was also limited, culturally, to the United without having access to the methodology through which the Kingdom, and more contested examples for this population (e.g., estimate is derived. However, from a social scientific standpoint, around the United Kingdom’s political exit from the European we recognize that people are frequently exposed to numbers in Union) may have elicited different results. Moreover, while we the news without necessarily having access to additional in- conceptually replicated our results across multiple studies and formation, for example, about the quality of the underlying evi- platforms—including a preregistered national sample—we did not dence (or indirect uncertainty). So how do people actually arrive investigate uncertainty around more emotionally charged topics in at a judgment as to what numbers are reliable and trustworthy in this study, such as uncertainty about personal health outcomes the face of uncertainty? Although we did not set out explicitly to (e.g., cancer), nor manipulated contestedness as an experimental investigate the mechanism by which people adjust their judg- factor. Indeed, there may be other circumstances (not examined ments in response to uncertainty, an exploratory mediation here) where a significant degree of uncertainty could elicit strong analysis on the nationally representative sample (experiment 4) emotional reactions. Finally, we attempted to improve the eco- clearly suggests that the main effect of uncertainty communica- logical and external validity of our manipulations by engaging in a tion (uncertainty vs. no uncertainty) on trustworthiness is fully real-world experiment on the live BBC News website. Although mediated by people’s perception of the uncertainty (see SI Ap- findings corroborated what we observed in controlled laboratory pendix for mediation analyses). In other words, this suggests that settings, the BBC study necessarily relied on a somewhat skewed the more uncertain people perceive the numbers to be, the less and self-selected sample. In addition, although we generally relied reliable and trustworthy they find them. The current results help on large and diverse samples, and our main effects were suffi- inform theoretical predictions about how people might respond ciently powered, we may not have had sufficient power to detect to direct uncertainty about numbers, and we encourage future very small effects in all post hoc comparisons. Sensitivity analyses research to further investigate potential mechanisms as well as showed, however, that given the sample sizes of experiments 3 and how people might respond to indirect uncertainty, such as addi- 4 (and assuming α = 0.05 and power of 0.80), we should have been tional information about the quality of the underlying evidence. able to detect small effects in these studies (f = 0.101, d = 0.20; In sum, prior research has investigated whether the provision and f = 0.107, d = 0.21, respectively). The smallest effects of in- of uncertainty can help signal transparency and honesty on be- terest reported in our paper are broadly beyond those thresholds half of the communicator, or—in contrast—whether communi- (e.g., d = 0.26 to 0.72). cating uncertainty decreases trust and signals incompetence (9, Nonetheless, even considering all of these boundary conditions, 15, 17, 36). By and large, our findings illustrate that the provision our results help inform and challenge strongly held—and often of numerical uncertainty—in particular as a numeric range— nonempirical—assumptions across domains about how the public does not substantially decrease trust in either the numbers or the will react to the communication of uncertainty about basic science, source of the message. Verbal quantifiers of uncertainty, how- facts, and numbers (1, 7). A key challenge to maintaining public ever, do seem to decrease both perceived reliability of the trust in science is for communicators to be honest and transparent numbers as well as the perceived trustworthiness of the source. about the limitations of our current state of knowledge. The high These findings were robust across topics (both contested and degree of consistency in our results, across topics, magnitudes of noncontested), mode of communication, and magnitude of un- uncertainty, and communication formats suggest that people “can certainty. More generally, the strong negative effects of verbal handle the truth.” However, if we want to effectively convey un- uncertainty appear consistent with prior findings that people are certainty about pressing issues, such as rising sea levels, the averse to more ambiguous statements (27, 43). As such, we hy- number of tigers left in India, the state of the economy, or how pothesize that the communication of numerical uncertainty may many people turn out to presidential elections; natural scientists, offer a degree of precision that reduces people’s tendency to statisticians, and social scientists should work together to evaluate view the admission of uncertainty as a sign of incompetence (1, 9, how to best present scientific uncertainty in an open and trans- 36). On the other hand, across all studies, the communication parent manner. As such, our findings can provide valuable guid- of uncertainty never significantly increased perceived trust or ance to scientists, communicators, practitioners, and policymakers 7680 | www.pnas.org/cgi/doi/10.1073/pnas.1913678117 van der Bles et al. the verbal condition (somewhat higher or lower). These numbers are based on alike, who are all united by a common interest in how to effectively reports by the UK Office for National Statistics (49), the International Union for communicate the truth in a so-called posttruth world. Conservation of Nature Red list (50), and the Intergovernmental Panel on Cli- mate Change (51), respectively. Materials and Methods Measures. After reading the text, participants first reported how the in- The survey experiments were completed in a web browser and took ∼12 min formation made them feel on a standard feeling thermometer from 0 = to complete. For experiments 1 to 3, we recruited participants on the plat- negative/unhappy to 10 = positive/happy (results are reported in the SI form Prolific. Prolific has been found to be similar to Amazon Mechanical Appendix). They were then asked to recall what number was reported in the Turk in terms of data quality, and better suited to recruit UK-based partic- text (open question), and whether they remembered any uncertainty being ipants (46, 47). Participants were paid £1.20 for their participation and were implied around this number (yes, no, don’t know, don’t remember). These not allowed to participate in more than one experiment. For experiment 4, questions served as manipulation checks and to increase the salience of the we used Qualtrics Panels to recruit a sample that was nationally represen- target number. The open text responses showed that most participants were tative of the United Kingdom population in terms of gender, age, and re- able to either correctly recall the target number (in experiment 2, where gion in which the participants lived. For experiment 5, a field study, we we coded all responses: 54.4%), or give a sensible estimate of the target collaborated with BBC News and recruited visitors of the BBC News website number (experiment 2: 30.2%), indicating that generally participants un- and app. This survey took ∼2 min to complete. Ethical approval for this re- derstood what we meant by “this number” in the questions that followed. search was granted by the Cambridge Psychology Research Ethics Committee Next, our key dependent variables were assessed: perceived uncertainty of (experiments 1, 2, and 5) and the Department of Psychology Ethics Com- the number (average of 2 items, “To what extent do you think that this mittee (experiments 3 and 4) of the University of Cambridge. All participants number is certain or uncertain?”:1 = very certain to 7 = very uncertain; gave informed consent before participation and received detailed debrief- “How much uncertainty do you think there is about this number?”: 10-point ing information afterward. SI Appendix includes tables with an overview of slider: not at all uncertain to very uncertain; r = 0.63), trust in the number the characteristics of the participants for each experiment (SI Appendix, (modeled after ref. 9; average of 2 items, “To what extent do you think this Table S1) and per condition in each experiment (SI Appendix, Tables S2–S6), number is reliable [trustworthy]?”: 7-point scale from 1 = not at all to 7 = which show that the experimental groups were balanced in terms of par- very reliable [trustworthy]; r = 0.88), and trust in the source (“To what extent ticipants’ age, gender, education level, and numeracy. do you think the writers of this report are trustworthy?”: 7-point scale from 1 = not at all to 7 = very trustworthy). In addition, we also asked people to Experiment 1. how uncertain the number made them feel (10-point slider, 1 = not at all to Sample and design. In experiment 1, we used a between-subjects design to test 10 = very uncertain; results reported in SI Appendix). After these dependent three forms of uncertainty communication (numeric vs. verbal vs. control) about variables, a series of unrelated variables were assessed (for more in- three topics (tigers vs. unemployment vs. climate change). Based on a priori formation, see SI Appendix). The experiment finished with questions about power calculations, which indicated we would need 1,075 people for 90% demographic information and a detailed debrief. power to detect a small (interaction) effect (f = 0.12) when α was set at 0.05, we decided to recruit 1,125 participants (125 per cell of the design; we did this for Experiment 2. experiment 2 and 3 as well). Three of these participants indicated to be below Sample and design. Experiment 2 followed on from experiment 1. Instead of 18 y of age and were excluded from further analyses. The sample thus con- varying the topics, however, we were interested in the effect of different sisted of n = 1,122 people [769 women (68.5%); average age, 37.72; SD, 12.12; magnitudes of uncertainty. This experiment therefore consisted of a control range, 18 to 72]. Compared to the UK population, this sample was relatively condition (no uncertainty communicated) plus a 2 (numeric vs. verbal un- highly educated. Organisation for Economic Co-operation and Development certainty communication) × 3 (lower vs. original vs. higher magnitude of data show that 18.8% of the 24- to 65-y-olds in the United Kingdom attained uncertainty) factorial design; so a total of seven conditions, to which par- primary and middle school education, 35.4% upper secondary education ticipants were randomly allocated. Based on a priori power calculations, (General Certificate of Secondary Education [GCSE] and A-levels), and 45.7% which indicated we would need 752 participants for 90% power to detect a attainted tertiary education (bachelor’s, master’s, PhD, etc.) (48). In our sample, small (f = 0.13) interaction effect between format and magnitude when α 1.6% indicated to have no educational qualifications, 38.5% indicated to have was set at 0.05, we recruited 877 participants from Prolific (∼125 per cell for attained upper secondary education, and 59.6% indicated to have attained the seven-cell design). The sample consisted of 582 women and 292 men tertiary education. On average, political orientation of the sample was slightly (average age, 34.68; SD, 12.02; range, 18 to 80). Similar to experiment 1, leaning toward liberal (M = 3.49, SD = 1.42, on a scale from 1 = very liberal to this sample was relatively highly educated compared to the UK population: 7 = very conservative). 1.1% indicated to have no educational qualifications, 39% indicated to Treatment and procedure. After agreeing to participate, participants were have attained upper secondary education, and 59.8% indicated to have asked several questions about their beliefs related to the three topics: about attained tertiary education. On average, political orientation of the sample the conservation of endangered animals, about the present state of the was slightly liberal (M = 3.40; SD = 1.42). country and economy, and about climate change (for more information on all Treatment and procedure. All participants read the same text as in study 1 about measures, see SI Appendix). After this, participants were randomly allocated unemployment, with either no uncertainty communicated, or uncertainty to be presented with one of nine texts. For example, the text about un- communicated numerically or verbally. The different magnitudes were either employment read as follows: the original magnitude that was communicated in experiment 1, which was a numerical range of 1,413,000 to 1,555,000 (95% CI around the point estimate Recently, an official report came out with new information about the of 1,484,000 unemployed people) or the sentence stating that the number unemployment rate in the United Kingdom. This report stated that be- could be “somewhat higher or lower.” However, in addition, lower un- tween April and June 2017, government statistics showed that an esti- certainty was communicated as a range of minimum 1,448,500 to maximum mated 1,484,000 people in the UK were unemployed. 1,519,500 (a 68% CI around the point estimate, which is a range that is half as Participants in the control condition only read about this central estimate, wide as the original) or through the wording “slightly higher or lower” (verbal without any information about uncertainty. For participants in the numeric condition). Higher uncertainty was communicated as a range of minimum 1,342,000 to maximum 1,626,000 (99.99% CI, which is a range that is twice as uncertainty communication condition, the exact same sentence finished with a wide as the original) or through the wording “a lot higher or lower.” Before numeric range: “.. .unemployed (minimum 1,413,000 to maximum 1,555,000).” reading this text, participants were asked some questions about their beliefs For participants in the verbal uncertainty communication condition, an extra about the state of the country and economy, and afterward, they were asked sentence was added to the text: “The report states that there is some un- the same exact questions as in study 1 (SI Appendix). certainty around this estimate, it could be somewhat higher or lower.” The control text about tigers reported that “an official report stated that in 2015 an estimated 2,226 tigers were left in India.” In the numeric uncertainty commu- Experiment 3. nication condition, a range of minimum 1,945 to maximum 2,491 was added, Sample and design. In experiment 3, we aimed to test various other numeric and in the verbal uncertainty communication condition, the exact same sen- and verbal uncertainty communication formats, still in the context of un- tence was used as in the unemployment condition. The text about climate employment for consistency using a relatively high magnitude of uncertainty. change reported that “an official report stated that between 1880 and 2012, This study had eight conditions, and we recruited n = 1,200 participants from the earth’s average global surface temperature has increased by an estimated Prolific, based on power calculations that indicated this would give us 90% 0.85°C.” In the numeric uncertainty condition, a range of minimum 0.65 to power to detect a small (f = 0.125) effect when α was set at 0.05. The sample maximum 1.06 was added, and once again the exact same sentence was used in consisted of 806 women and 388 men (average age, 36.65; SD, 11.98; range, van der Bles et al. PNAS | April 7, 2020 | vol. 117 | no. 14 | 7681 PSYCHOLOGICAL AND COGNITIVE SCIENCES 18 to 85). Just as in experiments 1 and 2, the sample was relatively highly Measures. Before reading the text, participants answered demographic ques- educated compared to the UK population (no educational qualifications, tions and questions about their beliefs about the state of society and the 1.3%; upper secondary, 34.7%; tertiary education, 63.7%) and on average economy and their attitudes toward migration [with three items from the European Social Survey (53): “Would you say it is generally good for the UK’s slightly leaning toward a liberal political orientation (M = 3.40; SD = 1.38). economy that people come to live here from other countries?”:1 = very bad After answering questions about their beliefs about the state of the country for the economy to 7 = very good for the economy; “Would you say that the and economy, people read a version of the following text, which was designed UK’s cultural life is generally undermined or enriched by people coming to live to read more like a short news article to increase ecological validity: here from other countries?”:1 = cultural life undermined to 7 = cultural life UK unemployment drops enriched; “Is the UK made a worse or a better place to live by people coming to live here from other countries?”:1 = worse place to live to 7 = better place Official figures from the first quarter of 2018 show that UK unemploy- to live; α = 0.91]. After reading the text, participants answered the same ment fell by 116,000 compared with the same period last year. questions as in experiments 1 and 2. Perceived trustworthiness of the source This puts the total number of people who are unemployed at 1.42 was assessed with the item, “To what extent do you think the civil servants million. who are responsible for these migration figures are trustworthy?” on a scale from 1 = not at all to 7 = very trustworthy. In addition, we asked people to The number of those in work increased and wage growth improved over what extent they thought the conclusions based on the number; the news the same period. However, weak incomes have been a problem for a article they just read; and journalists who write articles such as the one they decade. “It will take a long period of wages rising above the rate of read were trustworthy, and to what extent government statistics in general inflation for people to feel significantly better off,” one economics were reliable (on scales from 1 = not at all to 7 = very trustworthy [reliable]). commentator is quoted as saying. Experiment 5: Field Experiment with BBC News. This version served as the control condition. We tested three numeric Sample and design. For this field experiment, we worked with BBC News and uncertainty communication formats, three verbal formats, and a mixed the BBC’s Head of Statistics. After gaining experience with the process of numeric/verbal format (Table 1). running a field experiment in this context during a Pilot Study in September Measures. After reading the text, participants were asked the same questions 2019 (SI Appendix), we conducted the experiment on October 15, 2019, as in experiments 1 and 2, except now specified for each number (the fall in using BBC News Online’s coverage of the October Labor Market Release unemployment and the total number of unemployed people) for the recall from the UK Office for National Statistics. After the labor market figures question, their perception of uncertainty around the numbers (α = 0.80) and were released, we worked with the relevant journalists and the Head of trust in the numbers (α = 0.92). For the analyses, answers were averaged Statistics to select a target figure to communicate uncertainty before the across both numbers given that there were no meaningful or significant news article was published on the website. The journalists were responsible differences between the two. Trust in the source was assessed with the item for the content of the news article. The target figure we selected was the UK “To what extent do you think the civil servants who are responsible for these unemployment rate, which was the first figure mentioned in the news story unemployment figures are trustworthy?” on a scale from 1 = not at all to 7 = (Fig. 6). The field experiment had three conditions: visitors of the website very trustworthy. In addition, we asked people to what extent they thought were randomly shown a version of the news article in which the target journalists who write articles such as the one they read were trustworthy, figure was presented without any uncertainty (“... rose to 3.9%”); with a and to what extent they thought government statistics in general were re- verbal uncertainty cue (“.. . rose to an estimated 3.9%”); or with a numeric liable (on scales from 1 = not at all to 7 = very trustworthy [reliable]). range and verbal cue [“... rose to an estimated 3.9% (between 3.7 and 4.1%)”]. At the bottom of the first paragraph of the news article, readers Experiment 4. were invited to “Click here to take part in a short study about this article run Sample and design. Experiment 4 was preregistered at aspredicted.org (https:// by the University of Cambridge.” aspredicted.org/blind.php?x=d3xu67). We recruited 1,050 adults who lived in BBC News website visitors were able to participate in the study for about the United Kingdom to participate in this study via Qualtrics Panels, based 24 h. During that time, 2,462 people clicked on the survey link, which took on power calculations that indicated we would need 995 participants to people to the starting page of the online survey with information about the have 90% power to detect a small (f = 0.125) effect when α was set at 0.05. study and informed consent. The survey was completed by 1,700 people (18 This sample was nationally representative of the general UK population in of whom completed the dependent variables but not demographics): 549 terms of age, gender, and geography quotas (51% female; mean age, 45.34 y; people in the control condition, 557 in the numeric condition, and 594 in the SD, 16.47; age range, 18 to 86). In this sample, 8.9% of the participants had verbal condition. A technical issue that was created when the journalistic no educational qualifications, 44.8% had attained upper secondary educa- team updated the story after its first release resulted in participants in both tion, and 46.1% had tertiary education. On average, the sample was again experimental conditions also being shown the control condition version of slightly leaning liberal (M = 3.74; SD = 1.51). the story, without any uncertainty mentioned, between 10:00 AM and 10:49 Treatment and procedure. We aimed to test whether we would find the same AM UK time. We therefore had to exclude all participants in the experi- results when communicating uncertainty around a more contentious topic, so mental conditions who participated between in that time frame, which were we presented people with a text about migration statistics based on a BBC 69 participants in the numerical and 94 in the verbal condition. We also News article of these Office of National Statistics figures (52): excluded five participants who reported to be below 18 y of age, and one outlier who reported being 114 y old (which was extremely unlikely). The Migration figures: EU migration still adding to UK population final sample consisted of 1,531 people: 520 participants in the control con- Official figures from last year show that there were 101,000 more people dition, 463 in the numerical condition, and 470 in the verbal condition. We coming to the UK from the EU than leaving in 2017. This is the lowest EU had no control over the exact number of people that would participate in net migration figure since 2013, but it means that EU migrants are still this field study, so we conducted a sensitivity analysis to compute the effect adding to the UK population. size that we should be able to detect with 1,531 participants in three groups, α = 0.05 and 90% power: which is a small effect, f = 0.09. There were 1,131 Net migration is the difference between the number of people coming men (73.9%) and 344 women (22.5%) who participated, with an average to live in the UK for at least 12 months and those emigrating. The 2017 age of 44.82 (SD, 15.29; range, 18 to 86). The sample was relatively highly overall net migration figure (both from the EU and non-EU countries) is educated: 30.5% indicated to have obtained a higher degree (MSc, PhD, or also down, from record highs in 2015 and early 2016. equivalent), 43.6% a bachelor’s degree, 22.5% school (GCSE, A-level, or equivalent), and 1% indicated to have not completed formal education. However, “The figures show that the government remains a long way Measures. After reading information about the study and providing informed off from meeting its objective to cut overall net migration, EU and non- consent, participants first answered a question about their current affec- EU, to the tens of thousands,” one Home Affairs correspondent is tive state, “How does the information you just read make you feel?”: quoted as saying. on a feeling thermometer from 0 = negative/unhappy to 10 = positive/ This experiment had five conditions (Table 1): Besides the control condi- happy, and subsequently a comprehension question (SI Appendix). Next, we tion (above, no uncertainty), uncertainty was communicated numerically assessed perceived uncertainty (“How certain or uncertain do you think the with a range after the point estimate, or via “+/− two standard errors”;or unemployment rate figure in the story is?”: on a scale from 1 = very certain verbally using the word “around” before the estimate of 101,000, or with an to 7 = very uncertain), trust in the number (“How trustworthy do you think explicit verbal statement. the unemployment rate figure in the story is?”:1 = not at all trustworthy to 7682 | www.pnas.org/cgi/doi/10.1073/pnas.1913678117 van der Bles et al. 7 = very trustworthy), trust in the source (“How trustworthy do you think ACKNOWLEDGMENTS. Funding was provided for these studies by the Nuffield Foundation (Grant OSP/43227). The Foundation has funded this the statisticians responsible for producing the figure are?”:1 = not at all project, but the views expressed are those of the authors and not necessarily trustworthy to 7 = very trustworthy), competence of the source (“How the Foundation. Prof. Sir David Spiegelhalter and Dr. Alexandra Freeman competent do you think the statisticians responsible for producing the fig- and the Winton Centre for Risk and Evidence Communication are supported ure are?”:1 = not at all competent to 7 = very competent), and trust in the by a donation from the David and Claudia Harding Foundation. We also journalistic source (“How trustworthy do you think the journalist responsible thank Robert Cuffe, the BBC Business team, the BBC Home Affairs team, and for writing the story is?”:1 = not at all trustworthy to 7 = very trustworthy). the BBC Digital team for their help and support in conducting the field experiment, but the views expressed are those of the authors and not The questionnaire finished with asking participants for their age, gender, necessarily those of the BBC. We also thank Vivien Chopurian for her and the highest level of education they had completed. assistance with study preparations and analyses, and a diverse advisory panel (John Aston, Robert Cuffe, Ed Humpherson, Onora O’Neill, Amy Sippitt, and Data Availability. The datasets collected and analyzed in the reported studies Elke Weber) who supported the directions of this research. This research are available on the Open Science Framework, https://osf.io/mt6s7/ (DOI:10.17605/ could not have been undertaken without the assistance of all the partici- OSF.IO/MT6S7). pants, whom we also thank. 1. B. Fischhoff, Communicating uncertainty: Fulfilling the duty to inform. Issues Sci. 28. D. A. Clark, Verbal uncertainty expressions: A critical review of two decades of re- Technol. 28,63–70 (2012). search. Curr. Psychol. 9, 203–235 (1990). 2. S. van der Linden, R. E. Löfstedt, Eds., Risk and Uncertainty in a Post-Truth Society 29. M. J. Druzdzel, Verbal Uncertainty Expressions: Literature Review (Carnegie Mellon (Routledge, 2019). University, Pittsburgh, PA, 1989). 3. Edelman, 2018 Edelman Trust Barometer. https://www.edelman.com/research/2018- 30. D. V. Budescu, T. S. Wallsten, Consistency in interpretation of probabilistic phrases. edelman-trust-barometer. Accessed 5 March 2020. Organ. Behav. Hum. Decis. Process. 36, 391–405 (1985). 4. Edelman, 2017 Edelman Trust Barometer. https://www.edelman.com/research/2017- 31. D. V. Budescu, H.-H. Por, S. B. Broomell, Effective communication of uncertainty in the edelman-trust-barometer. Accessed 5 March 2020. IPCC reports. Clim. Change 113, 181–200 (2012). 5. Pew Research Center, Beyond distrust: How Americans view their government. https:// 32. S. C. Jenkins, A. J. L. Harris, R. M. Lark, Understanding “unlikely (20% likelihood)” or www.people-press.org/2015/11/23/beyond-distrust-how-americans-view-their- “20% likelihood (unlikely)” outcomes: The robustness of the extremity effect. J. Behav. government/. Accessed 5 March 2020. Decis. Making 31, 572–586 (2018). 6. D. Ariely, Predictably Irrational: The Hidden Forces that Shape Our Decisions 33. B. B. Johnson, P. Slovic, Lay views on uncertainty in environmental health risk as- (HarperCollins, 2008). sessment. J. Risk Res. 1, 261–279 (1998). 7. National Academies of Sciences, Engineering, and Medicine, Communicating Science 34. N. F. Dieckmann, E. Peters, R. Gregory, At home on the range? Lay interpretations of Effectively: A Research Agenda (National Academies Press, 2017). numerical uncertainty ranges. Risk Anal. 35, 1281–1295 (2015). 8. A. Gustafson, R. E. Rice, The effects of uncertainty frames in three science commu- 35. N. F. Dieckmann, R. Gregory, E. Peters, R. Hartman, Seeing what you want to see: How nication topics. Sci. Commun. 41, 679–706 (2019). imprecise uncertainty ranges Enhance motivated reasoning. Risk Anal. 37, 471–486 9. B. B. Johnson, P. Slovic, Presenting uncertainty in health risk assessment: Initial studies (2017). of its effects on risk perception and trust. Risk Anal. 15, 485–494 (1995). 36. B. B. Johnson, Further notes on public response to uncertainty in risks and science. 10. S. Dunwoody, F. Hendriks, L. Massarani, H. P. Peters, “How journalists deal with sci- Risk Anal. 23, 781–789 (2003). entific uncertainty and what that means for the audience” in 15th International 37. P. K. J. Han et al., Communication of uncertainty regarding individualized cancer risk Public Communication of Science and Technology Conference (PCST 2018) (Public estimates: Effects and influential factors. Med. Decis. Making 31, 354–366 (2011). Communication of Science and Technology, 2018), pp. 3–6. 38. I. M. Lipkus, W. M. P. Klein, B. K. Rimer, Communicating breast cancer risks to women 11. M. Lehmkuhl, H. P. Peters, Constructing (un-)certainty: An exploration of journalistic using different formats. Cancer Epidemiol. Biomarkers Prev. 10, 895–898 (2001). decision-making in the reporting of neuroscience. Public Underst. Sci. 25,909–926 (2016). 39. S. T. Fiske, C. Dupree, Gaining trust as well as respect in communicating to motivated au- 12. C. H. Braddock, 3rd, K. A. Edwards, N. M. Hasenberg, T. L. Laidley, W. Levinson, In- diences about science topics. Proc. Natl. Acad. Sci. U.S.A. 111 (suppl. 4), 13593–13597 (2014). formed decision making in outpatient practice: Time to get back to basics. JAMA 282, 40. O. O’Neill, Reith lectures 2002: A question of trust, lecture 4: Trust and transparency. 2313–2320 (1999). http://downloads.bbc.co.uk/rmhttp/radio4/transcripts/20020427_reith.pdf. Accessed 5 13. M. C. Politi, P. K. J. Han, N. F. Col, Communicating the uncertainty of harms and March 2020. benefits of medical interventions. Med. Decis. Making 27, 681–695 (2007). 41. S. Fischer, R. Fitzegerald, W. Poortinga, “Climate change” in British Social Attitudes: 14. M. C. Politi, M. A. Clark, H. Ombao, D. Dizon, G. Elwyn, Communicating uncertainty The 35th Report, D. Philips, J. Curtice, M. Philips, J. Perry, Eds. (The National Centre for can lead to less decision satisfaction: A necessary cost of involving patients in shared Social Research, 2018), pp. 146–171. decision making? Health Expect. 14,84–91 (2011). 42. S. Blinder, L. Richards, Briefing: UK public opinion towards immigration: Overall at- 15. M. Osman, A. J. Heath, R. Löfstedt, The problems of increasing transparency on un- titudes and level of concern. https://migrationobservatory.ox.ac.uk/resources/briefings/ certainty. Public Underst. Sci. 27, 131–138 (2018). uk-public-opinion-toward-immigration-overall-attitudes-and-level-of-concern/. Accessed 16. C. F. Manski, Communicating uncertainty in policy analysis. Proc. Natl. Acad. Sci. U.S.A. 5 March 2020. 116, 7634–7641 (2019). 43. C. Fox, A. Tversky, Ambiguity aversion and comparative ignorance. Q. J. Econ. 110, 17. R. E. Lofstedt, M. McLoughlin, M. Osman, Uncertainty analysis: Results from an em- 585–603 (1995). pirical pilot study. A research note. J. Risk Res. 9877,1–11 (2017). 44. Y. Bar-Anan, T. D. Wilson, D. T. Gilbert, The feeling of uncertainty intensifies affective 18. A. Hart et al., Guidance on communication of uncertainty in scientific assessments. reactions. Emotion 9, 123–127 (2009). EFSA J. 17, e05520 (2019). 45. M. A. Hillen, C. M. Gutheil, T. D. Strout, E. M. A. Smets, P. K. J. Han, Tolerance of 19. B. Fischhoff, A. L. Davis, Communicating scientific uncertainty. Proc. Natl. Acad. Sci. uncertainty: Conceptual analysis, integrative model, and implications for healthcare. U.S.A. 111 (suppl. 4), 13664–13671 (2014). Soc. Sci. Med. 180,62–75 (2017). 20. R. E. Lofstedt, F. Bouder, Evidence-based uncertainty analysis: What should we now 46. E. Peer, L. Brandimarte, S. Samat, A. Acquisti, Beyond the Turk: Alternative platforms do in Europe? A view point. J. Risk Res. 9877,1–20 (2017). for crowdsourcing behavioral research. J. Exp. Soc. Psychol. 70, 153–163 (2017). 21. A. M. van der Bles et al., Communicating uncertainty about facts, numbers and sci- 47. S. Palan, C. Schitter, Prolific.ac—a subject pool for online experiments. J. Behav. Exp. ence. R. Soc. Open Sci. 6, 181870 (2019). Finance 17,22–27 (2018). 22. Office for National Statistics, Labour market overview, UK: May 2019. https://www.ons. 48. OECD, Education at a Glance 2018: OECD Indicators (OECD Publishing, 2018). gov.uk/employmentandlabourmarket/peopleinwork/employmentandemployeetypes/ 49. Office for National Statistics, UK labour market: August 2017. https://www.ons.gov. bulletins/uklabourmarket/may2019. Accessed 5 March 2020. uk/employmentandlabourmarket/peopleinwork/employmentandemployeetypes/ 23. H. Cockburn, Donald Trump again claims to have largest presidential inauguration bulletins/uklabourmarket/august2017/. Accessed 5 March 2020. audience in history. Independent, 26 January 2017. https://www.independent.co.uk/ 50. J. Goodrich et al., Panthera tigris. The IUCN Red List of Threatened Species. https://dx. news/world/americas/donald-trump-claims-presidential-inuauguration-audience-history- doi.org/10.2305/IUCN.UK.2015-2.RLTS.T15955A50659951.en. Accessed 5 March 2020. us-president-white-house-barack-a7547141.html. Accessed 5 March 2020. 51. IPCC, “Summary for policymakers” in Climate Change 2013: The Physical Science Basis. 24. R. Meyer, How will we know Trump’s inaugural crowd size? The Atlantic, 20 January Contribution of Working Group I to the Fifth Assessment Report of the Intergov- 2017. https://www.theatlantic.com/technology/archive/2017/01/how-will-we-know-trumps- ernmental Panel on Climate Change, T. F. Stocker et al., Eds. (IPCC, 2013), pp. 33–36. inaugural-crowd-size/513938/. Accessed 5 March 2020. 52. Office for National Statistics, Migration statistics quarterly report: July 2018 (re- 25. C. R. Fox, G. Ulkumen, “Distinguishing two dimensions of uncertainty” in Perspectives scheduled from May 2018). https://www.ons.gov.uk/peoplepopulationandcommunity/ on Thinking, Judging, and Decision Making (Universitetsforlaget, 2011), pp. 21–35. populationandmigration/internationalmigration/bulletins/migrationstatisticsquarterlyreport/ 26. G. Ülkümen, C. R. Fox, B. F. Malle, Two dimensions of subjective uncertainty: Clues july2018revisedfrommaycoveringtheperiodtodecember2017. Accessed 5 March 2020. from natural language. J. Exp. Psychol. Gen. 145, 1280–1297 (2016). 53. European Social Survey, ESS Round 8 source questionnaire. https://www. 27. G. Keren, L. E. M. Gerritsen, On the robustness and possible accounts of ambiguity europeansocialsurvey.org/docs/round8/fieldwork/source/ESS8_source_questionnaires. aversion. Acta Psychol. (Amst.) 103, 149–172 (1999). pdf. Accessed 5 March 2020. van der Bles et al. PNAS | April 7, 2020 | vol. 117 | no. 14 | 7683 PSYCHOLOGICAL AND COGNITIVE SCIENCES

Journal

Proceedings of the National Academy of Sciences of the United States of AmericaPubmed Central

Published: Mar 23, 2020

References