M. Granger Morgan, Peter J. Adams, and David W. Keith conducted a survey, Elicitation of Expert Judgments of Aerosol Forcing, compiled and analyzed the results and produced the following article which has been accepted for publication by Climatic Change.

ELICITATION OF EXPERT JUDGMENTS OF AEROSOL FORCING
M. GRANGER MORGAN1, PETER J. ADAMS2, AND DAVID W. KEITH3
1Department of Engineering and Public Policy, Carnegie Mellon University, Pittsburgh, PA 15213 USA E-mail: granger.morgan@andrew.cmu.edu
2Department of Engineering and Public Policy and Department of Civil and Environmental Engineering, Carnegie Mellon University, Pittsburgh, PA 15213 USA E-mail: petera@andrew.cmu.edu
3Department of Chemical and Petroleum Engineering and Department of Economics, University of Calgary, Calgary, Alberta, Canada T2N 1N4 E-mail: keith@ucalgary.ca

1. Introduction

Working Group 1 of the third assessment of the IPCC (2001) reported that their largest uncertainty about the magnitude of anthropogenically induced radiative forcing was associated with the direct and indirect effects of aerosols (Figure 1).  Evidence on this forcing is available from a variety of sources including: laboratory studies; local measurements; satellite-based global measurements; mesoscale or cloud resolving models; and modeling studies based on general circulation models (GCMs).  Estimates can be based on "forward" calculations in which observed or modeled aerosols are used to calculate radiative forcings or on "reverse" calculations in which the aerosol forcing is computed as the "missing" forcing required for a GCM simulation of past climate to reproduce the observed temperature record (Anderson et al., 2003).  The IPCC noted that the "quantification of aerosol radiative forcing is more complex than the quantification of radiative forcing by greenhouse gases because aerosol mass and particle number concentrations are highly variable in space and time…The quantification of indirect radiative forcing by aerosols is especially difficult…[because] in addition to the variability in aerosol concentrations [it depends] on some quite complicated aerosol influences on cloud processes…" (IPCC, 2001).

Such formal elicitation of expert judgment has been widely used in applied Bayesian decision analysis (DeGroot, 1970; Spetzler and Staël von Holstein, 1975; Watson and Buede, 1987; von Winterfeldt and Edwards, 1986; Morgan and Henrion, 1990), often in business applications, and in climate and other areas of environmental policy through the process of "expert elicitation" (Morgan et al., 1978a; Morgan et al., 1978b; National Defense University, 1978; Morgan et al., 1984; Morgan et al., 1985; Wallsten and Whitfield, 1986; Stewart et al., 1992; Nordhaus, 1994; Morgan and Keith, 1995; Budnitz et al., 1995 and 1998; Morgan, Pitelka and Shevliakova, 2001).  The results of such studies provide a clear indication of the nature and extent of agreement within a scientific community and also allow conclusions to be drawn about how important the range of expert opinions is to the overall policy debate.  Sometimes apparent deep disagreements make little difference to the policy conclusions; sometimes they are of critical importance (Morgan and Henrion, 1990).

The method used here is based on a structured elicitation of each expert's judgment. It differs from group-based methods such as Delphi (Dalkey, 1969; Linstone and Turoff, 1975) of the more recent expert group method developed by Budnitz et al. (1995) in that we do not seek consensus between experts, nor do we provide a mechanism for iterative communication between experts.  An advantage of the method used here is that it can effectively test the range of expert judgments unhampered by social interactions, which may constrain discussion of extreme views in group-based settings.

Expert judgment is not a substitute for definitive scientific research.  Nor is it a substitute for careful deliberative expert review of the literature of the sort that is undertaken by the IPCC.  It can, however, provide a more systematic representation of the diversity of expert judgment than is typically provided in consensus reports, and thus valuable input to experts performing such reviews.  Indeed, it was a request from two of the authors of the current IPCC assessment that led us to conduct this study.  It can also provide insights for policy makers and research planners while research to produce more definitive results is ongoing. It was for these reasons that Moss and Schneider have argued such elicitations should become a standard input to the IPCC assessment process (Moss and Schneider, 2000).

2.  The Survey Instrument

We also asked for a judgment of total aerosol forcing.  Finally, since absorbing aerosols cause large surface forcings that affect the hydrological cycle, but are not captured by total aerosol forcing at the top of the atmosphere (Ramanathan et al., 2001), we also asked for a judgment of the surface forcing for aerosols.  In order to help experts keep the taxonomy of direct and indirect forcings straight, we included the simple diagram shown in Figure 2.

Given the substantial uncertainty in estimates of aerosol forcing, it is important to know how soon uncertainties might be reduced by future research.  We probed experts' judgment about the rate at which uncertainty might change in the future.  After the experts provided their estimates of top of the atmosphere forcings, we posed the following question:

Suppose we were to come back to you in 20 years and ask this question again.  Consider the full range of your uncertainty from lower to upper bound.  What is the probability that after 20 years of additional research at current levels of support the outer tails of your box plot for the current global average magnitude of <here we inserted the name of the specific forcing we were asking about>:

Please enter a separate number for each of the four contingencies.

______  will have gotten longer (i.e., taller)

______  will have gotten shorter by 0 to 50%

______  will have gotten shorter by 50% to 80%

______  will have gotten shorter by more than 80%

total probability = 1.0

Part 1 of the survey explained the motivation of the study.  Part 2 defined the forcings we would be asking about.  Part 3 provided background information on problems of bias and overconfidence in expert judgment.  Parts 4-9 contained the elicitation itself. Finally, in Part 10 we asked the experts to assess their own expertise on a scale from 0 (not familiar with this literature) to 7 (among the handful of top experts in the world) and to tell us about the kinds of information that played the most important role in shaping their judgments. 

The survey was designed to mimic protocols that we have employed in face-to-face interviews.  These protocols aim to minimize the effect of various common cognitive biases found in lay and expert judgment.  For each of the four standard measures of aerosol forcing, we first asked the experts to consider the factors that contribute to uncertainty before asking for their judgments about the strength of the forcing. In the elicitation itself, we first asked for extreme values so as to reduce the impact of anchoring, then asked the expert to consider counterfactual conditions that might widen their distributions so as to minimize over confidence, and then asked for interior points in the distribution before finally asking for a best estimate (Morgan and Henrion, 1990, Dawes, 1988, Kahneman et al., 1982). 

After doing this systematically for the case of the direct aerosol effect, we introduced the idea of "box plots" (Tukey, 1977) and asked the experts to transcribe their first set of responses into a box plot to represent their judgment about the forcing resulting from the direct aerosol effect.  In all subsequent questions, we asked experts to respond directly by constructing box plots.  In each case, the instructions read:

To minimize the risk of overconfidence, please start by drawing short horizontal lines to denote the lower and upper extreme values.  Ask yourself if you could explain smaller and larger values if they were found in the future, and if so, revise your bounds accordingly.  Then fill in the other elements of the box plot (X=5%;  =25%;  = best estimate;  =75%; X = 95%).

In the box plots displayed in this paper, a slightly different graphical convention was employed, as explained in the caption of Figure 3.

The survey instrument was iteratively tested and refined with two advanced Ph.D. students and one post-doctoral scholar studying aerosol forcings and was also reviewed by a senior colleague in the field.  A copy of the survey instrument is available at http://cdmc.epp.cmu.edu/docs/pub/survey.doc.

The survey was administered to experts by mail.  On January 18, 2005 we sent e-mail inviting 60 experts asking them to participate.  Resource constraints precluded inviting the entire expert community to participate.  Instead, in developing the list of experts we attempted to generate a representative sample of modelers and observationalists and to include experts in general climate science, cloud physics and aerosols. We built the initial list from our own knowledge of the field, a review of recent publications, and list of participants in recent meetings and workshops.  After asking a senior colleague to review the draft survey, and proposed list of participants, we concluded that the draft and list were too model-centric.  We revised the survey to make the treatment more balanced and made a special effort to add names in the other categories so as to achieve balance across different areas of expertise.

In our initial e-mail, we indicated that completing the survey would require approximately two hours and that we could offer a modest honorarium to those who could accept it.  Our e-mail messages were accompanied by a PDF copy of a "Dear Colleague" letter from Prof. Ron Prinn of MIT which explained that he had encouraged us to conduct the study.  He asked the experts to participate, noting that the IPCC Third Assessment had been "reluctant to provide even subjective estimates of the mean and standard deviation for indirect radiative forcing by aerosols."  He argued that "it is very important for policy-making to know if this situation persists."  He concluded: "I believe that the results from this effort, presuming its acceptance in a peer-reviewed journal, will prove valuable to the scientific community in the months ahead as they work to summarize the state of knowledge in this field in the Fourth Assessment."

On February 09 we sent follow-up email to 54 experts.  We received six declinations, in most cases on the basis that they did not have time.  We received no response to either message from 19 experts. 

Printed copies of the survey instrument were mailed to all 29 experts who agreed to participate.   Upon receiving the questionnaire, two responded that they did not believe they had the necessary expertise.  Of the remaining 27, 24 completed and returned the survey.  As explained below, one additional expert completed the survey well after this paper was submitted.  Table 1 lists the experts whose responses are included in this paper along with their affiliations.  In reporting results, numbers assigned to experts were randomized so that all results are anonymous.  We do this to assure that experts feel free to offer their frank opinions, uninfluenced by possible peer expectations or similar pressures.

Because all of the authors read each other's papers, some participated together in the preparation of the last IPCC consensus review, or are participating in the current IPCC review, and some have written papers together, we make no claim that the responses we have received are "independent" in the sense that they have not been influenced by each other's views.  However, because our objective is to sample the range of current expert opinion, and it is the nature of expert communities to engage in such consultation, we do not view this as a problem.  Readers are reminded we are not sampling from a distribution which describes the true values.  The judgment of one of the outliners may be correct, and those who share a consensus view may be wrong.  It is for this reason that we have cautioned against combining individual responses in the past, and do so as well in this case (Keith, 1996).

In two cases, experts returned a box plot that appeared highly anomalous.  We asked the two experts to confirm that they meant what they had drawn.  In both cases, they responded that they had not correctly understood the question, and revised their answers.  After this paper was submitted for publication, we received one additional response which contained three very anomalous responses.  We asked this expert for clarification but after waiting several weeks and receiving no response, chose to leave those responses out when we submitted the final version of the paper.

We received several responses to the question about how the uncertainty might change in 20 years, which did not sum to 1 (or to 100%).  When the error was small, we simply renormalized the four numbers.  In a few cases, where the error was > 20%, we checked with the experts, who, in each case, indicated they had not correctly understood the response mode, and provided us with a corrected response.  Once a draft of this paper was complete, we provided each expert with a copy and with their own code number, and gave each an opportunity to correct any major mistakes we might have made in interpreting their responses.

While we asked for anthropogenic aerosol forcing estimates only, Expert 9 chose to provide estimates of present day total (natural plus anthropogenic) values for sulfate, nitrate, organic carbon, and black carbon but did not include sea-salt or mineral dust.  Therefore, his estimates are only comparable to the others to the extent that the former are predominantly anthropogenic and the latter are predominantly natural.  Uncertainties regarding the fraction of mineral dust and organic carbon that are natural versus anthropogenic are not included in his range.

All of the survey responses we received were completed with a level of detail that clearly indicated that the experts had taken the task seriously.  Judging from the written comments and other indications of effort, all respondents probably devoted the two hours we had estimated, if not more, to completing the survey.  However, we did not ask the experts to report on how long they took to complete the survey.

3.  Results

Figures 3-8 report experts' judgments of the six measures of forcing that we elicited.   The table at the bottom of Figures 3-7 report how the experts anticipate their uncertainty judgment for absolute upper and lower bounds might change if we were to return to pose the same question "after 20 years of additional research at current levels."

All but one of the experts placed their best estimate of the value of the forcing from the direct aerosol effect between 0 and -1 W/m².  Six of those assigned a small probability (< 0.05) to the forcing from the direct aerosol effect turning out to be as negative as -2 W/m² or more.  Half the experts assigned at least some probability to the forcing from the direct aerosol effect being positive, but all but two placed the upper bound on possible positive forcing at or below 1 W/m².  Note that six of the nine experts whose self-assessment of expertise was 6 or 7 assigned no probability to the forcing being positive, and the remaining three assigned probabilities of between 0.05 and 0.25.  While the range of uncertainties reported by several of the experts is somewhat broader than that reported in 2001 by the IPCC (see Figure 1), the results are generally consistent.

Expert estimates for the semi-direct aerosol effect show greater variation.  The levels of self-reported expertise are lower, and there are no obvious trends with reported level of expertise.   There is not agreement about the likely sign of the forcing from the semi-direct aerosol effect.  Fourteen of the 21 experts who responded to this question gave best estimates that are positive, one gave a best estimate of zero, and five gave best estimates that are slightly negative (all with absolute magnitude < 0.2 W/m²).  For the direct aerosol effect there was at least some overlap in the uncertainty range of all the experts, while for the semi-direct aerosol effect this was not true.  For example, two experts assigned no probability to this forcing being positive, and three assigned no probability to this forcing being negative.  Part of this disagreement over the sign of the forcing produced by the semi-direct aerosol effect may result from ambiguity in how the community defines this effect.  The phrase was first used to describe reduction in cloud cover by atmospheric heating (Ackerman et al., 2000), which implies a positive forcing.  Indeed, Expert 13 commented that "By definition, the semi-direct effect is positive."  However, Johnson et al. (2004) found that absorbing aerosols above clouds can increase their liquid water path and reflectivity, which they termed "a negative semi-direct forcing".  Expert 10 explicitly stated that he did not include this negative effect in his assessment.  Expert 5 also commented on the definitional ambiguity.  This expert indicating that it was unclear where to include the possibility that black carbon "might brighten clouds in some circumstance more than it might diminish them by heating", and asked rhetorically, "does this still count as the "semi-direct effect?"

The semi-direct effect has been postulated at least since the work of Hansen et al. (1997), but relatively little attention was devoted to it until the INDOEX field campaign highlighted its importance (Ackerman et al., 2000).  Therefore, at the time of the Third Assessment Report, there was little basis for an assessment of its global magnitude.  It is mentioned in the final text of the report but it was not possible at that time to provide an uncertainty estimate.  Therefore, the uncertainties reported in Figure 4 represent an additional source of uncertainty not quantified in the IPCC summary figure.

Expert judgments regarding the indirect effects on cloud brightness (first indirect effect) and cloud lifetime/distribution (second indirect effect) are recorded in Figures 5 and 6.  Again there are no obvious trends with self-reported levels of expertise.  When comparing their responses to the IPCC summary figure (our Figure 1), it is important to bear in mind that the 0 to –2 W/m² uncertainty range given there refers only to the cloud brightness or first indirect effect.  The text and figure labels and captions make this clear in the main body of the report and in the technical summary.  However, the figure in the Summary for Policymakers, on which our Figure 1 is based, mentions this only in the caption.  This omission has caused some confusion.

Section 6.8.5 of IPCC Working Group I report states that "Available GCM studies suggest that the radiative flux perturbations associated with changes in cloud lifetime/distribution (second indirect effect) could be of similar magnitude to that of the first effect".  However, the authors "refrain from giving any estimate or range of estimates for the second aerosol indirect effect" because of a lack of observational support for the GCM studies and because of debate as to whether this effect can be interpreted as a radiative forcing in the normal sense.

Most experts who responded to our survey provided uncertainty ranges for both aerosol indirect effects that are much wider than those provided for the direct effect.  Of the 23 experts who provided judgments regarding the cloud brightness (first) indirect effect (Figure 5), five said this could be as strong as –3 W/m² or even more negative.  Fifteen gave uncertainty ranges from slightly positive to –2.5 W/m².  These estimates are broadly consistent with the IPCC summary figure.  The remaining three respondents did not provide absolute upper and lower bounds but provided 5^th and 95^th percentile confidence bounds that are similarly consistent with the IPCC summary figure. 

Despite the greater uncertainty, there was strong agreement that the forcing associated with the cloud brightness (first aerosol indirect effect) is negative.  While one respondent did not provide an estimate, the remaining 23 gave best estimates that ranged from - 0.4 W/m² to -1.25W/m².

For the cloud lifetime/distribution or second indirect effect, 22 experts provided responses.  Five said this effect could be positive.  Two of these said they thought it was conceivable that this effect is greater than 2 W/m² although these estimates fell outside their 95^th confidence intervals.  Eight experts said this effect could be –3 W/m² or more negative.  Expert 24 did not give absolute bounds to this effect while the remaining 13 experts said this effect could not be more negative than –2 W/m².  Collectively, these results indicate that the experts who responded assigned generally greater uncertainty to the cloud lifetime/distribution (second) effect than even the cloud brightness (first) indirect effect.  Given that the IPCC Working Group I did not estimate an uncertainty range for this effect, this represents a significant, but often unappreciated, uncertainty in aerosol climate forcing. 

There was similar agreement about the sign of the best estimates of the cloud lifetime/distribution or second aerosol indirect effect, although in this case, the uncertainty range produced by five experts assigned probabilities of > 0.05 that the value may actually be positive.  Two experts did not respond, one gave a best estimate of 0, the remaining 21 all gave best estimates that were negative, of which only 5 exceeded an absolute value of <1W/m².

Figure 7 reports the experts' estimates of total aerosol forcing.  Again we see no trends with levels of self-reported expertise.  All but one of the experts made a separate estimate of this value.  Expert 9 instructed us to treat his four distributions as independent, and perform a sum.

There is complete consensus that the best estimate of total aerosol forcing is negative.  Values of the experts' best estimates range from - 0.25 W/m² to - 2.1 W/m².  Fifteen of the 24 responses involve best estimate values between -1.0 W/m² and -1.5 W/m². A direct comparison with the IPCC results is difficult because this work includes uncertainties associated with the semi-direct effect and cloud lifetime/distribution (second) indirect effect that were not estimated in the Third Assessment Report.  While the range of "best estimates" presented here appears to be comparable to the estimate of IPCC 2001, the level of associated uncertainty assessed by almost half the experts appears to be considerably greater than that suggested by the IPCC in 2001.  Most of the additional uncertainty reported here results from inclusion of uncertainties associated with the semi-direct effect and cloud lifetime/distribution (second) indirect effect.

All of the 17 experts who responded to the question about surface forcing (Figure 8) gave best estimates that were negative, ranging from -1 W/m² to - 4 W/m².  Several who reported very low levels of expertise choose not to respond.  All but two of the responses involved an uncertainty range whose absolute value was wider than 2.5 W/m² and five experts provide estimates with an uncertainty range of > 7 W/m².

Note that several of the responses reported in Figures 3-8, the 0.05 and 0.95 points on the box plots are significantly closer to the ends of the boxes than one would anticipate for a singly peaked probability density function.  Since most of the experts probably did not intend to report a bimodal distribution this suggests that several have either reported boxes that are too long, or tails and 0.05/0.95 values that are too tight. 

In the past, in conducting face-to-face expert interviews using a more elaborate response mode, it has been possible to avoid obvious inconsistencies or to rectify them when they arose.  However, even in those cases, we have seen clear signs that experts have produced distributions that are too narrow (Morgan and Keith, 1995).  Given the large literature on overconfidence (Morgan and Henrion, 1990), we are inclined to believe that in many of the responses we have received in the current study, the problem is tails and 0.05/0.95 values that are too tight, not boxes that are too wide.  We pointed out this issue in a follow-up e-mail to the experts.  Three experts revised their distributions.  Six responded that they understood the problem but that since only the extremes of the distributions were affected, they did not want to make changes.

In summary, we caution readers against placing faith in the precise values of the extremes of some of the distributions we have reported, especially those in which the 0.05/0.95 tick marks are very close to the ends of longer boxes. However, we believe that the 0.25-0.75 boxes, and reported best estimates, do a good job of correctly reporting the experts' views.

In addition to broadening several of his distributions to correct this problem, Expert 7 also wrote: "My initial estimates were primarily based on simulations with global models.  Now tying it more to observational data the forcing can be more negative… It is not so much the properties of aerosol, which introduce these uncertainties but rather the relative altitude positioning of aerosol with respect to clouds."

Past work has indicated that "forward" calculations of total aerosol forcing tend to lead to larger uncertainties than "reverse" calculations (Anderson et al., 2003).  Sixteen experts indicated that they simply relied on forward calculations to provide their judgments while only Expert 16 indicated a use of pure reverse calculations.  Of the remaining experts, Expert 3 indicated that he used forward calculations "tempered by reverse considerations", Experts 8, 10, 11, and 20 indicated consideration of both, and Experts 14 and 23 specified that they used reverse calculations for total aerosol forcing and forward for the individual effects.  Given the small number of experts who relied on reverse methods for total aerosol forcing, it is difficult to determine whether there were systematic differences between experts using forward and reverse methods to assess total aerosol forcing.  While the experts using reverse methods did reject the possibility that aerosol forcing may be more negative than –2.5 W/m² (i.e., the anthropogenic greenhouse gas forcing), they were not the only experts to do so.

Table 2 lists the number of times an expert ranked a given factor as a major contributor to uncertainty in the direct, semi-direct, or either of the indirect effects.  For the direct effect, factors controlling the total mass burden of anthropogenic aerosol (anthropogenic mass emission rates, production of condensable gases) were mentioned by more than 10 experts in addition to physical and chemical properties of carbonaceous aerosols (composition of primary emissions and aerosol mixing state).  While composition of primary emissions were also often mentioned in the context of the brightness (first) indirect effect, uncertainties with respect to cloud microphysics, dynamics, amount, and distribution were important for both indirect effects and the semi-direct effect.

Experts do not expect that uncertainty about aerosol forcing will be resolved quickly. Of the 20 experts who provided estimates of how uncertainty about total aerosol forcing will change in 20 years, all but three thought that the probability that uncertainty would grow was larger than, or equal to, the probability that it would shrink by >80%; and only two thought that there was a greater than even chance that uncertainty would shrink by more than 50% (Figure 7).

4. Conclusion

While best estimates of average anthropogenic aerosol forcings obtained in this survey are generally consistent with values suggested by the IPCC in 2001, the range of uncertainty assessed by a number of experts is significantly larger.  Most of the additional uncertainty reported here results from the inclusion of uncertainties associated with the semi-direct effect and cloud lifetime/distribution (second) indirect effect.  A minority (five experts) also suggested that the cloud brightness (first) indirect effect could be as strong as –3 W/m², somewhat beyond the range reported by IPCC in 2001.  Consensus was strongest among the experts for the direct aerosol effect and for the cloud brightness effect (first aerosol indirect effect).  While forcing from the semi-direct effect was thought to be small (absolute values of all but one of the experts' best estimates were  < 0.5 W/m²) there was not agreement on the sign.  All best estimates of total aerosol forcing were negative, with values ranging between -0.25 W/m² and -2.1 W/m².  Where uncertainties were greatest (i.e., the indirect effects) or there was strong disagreement (i.e., the sign of the semi-direct effect), aerosol-cloud interactions were mentioned as major contributors to uncertainty. 

Acknowledgments

We thank the experts who participated in this study for their cooperation and assistance. We thank Kaiping Chen, Jeffrey Pierce, Serena Chung and Spyros Pandis for assistance in the development of the survey. We thank Barbara Bugosh and Patti Steranchak for administrative and secretarial assistance. This work was supported by National Science Foundation award SES-034578.

When in print, a link will be provided for the article including all tables and figures.

The Survey