Quantifying Uncertainty on Thin Ice

The IPCC’s fourth assessment report projecting sea level rise in 2100 of 18 to 59 cm excluded the contribution from ice sheets because the ice sheet models were not up to snuff.  They still aren’t, but researchers Bamber and Aspinall at the University Bristol have found a work-around: structured expert judgment (SEJ). Their first results were published in Nature Climate Change on Jan. 6 2012 and reveal a contribution to sea level rise from ice sheets in 2100 whose median estimate is 29 cm and whose 95th percentile is 84cm.

Ouch. Taking into account contributions from glaciers and ice caps (12.4 ± 4 cm) and thermal expansion of the ocean (14-32 cm) we’re looking at a range of 33-132cm in 2100, according to Bamber and Aspinall.

The media and blogosphere is abuzz.  See here, here, here, herehere, here, here , here, here, and here.

The article’s supplementary online material gives a feel for structured expert judgment. Figure S1 below gives “range graph plots showing the individual responses by experts to the key quantitative questions in the 2010 survey (blue) and repeat 2012 survey (red). Also shown are the Decision Maker (DM) pooled estimates based on self weighting (“confidence” multiplied by “expertise” from items 1-3, see questionnaire), equal weights and performance weighting using the Classical Model of Cooke. In general the latter, performance-based DM solutions have the smallest 90 percent credible ranges, providing an optimized pooling estimate of the associated uncertainty range.”

Reprinted by permission from Macmillan Publishers Ltd: Nature Climate Change, An expert judgement assessment of future sea level rise from the ice sheets,J. L. Bamber, W. P. Aspinall, copyright 2013

Reprinted by permission from Macmillan Publishers Ltd: Nature Climate Change, An expert judgement assessment of future sea level rise from the ice sheets,
J. L. Bamber, W. P. Aspinall, copyright 2013

“M” denotes modeler, “O” denotes observationalist, the results concern contributions in year 2100. Experts were combined according to “self-weights” (they assess their own expertise and confidence), “equal weights” and “Performance based weights”. The latter are determined by each expert’s statistical accuracy as measured on questions from their field to which true values are known post hoc, and informativeness. Note that self-weights and equal weights tell the same story, but Performance weights yield a more informative decision maker.

There is lots of good stuff in this article. The assessment of experts’ and decision makers’ statistical accuracy was greatly enhanced in a follow up elicitation conducted last October in which I was involved. On this occasion a serious effort was made to capture tail dependence in the uncertainty of drivers of ice sheet dynamics. Write up is proceeding with all deliberate speed.

Is structured expert judgment science? Well, experts’ performance is measured with standard statistical tools of hypothesis testing based on their assessments of variables from their field to which answers are known post hoc. In this very real sense, the results of an SEJ are falsifiable (pace Karl Popper), and are sometimes falsified. Is ignore effecting we cannot competently model better science? SEJ is for quantifying uncertainty, not removing it. For that we must do real science.

About Roger Cooke

Roger Cooke is a senior fellow and the first appointee to the Chauncey Starr Chair in Risk Analysis at Resources for the Future.

Views expressed above are those of the author. Resources for the Future does not take institutional positions on legislative or policy questions. All information contained on Common Resources is intended for informational and educational purposes and may only be used for these purposes. Please see RFF's Terms of Use for further information.

3 Responses to “Quantifying Uncertainty on Thin Ice”
  1. Alan Roth says:

    Roger, There has been a serious error in this story. The IPCC backed off of its projection of sea level rise in the FAR when at the end of 2007 it issued a Synthesis Report that was primarily intended to put a synopsis of the 3 major reports that were issued that year as part of the FAR but it also made corrections where necessary. One of those was in the projection for sea level rise. They changed it to no specific maximum because there were too many unknowns. But more importantly, they said the ice sheet dynamics (not just the melting) represented the major unknown and such dynamics could have a major impact on sea level rise. Dr. Richard Alley was one of the lead authors of this section of the FAR and it was his influence that prevailed. But even in the first version in early 2007, it was mentioned that the projections did not include the role of ice sheet dynamics. If you listen to what Dr. Alley is now saying, those dynamics could give us a sea level rise by 2100 of multiple meters. So this structured expert judgment is really worthless. The threat is much greater than expressed by the SEJ.

  2. Roger Cooke says:

    Thanks for your comment. Does “serious errors” refer to my blog or to the Bamber Aspinall article, or to something else? Richard Alley was a contributor to the Bamber and Aspinall SEJ paper, so his views are in the mix. It would be helpful if you could provide a reference for what “Richard Alley is now saying”. The structured expert judgment process aims for a traceable and defensible combination of experts’ views.

Check out what others are saying...
  1. […] enable rational consensus. This has been going on for a long time, but is new for climate science. I described an example regarding ice sheet dynamics early last […]

Leave A Comment