Arguments
Climate models accurately predicted global warming when reflecting natural ocean cycles
Posted on 21 July 2014 by dana1981
Predicting global surface temperature changes in the short-term is a challenge for climate models. Temperature changes over periods of a decade or two can be dominated by influences from ocean cycles like El Niño and La Niña. During El Niño phases, the oceans absorb less heat, leaving more to warm the atmosphere, and the opposite is true during a La Niña.
We can't yet predict ahead of time how these cycles will change. The good news is that it doesn't matter from a big picture climate perspective, because over the long-term, temperature influences from El Niño and La Niña events cancel each other out. However, when we examine how climate model projections have performed over the past 15 years or so, those natural cycles make a big difference.
A new paper led by James Risbey just out in Nature Climate Change takes a clever approach to evaluating how accurate climate model temperature predictions have been while getting around the noise caused by natural cycles. The authors used a large set of simulations from 18 different climate models (from CMIP5). They looked at each 15-year period since the 1950s, and compared how accurately each model simulation had represented El Niño and La Niña conditions during those 15 years, using the trends in what's known as the Niño3.4 index.
Each individual climate model run has a random representation of these natural ocean cycles, so for every 15-year period, some of those simulations will have accurately represented the actual El Niño conditions just by chance. The study authors compared the simulations that were correctly synchronized with the ocean cycles (blue data in the left frame below) and the most out-of-sync (grey data in the right frame) to the observed global surface temperature changes (red) for each 15-year period.
Red: 15-year observed trends for each period. Blue: 15-year average trends from CMIP5 runs where the model Niño3.4 trend is close to observations. Grey: average 15-year trends for only the models with the worst correspondence to the Niño3.4 trend. The sizes of the dots are proportional to the number of models selected. From Nature Climate Change
The authors conclude,
When the phase of natural variability is taken into account, the model 15-year warming trends in CMIP5 projections well estimate the observed trends for all 15-year periods over the past half-century.
It's also clear from the grey figure that models that are out-of-sync with the observed changes in these ocean cycles simulate dramatically higher warming trends over the past 30 years. In other words, the model simulations that happened not to accurately represent these ocean cycles were the ones that over-predicted global surface warming.
The claim that climate models are unreliable is the 6th-most popular contrarian myth. The argument is generally based on the claim that climate models didn't predict the slowdown in global surface warming over the past 15 years. That's in large part because during that time, we've predominantly experienced La Niña conditions. Climate modelers couldn't predict that ahead of time, but the models that happened to accurately simulate those conditions also accurately predicted the amount of global surface warming we've experienced.
Yu Kosaka and Shang-Ping Xie from the Scripps Institution of Oceanography published a paper in Nature last year taking a similar approach to that of Risbey and colleagues. In that study, the authors ran climate model simulations in which they incorporated the observed Pacific Ocean sea surface temperature changes, essentially forcing the models to accurately reflect the influence from these ocean cycles. When they did that, the authors found that the models simulated the observed global surface temperature changes remarkably well.
The results of these studies give us two important pieces of information:
Note: this post has been incorporated into the rebuttal to the myth 'Models are unreliable'.
Dana, which parts of planet would you say that the models "accurately predicted"?
Russ... Instead of alluding to something, please make the statement you wish to make.
Rob,
I'm not making a statement; I'm asking a question.
In reference to "Figure 5: Composite sea surface temperature (SST) spatial trends", where specifically did the selected models show any predictive skill whatsoever?
Russ... Your post barely passes the "no link only posts" rule in the comments policy. If you're going to "ask a question" I would suggest you be able to discuss the point rather that just post a rhetorical driveby.
Russ R @1&3, first, as the article makes quite clear, it is not claimed that any particular model is better at predicting ocean osscilations. It is claimed that models that better match the observed trends in ocean temperatures in the El Nino 3.4 region also better match global surface temperature trends. The El Nino 3.4 region is approximately on the equator (ie, in line with Papua New Guinea) and in the middle of the Pacific (approximately directly below the Berring Strait on the map you show). As you can see, the trends in those areas are similar between the best five models and observations (if nowhere else).
Second, the maps you osscilate between are 15 year trends, starting with a very warm year in 1998. Short trends are strongly dominated by outliers near the extremities. As it happens, 1998 was arguable the strongest El Nino on record (and second strongest on my preffered index). 2011 was arguably the strongest La Nina on record, a La Nina that continued into 2012. The strong cooling trend, therefore, shown in the observed map therefore represents the presense of these extreme values. That the average of five model runs does not show such extreme values is hardly a surprise. Clearly the models to not reproduce the exact observed ENSO behaviour, but still reproduce observed GMST trends very well with a far milder reproduced ENSO oscillation. As noted in Kosaka and Xie, when a model is contrained to reproduce the observed ENSO fluctuation, it gets an even better match on the trends.
Finally, here are the SST anomalies for 2012:
You will notice that the "observed trends" from your GIF are very poor predictors of SST anomalies. That is because, as noted, they show a short term trend and are dominated by the extreme values in 1998:
And for comparison, here is the 1998-2012 trend using the same dataset:
Tom,
The paper's authors make the claim in the abstract that their cherry-picked subset of climate models "have provided good estimates of 15-year trends, including for recent periods and for Pacific spatial trend patterns." (http://www.nature.com/nclimate/journal/vaop/ncurrent/full/nclimate2310.html)
The comparative image I uploaded was the authors' own depiction of recent 15-year spatial trends, both modeled and observed. (Image credit: Bob Tisdale)
I have a hard time seeing how their composite of the "best" models provides a good estimate of the actual warming trends seen anywhere in the world, Pacific or otherwise.
Russ, I am frankly having some difficulty believing you have read that paper rather than other people's slant on it. You claim "cherry-picked" subset, but the second sentence is:"Some studies and the IPCC Fifth Assessment Report suggest that the recent 15-year period (1998–2012) provides evidence that models are overestimating current temperature evolution"
If that is the postulate they are studying, then why would study of any other interval matter? Secondly, look at what they are testing for: The ensemble mean is composed from runs with ENSO in many different phases whereas what is observed is one particular instance of ENSO. Ergo, models in phase should be better predictors than models without.
Third, look at the legend. The numbers are trends (K/decade), (and NOT temperature) so range from -1.0-1.0 is rather small. The colouring the of trends is emphasise the spatial pattern difference between in-phase and out-of-phase ENSO models compared to observed.
Looks to me like you have extremely unrealistic expectations of model skill and certainly no modeller claims greater skill.
scaddenp,
Thanks for joining the discussion. Perhaps you can answer my question.
In reference to Figure 5(a) and Figure 5(c) (shown above @ 1.), for which part of the world did these carefully selected "in-phase" models even manage to predict the correct sign of the observed warming trend, let alone its magnitude?
The authors are the ones making the explicit claim that "climate models have provided good estimates of 15-year trends, including for recent periods and for Pacific spatial trend patterns."
All I would like to know is where might I find some of these "good estimates" of "spatial trend patterns", because they're certainly absent in the Pacific over the 15 years the authors presented.
Russ... Have you read the paper yet?
As per text, I see the cool east Pacific pattern of the observed in the inphase models whereas out-of-phase has the warming over this region. The text is about the ENSO-affected areas, so that is where I look.
Let's get a couple of things crystal clear. Models have no skill at decadal level prediction and dont claim to. Part of this is because especially ENSO (but also other modes of internal variability) is not predictable and not a single model run will have reproduced the actual ocean modes observed. Since ENSO is the perhaps the biggest contributor to internal variability in surface temperature, you would expect model runs that were in-phase to be better predictors than out-of-phase. This is more than adequately demonstrated in the paper, especially when taken as a whole.
In terms of model reliability, internal variability averages to climate of long enough period (30 years) and so model ensemble mean should be a reliable indicator of climate.
Rob, it looks to me like someone (Tisdale?) has jumped on the paper, misrepresented one figure to feed the meme "models are not reliable" and Russ has fixated on that. It remains to be seen whether Russ is only looking for confirmation bias or is going to read the paper for understanding instead.
Rob Honeycutt,
I've read everything made available outside the paywall.
Does that mean I'm not allowed to ask questions?
Russ... Previously you stated that, "All I would like to know is where might I find some of these 'good estimates' of 'spatial trend patterns' because they're certainly absent in the Pacific over the 15 years the authors presented."
Perhaps, if you're not getting the answers you want from the abstract and illustrations, you might see if you can track down the full paper.
I would suggest that you (and Tisdale) are reading something into Fig 5 that is not of consequence for the purposes of the study. No one is expecting the models to create a perfect match image. What they were looking for were models that were in phase with the ENSO cycle.
Well fig 4 (c,d) from the paper is also reproduced above and I think it is likewise telling, but it doesnt tell you about spatial patterns especially in ENSO affected regions. (ie without Fig 5, you couldn't be sure the results in fig 4 were for the stated reasons). (a and b from Fig 4 show essentially the same information but are compared to GISS instead of C&W). Apologies for tone Russ, I didnt realise you didnt have access to the whole paper.
Russ R @6:
1) The 15 year trends mentioned in the abstract are for the GMST as I have already mentioned, and as is stated in the OP, and illustrated in the OP with figures from the paper. Your failure to acknowledge this point makes your comments look like a calculated excercise in distraction.
2) Without reading the paper (which is behind a paywall), I cannot say which Pacific spatial trend patterns they are drawing attention to. However, I can see that the spatial trend patern in the eastern and central, tropical and southern Pacific are a reasonable match. Further, I note your exclusion of the middle panel of the figure showing regional trends. As a comparison is being made between the performance of two groups of models, excluding the data for one group of models suggests cherry picking.
3) I note from the abstract that comparison was made between models that were merely in phase with observed ENSO changes. That is, if they went from El Nino to neutral or La Nina; or from neutral to La Nina over the period, they counted. That was a very weak hurdle, and one which will clearly not match the observed, very sizable ENSO trend - and hence effect on GMST, or regional Pacific trend patterns. Indeed, the strong ENSO trend observed is far more likely to overwhelm the overall GW trend, resulting in negative rather than weakly positive trends. Given this, the contrast in trend strength between two model sets is likely crucial to understanding the claims made regarding Pacific spatial trend patterns.
The paper is demonstrating that models that by chace had a cooling La Nina trend for the last 15 years similiar to what just happened, tend to be at the low end of the range of model predictions just where observations are, meaning that observations are in keeping with a continued high rate of global warming and that during the next El Nino predominent period the earth's temperature should catch up again. Also imply that when a EL Nino trend occurs rate temperature rise accelerates and when La Nina occurs it decelerates.
The rate of temperature change in the observations graph is interesting though?
Seems the NH hemisphere has increased its rate of temperature increases quite a lot whilst the southern ocean rate seems to have slowed down.
Can see the LA Nina cooling off Americas.
Not sure why the of rest of Southern Ocean is cooling, in marked contrast to the NH, presuming something to do with ocean currents or the winds or both?
Any thoughts?
@15 Tom Curtis says "Further, I note your exclusion of the middle panel of the figure showing regional trends. As a comparison is being made between the performance of two groups of models, excluding the data for one group of models suggests cherry picking."
The middle panel that Russ R omitted was the one showing regional trends of the worst models. Cherry picking would be selecting the worst models to compare against observations. The correct, scientific procedure is to pick the best 4 of 18 models to compare against observations.
I do find it interesting though, that there is a better match between panels A and B (the best and the worst models) than there is between A and C (best models and observations).
I note that other posters have accused Russ R of cherrypicking by selecting the 1998 to 2012 period. That period was chosen by Risbey et al.
Charlie A - read more carefully. Russ accused Risbey of cherry-picking but the second line of paper explains why that interval was chosen. No other interval makes sense to the purpose of the exploration.
A and B are similar in that they have more red than observed, but the spatial pattern in ENSO area is what is being discussed and best has cold east just like observed and completely different to B. Jumping to conclusions without reading the paper is pointless.
Charlie A @17, there are a number of reasons for differences between the models projections and the observed values. One such reason, for example, is the fact that the models only use historical Total Solar Irradiance (TSI) up to 2008, and then repeat solar cycle 23 (April 1996 to June 2008) thereafter. As the start of cycle 23 had a higher TSI than the start of cycle 24, and as the TSI rose faster at the start of cylce 23 than in cycle 24, this means CMIP5 models overestimate TSI for the last part of the trend period. This will lead to their warming trends being overestimated by some small amount. Similar problems apply to volcanic forcings, and anthropogenic forcings.
These are not issues addressed by Risbey et al. They are confounding factors in the study. The proper way to address that is to show the SST trends in the models with the same ENSO phase and those with the opposite ENSO phase. Both will include the confounding factors. Consequently the effect of the difference in the ENSO phase will be found in the difference in trends between the two. It is that difference that needs to be compared to observed trends to see if they have the same spatial pattern.
By not including the middle panel, Russ (and Tisdale) prevent us from making that comparison. They have included two pieces of relevant information, but deliberately excluded the third piece which is germaine to the analysis. That is cherry picking. (They also make detailed comparison of the panels they included difficult be alternating them in a GIF so that they cannot be compared at the same time - something I consider to be a bad practise).
This is the study I had in mind here.
Let me make a couple of my points "crystal clear".
So, back to my original question... in what region of the world did these best 4 "in-phase" models show any predictive skill over the 1998-2012 time period that the authors presented?
Russ... I believe you are grossly underestimating the expertise of the researchers and reviewers involved in this paper, while over inflating your (and Tisdale's) capacity to comprehend the methods being used.
First off, these were model runs rather than different models. And the paper explains exactly why they were selecting specific runs. They were choosing runs that were phased with the ENSO cycles. That doesn't make it cherry picking (the suggestion being an act of grasping for straws on your part). They selected those model runs as a new and unique way of testing the models.
In case you don't understand, and as I've stated before, climate modelers do not expect for their models to phase with ENSO because it's not possible. Over time the ENSO phases all balance out, so what matters is the results over longer periods of time.
Risbey et al were merely saying, "What if we select the in-phase runs and use that to test the predicability of the models." They could have turned up something far more interesting in their results, like identifying some aspect of the models that had been previously overlooked. But, what they found was the models actually do a pretty good job.
Your complaints about fig 5 (a) and (b) (c) are completely meaningless. It's more grasping at straws. Somehow those two figures don't look enough alike to satisfy you (and Tisdale). Well then quantify it. Roll up your sleeves and do some real science. If you think that somehow that difference is something significant that was missed by all six of the authors and the list of expert reviewers, then you have a big job ahead of you explaining, in detail, with real numbers, what the difference actually are and why they are meaningful.
As for your "original question," it's not even rational. You're making an absurd insinuation that a small subset of model runs is going to predict exact regional temperature anomalies. Your question is a straw man in and of itself.
The pompacity of your position borders on the ridiculous.
Russ, your "cherry picking" complaint is groundless. The researchers' goal was to identify a source of model inaccuracy at a 15 year timescale. The researchers did not conclude that those particular models are better than other models at projecting global temperature. As Rob pointed out, the researchers selected only particular runs. The models used for those runs did not accurately predict ENSO events in other runs, nor will those models accurately predict ENSO events in future runs. The researchers did not claim that climate models are better than previously thought. They "merely" identified a still-unsurmounted barrier to models projecting well at short timescales.
Russ,
It appears to me that you have read everything not behind a paywall, and then didn't understand what you read.
The authors selected CMIP5 models on their ability to replicate the Nino3.4 index. That index is based on the sea surface temperature between 170W and 120W, 5N and 5S. Data shows cooling in that region during 1998-2012, and the 5 best models also show cooling in that region during the same period. Panel b of figure 5, which you omitted because WUWT omitted it (and because apparently you can't be bothered to spend $5 to read the article you criticize) shows that the 5 worst models show warming in that region during the same period.
When the stated aim of the paper is to determine whether there is really something wrong with GCMs or not, comparing a "best" subset to a "worst" subset is not only appropriate, it is often enlightening.
Dana,
Thanks for this, which shows again that successful ENSO prediction may be the missing key to short-term climate modelling.
In that regard, allow me to draw your attention to a series of remarkable posts at ContextEarth, where our intrepid blogger has managed to successfully retrodict ENSO for multiple centuries into the past with surprising fidelity. The trick is to use Matheiu functions (which are similar to trig functions in the elliptical co-ordinate system) rather than sine waves. This models the sloshing of water in the Pacific basin, and is tied to at least one lunar cycle.
Rules prohibit me from posting links, but Google should find it. C.E. remains anonymous for now, but he or she is apparently aiming for publication. So keep your eyes open.
Keith
Rob Honeycutt,
"I believe you are grossly underestimating the expertise of the researchers and reviewers involved in this paper"
Really? What exactly is Lewandowsky's "expertise" in climate modeling? Is "zero" a gross underestimate?
"You're making an absurd insinuation that a small subset of model runs is going to predict exact regional temperature anomalies."
That's funny... the authors themselves wrote: "We present a more appropriate test of models where only those models with natural variability (represented by El Niño/Southern Oscillation) largely in phase with observations are selected from multi-model ensembles for comparison with observations. These tests show that climate models have provided good estimates of 15-year trends, including for recent periods and for Pacific spatial trend patterns."
Nobody's asking for "exact" anything. I'd like to know which spatial trend pattern estimates from their selected models were even "good"? A correct average doesn't mean much if every underlying region is wrong.
If you spend a night at the roulette table, and you get every single bet wrong, can you still claim to possess predictive skill because the average value of your picks was close to the table's average result?
In her Carbon Brief blog post, Slow surface warming since 1998 is “not exceptional”, say scientists, Roz Pidcock discusses the findings of Well-estimated global surface warming in climate projections selected for ENSO phase, Risbey et al in conjunction with the findings of a second paper, Changes in global net radiative imbalance 1985-2012, Richard P. Allan et al.
Pidcock's post nicely supplements Dana's OP.
Russ... The whole concept was Lew's idea. Did he perform the modeling tests? Probably not. That was left for the researchers who had specific skills in that area.
Read again, Russ: "...natural variability (represented by El Niño/Southern Oscillation) largely in phase with observations..."
Again, your expectation of what models do is absurd. The authors were trying to get the best in-phase runs for a specific region in the eastern Pacific. That's all they're doing. They looked at the 35 runs and selected the ones that were largely in phase with observed ENSO cycles.
How hard is this to comprehend? Really?
Russ: An obviously decent metric of how well the models projected the Pacific spatial trend patterns is the difference in match to observations, of the model runs that worst matched the ENSO index versus the runs that best matched. You can see that in HotWhopper's post by scrolling down to the images that contain the Risbey et al. Figure 5 images.
Russ, this is verging on sloganeering and repetition. You have had it explained to you but at this stage it looks like willful failure to understand.
If you think they what they did was a cherry-pick, will you please explain to us what you think is the appropriate way to test their hypotheses (not yours) in a way that could use the full data set? I would perhaps suggest to the moderators that Russ's posts be deleted if wont either answer the question or withdraw the accusation.
@Russ R #1 "..hich parts of planet would you say that the models "accurately predicted"?" and
Russ R #26: " I'd like to know which spatial trend pattern estimates from their selected models were even "good"? "
Obviously, the 4 selected model runs are good in the Enso 3.4 area. The area for which they were selected as being good. Texas sharpshooting at its best.
These are the same models that will be the source for downscaling runs to create the regional predictions that are so popular in the adaptation community, so the poor performance of regional trends outside of the Enso 3.4 area gives and indication of the usefulness of such downscaling.
@Tom Curtis #19 "...the models only use historical Total Solar Irradiance (TSI) up to 2008, and then repeat solar cycle 23 (April 1996 to June 2008) thereafter. ......This will lead to their warming trends being overestimated by some small amount. "
This sounds dangerously close "it's the sun" Most Used Climate Myth (upper left sidebar).
Figure 2 of this paper show the "small amount" by which forecasted trends have diverged from reality in the sort period of true forecast vs. hindcast. Look closely at the trends from recent observations vs the models. Note it is nearly outside the 2.5 percentile line.
[PS] This is starting to drift well away from discussing the paper that is the subject of this thread. For general discussions of modelling skill, please put any responses in the "Models are unreliable" thread
Charlie A
"This sounds dangerously close "it's the sun" Most Used Climate Myth (upper left sidebar)"
This sounds more like you have completely misunderstood. Would that be deliberate? "its the sun" tries to explain observed warming by changes in the sun. Tom's argument is explaining why models (which have to guess actual forcing) get it wrong if the actual forcing is different. Happens both ways.
The subject of this thread is a paper looking at why observed is low compared to ensemble mean. That they are low is acknowledged in opening of the paper. Does the data support the hypothesis that this is due to state of ENSO? They contend yes and present data to support that.
This has important implications. If the paper is correct, then trends will rapidly increase when ENSO moves positive. Agreed? Or perhaps Charlie you think ENSO is going to stay negative forever?
Russ, scaddenp has provided yet another way of phrasing the purpose of the study. It was entirely possible that the model runs best at matching the ENSO index stubbornly would have been nearly as poor at projecting global surface temperature as were the model runs worst at matching the ENSO index. The conclusion would have been that failure to project ENSO timing was not really a major reason for the models' poor projections in 15-year timescales.
Charlie A, actually thinking about this more, would it be fair to characterize your position as believing that models are hopelessly wrong (despite the results of this paper) for reasons that have nothing to do with ENSO and that positive ENSO conditions will still result in observed temperatures running below ensemble mean?
If I have got this wrong, then can please state more clearly what your position is, especially in light of this paper?
I'll list a few of the key points this blog post made, and then explain more clearly my concern. Please tell me if you disagree with any of these statements made in the blog post
1. " ...because over the long-term, temperature influences from El Niño and La Niña events cancel each other out. However, when we examine how climate model projections have performed over the past 15 years or so, those natural cycles make a big difference."
2. "They looked at each 15-year period since the 1950s, and compared how accurately each model simulation had represented El Niño and La Niña conditions during those 15 years, using the trends in what's known as the Niño3.4 index."
3. "Each individual climate model run has a random representation of these natural ocean cycles, so for every 15-year period, some of those simulations will have accurately represented the actual El Niño conditions just by chance."
-------------------
Everything sounds fine to me up to this point. For each 15 year period, the authors of the paper select that subset of model runs where trends in the Enso 3.4 region best match observations. If they pick the models where enso 3.4 trend best matches observations, I would expect a good match in that area, as shown in fig 5 cell a. However, the authors make a more general claim. They claim the selected "climate models have provided good estimates of 15-year trends, including for recent periods and for Pacific spatial trend patterns."
Surely, the authors and the people posting here at SkS mean some other Pacific spatial trend pattern other other than the Enso 3.4 trend for which those specific models were selected.
Correct?
The second area of confusion is related to what is the expected performance of the models not in phase with actual real-world Enso.
While the 15 years period 1998-2012has had more La Nina years than El Nino, the ratio hasn't been spectacularly, abnormally lopsided compared to past history. See NOAA ONI table. So, if the random Enso phases of the model runsis the cause of mismatch in GMST trends between models and observations over the last 15 years, then quite a few model runs should have 15 year trends that lie above observations. Correct?
To put it another way, if the only model problem is phasing of Enso, and the current 15 year GMST trends are below all model runs (or perhaps only below 97.5% of all model runs). then I would expect that either 1) that La Nina in the real world over the last 15 years is at or above the 97 percentile point, or 2) that the distribution of Enso in the entire CMIP5 ensemble of model runs is overwhelmingly biased towards El Nino.
#1 is not true. I have not inspected the model oututs, but I doubt that #2 is true.
Russ wrote: "The authors took an ensemble of 38 models, and selected a narrow subset (~4) for analysis, excluding the other models."
Russ, you do get that people are objecting because this is straight up false, right? You keep plowing ahead as if you've got great points, and you would... if the basic foundations of your arguments weren't fictional.
You might as well be arguing that 'the Risbey paper is bad because it advocates killing puppies'. I understand your outrage at the heinous things you imagine them to have done... but given that these offenses exist in your own mind, its an exercise in self-delusion which is painful to watch. Either the Risbey paper picked ~4 climate models out of 38 or it didn't. Until you can connect with reality enough to see the truth on that point (and various others) you are arguing from a set of 'facts' different from the rest of us, and thus naturally reaching different conclusions.
scaddenp @30,
I'll happily answer your question, as soon as someone answers mine.
I asked first (two days ago). You can kindly wait your turn.
CBDunkerson @38
Perhaps you might like to try this simple quiz...
Answers:
What part of my statement was "straight up false"?
[Dikran Marsupial] Please can both sides of this discussion dial back the tone. RussR, answer scaddenp's question and address CBDunkerson's question directly, without sarcasm. If you want scaddenp to answer your question, please restate it politely. I will be monitoring this discussion and will summarily delete any post that violates the comments policy. Note especially:
No profanity or inflammatory tone. Again, constructive discussion is difficult when overheated rhetoric or profanity is flying around.
Please can everybody resist responding to RussR's post until he has first answered these two questions in a reasonable manner.
Perhaps CBDunkerson @38 is taking exception to your use of terms model and model run interchangeably.
I note that the authors also do this. For an example, see the caption to first figure in this blog post where "The sizes of the dots are proportional to the number of models selected. "
It is unclear to me whether the dot size is truly the number of models selected or the number of model runs.
Russ R. - From the paper:
In the 1998-2012 period 4 models met that criteria, for 1997-2011 a different subset, and so on for each 15-year window. As noted above, "The sizes of the dots are proportional to the number of models selected" for each window.
Your false statement is "excluding other models" - all of them were considered for each window, and subsets were selected for each window based on the stated similarity criteria to see how they differed.
ENSO-like variations in the models differ in phase based upon the individual runs. Since the entire purpose of the paper was to see if those models in phase with observational ENSO matched better or worse than those not in phase, the criteria used is entirely reasonable.
"ENSO-like variations in the models differ in phase based upon the individual runs."
I think a couple of people posting here probably haven't figured out that a finite number of runs (I think a few dozen are typical) which exhibit internal variability means that it's largely chance as to which model runs (and therefore which models) will be in phase for the historical recent ENSO history for each 15 year period.
Run each model a few more dozen times, select according to their algorithm in the same way, and for each 15 year period the model's whose runs match the historical 15 year period will differ ...
I think it's a rather ingeneous approach. Some model teams can (and have) performed hindcast runs plugging in known values for natural variability but that's expensive, much more expensive than this approach, and can't be done external to the team (unless you happen to have a massive supercomputer sitting in your basement).
RussR does not appear to understand the most basic aspects of the paper, or of climate modeling. His tone and anger suggest that his misreading / misinterpretation of the subject and paper is either consciously or unconsciously willful.
I would suggest that arguing with him is a complete waste of time. Certainly, correct his mis-statements, for the sake of lurkers and other readers, but engaging him directly is pointless.
The abstract is only 5 sentences long. The last two sentences (with my bolding) are:
"We present a more appropriate test of models where only those models with natural variability (represented by El Niño/Southern Oscillation) largely in phase with observations are selected from multi-model ensembles for comparison with observations. These tests show that climate models have provided good estimates of 15-year trends, including for recent periods and for Pacific spatial trend patterns."
So the authors selected a subset of the model runs and compared them to observations. They found that this subset of model runs provided good estimates for Pacific spatial trend patterns.
Can someone clarify what is meant by "Pacific spatial trend patterns" in this context. Do the authors mean basin-wide trend patterns; trend patterns in the same Enso 3.4 area used to select the subset of models; something else entirely?
I am not totally sure about what question Russ R refers to, but I believe the question may @8. This I answered directly in here though Russ obviously understand spatial pattern differently from me and the authors.
I think Russ is interested in pushing the point that even the selected scenarios do not match observations particularly well in absolute terms (does he really expect any ensemble to do that??) but seems much interested in this rhetoric point than in the more interesting questions as to why and the relative matching skill of phased and anti-phased scenarios.
scaddenp@45,
Apologies, I didn't realize that the first paragraph of your comment @10 was actually a response to my oft-repeated question (@1, @3, @8, and again @21)... which is why I kept on asking, with increasing frustration.
Since you've been kind enough to answer me, I'll be happy to answer your question @30. (I'll put that response its own dedicated comment next.)
"I think Russ is interested in pushing the point that even the selected scenarios do not match observations particularly well in absolute terms"
The point I've been pushing is this... the paper claims that the "in-phase" models accurately predicted global warming over the 15-year time period AND that the models "provided good estimates of 15-year trends, including for recent periods and for Pacific spatial trend patterns".
I don't take major issue with the first part... the average trends. A minor concern is that the selection criteria (NINO3.4 Index trend) is correlated with the outcome (global trends), meaning that to some extent the study will suffer from retrospective selection bias (which I called "cherry-picking").
But claiming that the models made even "good" spatial trend pattern predictions appears absolutely wrong, as shown by the authors themselves (Figure 5.) where they show a regional comparison of model predictions (best and worst) vs. observations. The relationship between even the "best" model predictions and the data is backwards in every ocean region except for the one region used for selection of the "best" models.
So, two points to make here.
" A minor concern is that the selection criteria (NINO3.4 Index trend) is correlated with the outcome (global trends), meaning that to some extent the study will suffer from retrospective selection bias (which I called "cherry-picking")."
This comment makes no sense. Cherry-picking is a very serious allegation to make about a published paper since it implies fallacious argument with the underlying suspicion of scientific fraud. Unsubstantiated accusations like this are not tolerated by the comments policy. I ask again, how could you test their hypothesis with this data in a way that doesnt use that selection method?
To your points:
1/ The "good" is being used to describe the pattern (not cell-by-cell estimate) compared to the anti-phased ensemble.
2/ It is no news to climate science that models are poor at regional level. Eg see Kerr (2013) (which was discussed at Realclimate here.)
However it does not follow that "if they cant do regional, then global is wrong". This is discussed in detail in papers referenced in the RealClimate article.
This is my personal view on this paper. The paper takes a novel way to test the hypothesis that poor match between ensemble mean and observations is due fact the model mean includes many different states of ENSO whereas observations a "one member of the ensemble". The paper does demonstrate that a mean created from runs which are in phase with actual state are a closer match to observed global temperature. This does underline the importance of ENSO on short term global temperatures. I am sure everyone is very surprized by that result (not!).
I do not think the paper can preclude (and the authors make no such claim) that there are other problems with the modelling. Beyond well-known problems with models, the question about accuracy of aerosol forcing seems to need more data (at least another year) from the Argo network. There could obviously be other errors and inaccuracies still hidden in modelling of feedbacks.
However, what you can conclude is that there is not as yet conclusive evidence of some unknown failure in the models on the basis of a mismatch between ensemble mean and observations: It would appear that issue of ENSO is quite sufficient to explain the mismatch in global surface temperature for such a short term trend.
scaddenp @47,
I'll withdraw my "cherry-picking" comment for two reasons:
1. It was never intended as an allegation of fraud, rather of selection bias, in that their selection criterion for the "best" models (agreement by chance with ex-post ENSO trends) was likely correlated with the model output being used to assess of predictive skill (global temperature trends). The higher the correlation, the more the method would treat luck as skill.
2. On further review it turns out that these two items (NINO3.4 trend and global temperature trend) are not nearly as correlated as I had imagined. When looking at 15-year trends, the correlation coefficient is only 0.13 between them.
So... withdrawn.
Thank you Russ. That is appreciated.