Arguments
At a glance - How reliable are climate models?
Posted on 30 May 2023 by John Mason, BaerbelW
On February 14, 2023 we announced our Rebuttal Update Project. This included an ask for feedback about the added "At a glance" section in the updated basic rebuttal versions. This weekly blog post series highlights this new section of one of the updated basic rebuttal versions and serves as a "bump" for our ask. This week features "How reliable are climate models?". More will follow in the upcoming weeks. Please follow the Further Reading link at the bottom to read the full rebuttal and to join the discussion in the comment thread there.
At a glance
So, what are computer models? Computer modelling is the simulation and study of complex physical systems using mathematics and computer science. Models can be used to explore the effects of changes to any or all of the system components. Such techniques have a wide range of applications. For example, engineering makes a lot of use of computer models, from aircraft design to dam construction and everything in between. Many aspects of our modern lives depend, one way and another, on computer modelling. If you don't trust computer models but like flying, you might want to think about that.
Computer models can be as simple or as complicated as required. It depends on what part of a system you're looking at and its complexity. A simple model might consist of a few equations on a spreadsheet. Complex models, on the other hand, can run to millions of lines of code. Designing them involves intensive collaboration between multiple specialist scientists, mathematicians and top-end coders working as a team.
Modelling of the planet's climate system dates back to the late 1960s. Climate modelling involves incorporating all the equations that describe the interactions between all the components of our climate system. Climate modelling is especially maths-heavy, requiring phenomenal computer power to run vast numbers of equations at the same time.
Climate models are designed to estimate trends rather than events. For example, a fairly simple climate model can readily tell you it will be colder in winter. However, it can’t tell you what the temperature will be on a specific day – that’s weather forecasting. Weather forecast-models rarely extend to even a fortnight ahead. Big difference. Climate trends deal with things such as temperature or sea-level changes, over multiple decades. Trends are important because they eliminate or 'smooth out' single events that may be extreme but uncommon. In other words, trends tell you which way the system's heading.
All climate models must be tested to find out if they work before they are deployed. That can be done by using the past. We know what happened back then either because we made observations or since evidence is preserved in the geological record. If a model can correctly simulate trends from a starting point somewhere in the past through to the present day, it has passed that test. We can therefore expect it to simulate what might happen in the future. And that's exactly what has happened. From early on, climate models predicted future global warming. Multiple lines of hard physical evidence now confirm the prediction was correct.
Finally, all models, weather or climate, have uncertainties associated with them. This doesn't mean scientists don't know anything - far from it. If you work in science, uncertainty is an everyday word and is to be expected. Sources of uncertainty can be identified, isolated and worked upon. As a consequence, a model's performance improves. In this way, science is a self-correcting process over time. This is quite different from climate science denial, whose practitioners speak confidently and with certainty about something they do not work on day in and day out. They don't need to fully understand the topic, since spreading confusion and doubt is their task.
Climate models are not perfect. Nothing is. But they are phenomenally useful.
Please use this form to provide feedback about this new "at a glance" section. Read a more technical version via the link below!
Click for Further details
In case you'd like to explore more of our recently updated rebuttals, here are the links to all of them:
| Myths with link to rebuttal | Short URLs |
| Ice age predicted in the 1970s | sks.to/1970s |
| It hasn't warmed since 1998 | sks.to/1998 |
| Antarctica is gaining ice | sks.to/antarctica |
| CRU emails suggest conspiracy | sks.to/climategate |
| What evidence is there for the hockey stick | sks.to/hockey |
| CO2 lags temperature | sks.to/lag |
| Climate's changed before | sks.to/past |
| It's the sun | sks.to/sun |
| Temperature records are unreliable | sks.to/temp |
| The greenhouse effect and the 2nd law of thermodynamics | sks.to/thermo |
| We're heading into an ice age | sks.to/iceage |
| Positives and negatives of global warming | sks.to/impacts |
| The 97% consensus on global warming | sks.to/consensus |
| How reliable are climate models? | sks.to/model |
Agree that temperature models are v useful especially at the global level, in fact they have been amazingly accurate. Less useful are rainfall models; I’ve seen examples of backcasting which are hopeless.
And as models are downscaled, uncertainties increase; when they’re used to predict future species distributions for instance, there are many uncertainties. It’s fine if everyone understands this but the problem is that such uncertainties tend to be downplayed, especially when a popular account appears.
This is exacerbated by the beautifully created, highly coloured and smoothed maps that can appear without any indication of errors/uncertainties, and these can rapidly become a credible reality – but the map is never the territory.
Some modellers get very prickly if you point this out but I think senior scientists and journal editors should be more diligent in ensuring such uncertainties are clearly stated, though of course this makes for a less relatable message to non-specialists.
PSBaker @1 ~ you are quite right to say that the uncertainties re rainfall are making for a less relatable message to non-specialists [such as me ! ].
Rising sea level, rises in severity & duration of heat waves/ droughts/ floods are all important in the medium term. But a broad-brush picture of what the future holds, is quite sufficient for "us" to base our policy decisions on.
Mathematical delineation of uncertainties is relevant to the scientific specialist ( and especially to the hydrologist re rainfall variation) . . . but uncertainties are, for the rest of us, probably not worth addressing, unless you feel something misleading or nefarious is being concealed by their omission.
PSBaker #1 - yes rainfall can be difficult, especially when convection is involved! Dynamic rainfall along fronts models far better in my experience - less ingredients needed.
Looking for ways to explain uncertainty is certainly important because lay-folk tend to see things in black and white, whereas uncertainties drive science forward. This is a point that needs making repeatedly as it crops up.
Regarding the general topic of uncertainties: this handbook might come in handy when talking about them.
How reliable are computer models ?
According to the chart below - not very !
Its unfortunate that so much of the scientific literature relies on these models in making projections.
Gordon @ 5:
Congratulations on exposing the sources you are using for your comments here. It is unfortunate that so much of the "contrarian" talking points keep going back to the same unreliable sources.
This looks like another variation of a diagram from John Christy's flawed work, which has been debunked many times before. It is even featured in the Models are unreliable page that this short "at a glance" is updating. It is unfortunate that people like you can't be bothered to read the full blog posts you are challenging.
Here is the figure from that SkS page:
Here are a couple of RealClimate posts on the matter:
From 2016
From 2017
...and the key figure from those posts.
Oh, and if you want to read about how unreliable Christy's satellite temperature data has been over the years, read about it here and take a look at this graphic showing how often Christy has had to fix errors:
PSBaker @1:
You need to stop reading contrarian sources that ignore the uncertainties that the scientists are presenting and try to pretend that it is the scientists that are ignoring the uncertainties.
Here is a recent RealClimate post that compares models and observations. They update it each year.
Here is the main figure from that post. Notice how it has a shaded area showing the "Model ensemble spread"? Is that too "hidden"?
You may also wish to read the links and look at the graphs I have presented above, in response to Gordon.
A question for Gordon: Do you understand what "300-200hPa tropical temperature anomalies" are?
Bob @ 6
Why does the IPCC run a similar chart in AR5 ?
Gordon @10... It's not a similar chart, not least of all because it's an actual global mean projection rather than mid-troposphere.
Ron @11 similar insofar as the models are running hotter than the observed temperatures.
Gordon @10
The IPCC WR5 projection in your comments includes observational data from 1970 - 2012 only. Its old data possibly from an older IPCC report. The observational data is quite close to the modelling for much of that period but the years 2005 - 2012 (approx) in that graph clearly fall significantly below the modelling mid line prediction. But this is a relatively short time frame, and it represents short term natural variation that models can't fully predict in terms of timing. This is the alleged pause in surface temperatures, which amounted to a flat period in the warming trend of about 7 years and was due to the influence of natural variation ( in simple terms)
The graph posted by Bob Loblow @ 8 includes observational data from 1970 - 2022 and so its much wider and more recent data, and its obvious that the warming trend from 2012 - 2022 has swung back to near the model mid line prediction, and that the observational trend is tracking quite close to the modelling overall for the full period 1920 - 2022. The modelling is obviously not running significantly hot.
I'm surprsied you didn't notice any of this.
Gordon ~ Dr Christy's chart of 300-200 hPa . . . .
represents the atmosphere above the summit of Mt Everest.
Even a Yeti would die at those altitudes.
Possibly Dr Christy did not inform the Congressmen of the uselessness of his chart. But he still drew up the chart for them.
@ Bob Loblaw 8: I was not referring to contrarian sources, nor the global models of which you give an example.
Read Nissan’s et al paper to see what I mean DOI: 10.1002/wcc.579 “Climate models are unable to represent future conditions at the degree of spatial, temporal, and probabilistic precision with which projections are often provided, which gives a false impression of confidence to users of climate change information.”
Or more recent critiques
https://www.nature.com/articles/s41558-023-01632-5
https://www.nature.com/articles/s41558-023-01650-3
[BL] Full link to the first reference mentioned is this
Second and third links turned into actual links..
The web software here does not automatically create links. You can do this when posting a comment by selecting the "insert" tab, selecting the text you want to use for the link, and clicking on the icon that looks like a chain link. Add the URL in the dialog box.
Gordon @12... For the tropical mid-troposphere that's not an accurate statement, though, since those "observed temperatures" are known to be very hard to collect and have a very high level of uncertainty. It's as likely the obs are running cool. That's an area of the atmosphere (specifically selected by John Christy) where the modeling is probably giving us better information than oberserations.
Tip of the hat to Gordon for reminding us of why Skeptical Science must unfortunately exist.
Doug Bostrom @17 ~ A good point. Very !
PSBaker @15 ~ it would be helpful if you gave more detail on the 3 papers you mention.
Your first paper (H.Nissan et al., 2019) appears very vague, and talks in a general way about farmers' need for medium term weather predictions wrt pesticide spraying scheduling and suchlike agricultural management. Also talks (slightly) about 30-50 year plannings for dam construction. Overall, the paper had such an unapt & vague manner, that I began to suspect the authors were using AI-generated [ChatGPT] . This was not helped by their mention of soil-moisture predictions in three disparate parts of Bangladesh, nor of vague reference to rainfall pattern prediction in Kenya/Somalia. PSBaker, your quoted extract from the paper provides little-to-nil relevance to global climate modeling . . . and yet you seem to be using the #15 quote to circle back to a disparagement of climate models. [ If this was not your intent, then please be very specific on the point you were wishing to make. ]
Your second paper ( J.J.Lambrechts 2023) was paywalled [for me] but a half-Extract talked of micro-climates in a manner that suggested the body of the paper was not relevant to global climate modeling. Remember, this thread is a computer climate model thread.
Your third paper (Maclean & Early, 2023) was similarly unapt . . . but the paper's body did provide some amusement :- "We model ... distributions of 244 heathland and grassland plant taxa using both macro- and microclimate data and project these distributions ... [regarding] improving protection of refugial populations within species' geographic range ..."
Refugial populations of humans ~ would be more relevant to this thread.
Rob @ 16
Why do science.org carbonbrief.org and nature.com all report that there is an issue with the models running too hot ? If there is an issue shouldn't it be addressed in this rebuttal ?
Gordon... I get the sense you didn't actually read those articles because they state that "some" of the models run hot, not that "models run too hot."
Zeke's article on Carbon Brief that you've linked is a good one to read.
Perhaps you're also not understanding the expectations of climate models. No one expects that climate models are going to give us a precise pathway for global temperature. What they are intended to do is inform us in a way that benefits our understanding of the climate system and the likely impacts of our behaviors.
I've seen many a climate modeler saying, "Models are always wrong, but observations of the future are currently unavailable."
Climate models have actually done a very good job of projecting future temperatures over the years. Here's another Zeke piece on that topic.
You started commenting using Christy's work and haven't attempted to defend it. You're pulling up other examples of modelers doing their work improving the skills of their field by openly discussing areas of concern and methods to address them. But out of that you're somehow concluding, erroneously, that, "It's unfortunate that so much of the scientific literature relies on these models in making projections."
Your statement there is bizarre because, well, how else would one make projections if not by using models? And, your conclusion that the models are poor reveals your motivated reasoning on the subject, when in truth the models are not poor. They're really incredibly good.
Can models get better? Absolutely.
Are models a waste of time and money? Clearly not since they've consistently proven to be accurate within the range of uncertainties necessary to inform us of the challeges we face with climate change.
Gordon... "If there is an issue shouldn't it be addressed in this rebuttal?"
It is addressed in this rebuttal. It's clearly stated that, "Climate models are not perfect. Nothing is. But they are phenomenally useful."
That statement is appropriately inclusive of the points you're bringing up.
Rob @ 20
Lets just look at the headlines:
"Use of ‘too hot’ climate models exaggerates impacts of global warming" "Guest post: How climate scientists should handle ‘hot models’" "Climate simulations: recognize the ‘hot model’ problem"
I will restate that It's unfortunate that so much of the scientific literature relies on these models in making projections. This is especially true in light of the concerns that have been raised in the linked articles and the fact that these projections are used by policy makers to determine our future.
I fail to see in the rebuttal any mention that there is an issue with models running too hot - just that the models are not perfect. If the reverse was true and the models were running colder that the observational data would the rebuttal still be primarily about an issue of perfection ?
Don't just look at the headlines, Gordon. Read the materials in full.
It's clearly no "unfortunate" precisely because the models do a very good job. You are merely confusing scientists seeking ways to improve the models with thinking that means they're all bad. You're essentially motivated to throw the baby out with the bathwater.
The rebuttal states that models are not perfect.
What's also clear, and is extensively discussed in each of the articles you've linked to, is the "models" being discussed are a subset of the ensemble mean. They're discussing a few of the models. They aren't saying all models run hot. Some actually run cool. And different model runs can run hot or cold. And modelers will often tweek the weighted balance of models to produce better results.
Before dismissing a major body of research because it doesn't conform to what you think it should do, perhaps it would be appropriate to do more research and perhaps even talk to an actual scientist who does modeling so you can better understand how they do their work.
Just a suggestion.
Like John Christy :-)
John Christy's presentations are cherry picked and deliberately misleading. So, no. Not like John Christy.
FYI... Christy doesn't do any climate modeling. He manages a satellite data set.
That's rather funny considering how much of John Christy's work has been done by others, who caught his mistakes...
"Let's just look at the headlines."
No, let's not, under any circumstance. It is possibly the best way to be misinformed and disinformed. If that is your idea of thinking critically, you have a major problem.
PSBaker @ 15:
First, let's address your statement "I was not referring to..."
In your first comment (#1), you weren't referring to anything specific. You finished off with a broad, sweeping generalization about "senior scientists and journal editors". This gives the appearance that you are casting a wide net - as Eclectic said in comment #2 "unless you feel something misleading or nefarious is being concealed by their omission."
So, now that you have actually provided some specific references, let's look at them. Eclectic has already made some comment in #18, but I have a few more points to make.
For your first reference (link to full artcle):
Thus, your first reference is not a broad condemnation of climate models - and basically sounds like a cry that it's the fault of climate scientists that non-specialists don't pay attention to the well-documented efforts of those climate scientists to explain the limitations of climate models.
Now, for your two references to Nature Climate Change:
So, your second and third references also do not reflect on unreliability of global climate models.
P.S. The next time you want to post a comment similar to #1, please provide the references you are relying on in your first post. And as Eclectic has stated in #18, you should also be providing some sort of detail on what it in those papers that you think is relevant, and what point you want to make.
[BL] Typo corrected, as per comment below...
In #28, Philippe states "considering how much of John Christy's work has been done by others, who caught his mistakes..."
Maybe Gordon thinks this is a feature, not a bug. I have actually known people to use this approach. It more or less works like this:
At the end, you've got your publication, and someone else has done most of the work for free.
If the first draft was so bad it can't be fixed, no problem - you didn't spend much time on it anyway.
Sometimes you'll get lucky and a poorly-informed editor will send it to people that can't properly review it, and the journal will accept it in it's original crappy version.
Even better: your original crappy version can be sent to a journal where the editor is your pal, and will accept it no matter how crappy it is because they like the conclusions.
The few people actually involved in reviewing the paper in the early stages get to know that you are a crappy scientist, but most people only see the final paper and judge you on that.
John Christy's mistake has been to see steps 2-5 play out in the published literature, instead of behind-the-scenes review. The mistakes and corrections by others are all part of the public record.
...but at least John Christy gets to put several papers on his C.V. (albeit bad ones, but it's the quantity that matters, not the quality). So, you know, glass half full.
Bob Loblaw @31 ~ horrible to see how the Christy sausage is made !
Bob Loblaw @30 ~ thank you for the detailed critique. But you may (or perhaps may not) wish to make a small typo correction, where at about 70% of the way down, you quote the paper saying:
~ "heathland and grassland plant taxa"
~ which you rendered as: "heartland and grassland plant taxa".
Dr Freud (and I ) can't imagine how that slip happened. But delightful !
[BL] Thanks. Corrected. That's what happens when you re-type instead of copying and pasting...
So...how reliable are climate models? The title poses a quantitative question that the article never answers.
Goodness @Bob Loblaw @eclectic !
Much of your ire seems to be focussed on lack of specificity, vagueness etc …
Sorry, I didn’t realize there is some sort of rule about it … but if you look at the OP, it is a model of vagueness, no papers cited – frankly, it’s pretty much waffle.
In my response I did provide examples, albeit the first to hand. I could have done better, but since @eclectic stated “ . . . but uncertainties are, for the rest of us, probably not worth addressing” I felt that, together with the lack of specificity in the OP, this gave me some liberty to extemporize. I also did not want to finger specific papers since I know some of the scientists involved, who have enough problems trying to navigate their careers without feeling picked upon.
I’m not surprised by the response though, this has been my experience through the latter part of my career – that criticizing modelling evokes a particular snarky rage from the priesthood who elect to guard the eternal flame.
My central point however remains valid, the models are good at somethings, lousy at others and there’s a whole sub-industry of scientists applying them to make unrealistic projections. We who work in the field, trying to help (in my case poor farmers) find them of little use and even counterproductive.
That, in my humble opinion, is what the discussion of climate models should be addressing and what I, in my albeit halting fashion, was trying to convey.
Here’s Dr Baethgen covering some of these points better than I can, in a lecture from 2020, (start ~8 mins), esp. 18 & ~35m “So when you see these beautiful maps with reds and greens, don’t trust them, remember that behind that colour is a big uncertainty.”
https://worldcoffeeresearch.org/news/2020/watch-a-new-way-to-think-about-climate-change
[BL] Link activated. As previously stated, you need to do this yourself when preparing your comment.
Gootmud @ 33 and PSBaker # 34:
I will take those comments about lack of answers and vagueness in this specific post as a clear indication that neither of you have bothered to follow the links in this post to other posts that have additional details. The full "How reliable are Climate Models?" post has a basic and an intermediate tab with increasing level of detail.
I know that information about those extra details are hidden deeply in this post. You have to read all the way to the end of the very first paragraph in the green box at the top of this post to find where it says "Please follow the Further Reading link at the bottom to read the full rebuttal and to join the discussion in the comment thread there." And the actual links to 15 related SkS posts are also hard to find, being buried at the bottom under a big red heading that says "Click for further details".
[Still searching for that html sarcasm tag]
Bob @35: I was commenting on this post, not on other posts it links to. I thought the whole point of the at-a-glance series was to present entry-level answers for people without the time or the mettle for more detailed versions. I was expecting an answer, albeit a simplified one, but one that addresses the question framed. Do you see it differently?
PSBaker @ 34:
And your response is pretty much what I expected from you. You consider the people that put hard work into models that you admit "are good at some things", and call them "the priesthood who elect to guard the eternal flame".
The full rebuttal is pretty clear about the type of models being examined - global climate models. These are not designed to do everything, and never will be. Nobody ever has a "model of everything".
The examples you gave are very limited in scope, yet you have decided that "There’s a whole sub-industry of scientists applying them to make unrealistic projections."
You have dismissed my previous comments with the phrase "Much of your ire seems to be focussed on lack of specificity, vagueness etc." In comment #30, I gave specifics comments on what I read in the Nature Climate Change articles you referenced. You have not provided any response or rebuttal to those comments of mine.
For someone who claims "That... is what the discussion of climate models should be addressing", you seem to be very reluctant to actually engage is serious discussion of the references you supplied. Instead, you just call it "a particular snarky rage".
Given that you will be unlikely to actually engage in any discussion if I provide comments on the video you link to, I think I'll save my effort.
Gootmud @ 36:
Given that this specific post - in the very first sentence in the green box - says "the added "At a glance" section in the updated basic rebuttal versions", it is safe to say that yes, I see it differently.
I know this is probably "impossible expectations", but I expected readers to actually read the post, and understand that this post is just a way of presenting the new material added to the full post. After all, the second sentence in the green box at the top says "This weekly blog post series highlights this new section of one of the updated basic rebuttal versions and serves as a "bump" for our ask."
Gootmud @ 36:
If my comment at #38 seems to be an overreaction, I think it is partly due to your comment #33 appearing as if it was just a general overall dismissal of the new "At a glance", rather than a constructive suggestion of ways to improve it.
If you want to try again, giving specifics as to how you think such an answer to the question could be introduced into the text that has been posted in the OP, then that would be far more helpful.
Gootmud @ 33 etc ,
the OP title indicates a brief description of the subject. Nicht wahr?
If you wish an exhaustive description/analysis , then you must read further into the subject. But best if you first decide on exactly what you wish to obtain ~ do you wish for results that are adequately reliable for practical purposes [and evidently they are ] . . . or do you wish for some mathematical quantification of "reliability" (in which case you will need to produce some cutting-edge methodology for the assessment).
PSBaker @34 & elsewhere ,
There is a great deal of vagueness everywhere, to be sure ;-)
However ~ "Driving with a fuzzy view of the climatic road ahead . . . is better than driving into the future with eyes closed." [Or have I misremembered that aphorism by Sun Tzu ? ]
But for short-medium term purposes in agriculture (including coffee growing) . . . it is fuzzily unclear why you would criticize models of 30+ year resolution [i.e. "climate" ] for not being useful in the shorter term. And to quote again perhaps from Sun Tzu :-
~ "Do not be angry that an elephant is not the size of a mouse."
Bob @38: sorry, I'm finding that super confusing. If the at-a-glance section isn't meant to summarize the answer, I don't understand what it's for.
Or perhaps what's confusing is the answer it's summarizing? The basic answer seems likewise wide of the mark. It restates the quantitative question as more of a binary--do the models work or not?--and suggests they do as demonstrated by hindcasting. But then it shows a graph saying they're too conservative. And then it cites Hausfather's claim that 14 of 17 projections are indistinguishable (another binarization) from what actually occurred, which obviously leaves three others. I'm still left wondering...how reliable are climate models?
What would help is more discussion of how we should think about reliability, as that's the title's load-bearing characterization. Are models that are right 14 out of 17 times reliable? Reliable for what purposes?
Gootmud @41 & prior ,
as I mentioned @40 , you need to clarify your thinking on reliability.
Especially re the purpose of reliability. Then you will be less confused.
Gootmud @ 41:
In the long-term, the "at a glance" is not intended as a stand-alone item. It's just the opening for the full rebuttal. Something more than a headline, but still something that is supposed to introduce the full article.
It sounds like you want something more like an abstract for a paper - something that very briefly introduces the subject and very briefly gives the answer. That could be a constructive improvement - something worth considering.
Full disclosure: I have not been active in the writing of these at-a-glance updates, although I am part of the SkS "team".
Keep in mind that the rebuttals here at SkS are responses to certain common myths found in the contrarian meta-world. The specific myth for this rebuttal basically comes down to an argument that models are completely useless. The Freeman Dyson quote at the top of the main article starts with "[Models] are full of fudge factors that are fitted to the existing climate..." Similar sentiments are often expressed more or less in the form "they just make the models do whatever they want". You don't need to show that climate models are perfect to dispute that myth - just show that there are (a lot of) things that they can do well.
Evaluating models is a complex process. A full-scale global climate model produces far more output than we actually have in weather/climate observations. Far more spatial resolution in temperature, humidity, wind speed, radiation, etc - both vertically and horizontally.
And global climate models are really an assembly of many other sub-models. A radiation model. A cloud formation model. A precipitation model. A fluid dynamics model. A surface evaporation model. (My background mostly focuses on microclimate models, incorporating surface conditions and soil temperatures.)
Each of these sub-models will undergo its own evaluation, usually on a localized scale where far more detailed observations are available and can be used to confirm proper model performance. Then, when all sub-models are integrated into a global climate model, more validation is done with global observations.
Often, the global climate model will contain simplified models, due to the need to run them for thousands of points at high temporal resolution for long periods of time. The simple models can be tested against the more complex models that have been validated against more detailed observations.
As you said: "reliable for what purposes?". There is no simple answer to quantify "reliability" under any circumstances, and even complex answers require an answer to that "what purpose?" question, first.
For what it is worth, RealClimate has a recent post on model evaluation:
Evaluation of GCM simulations with a regional focus