Arguments
Skeptical Science Study Finds 97% Consensus on Human-Caused Global Warming in the Peer-Reviewed Literature
Posted on 16 May 2013 by dana1981, John Cook
A new survey of over 12,000 peer-reviewed climate science papers by our citizen science team at Skeptical Science has found a 97% consensus among papers taking a position on the cause of global warming in the peer-reviewed literature that humans are responsible.
Lead author John Cook created a short video abstract summarizing the study:
The Abstracts Survey
The first step of our approach involved expanding the original survey of the peer-reviewed scientific literature in Oreskes (2004). We performed a keyword search of peer-reviewed scientific journal publications (in the ISI Web of Science) for the terms 'global warming' and 'global climate change' between the years 1991 and 2011, which returned over 12,000 papers. John Cook created a web-based system that would randomly display a paper's abstract (summary). We agreed upon definitions of possible categories: explicit or implicit endorsement of human-caused global warming, no position, and implicit or explicit rejection (or minimization of the human influence).
Our approach was also similar to that taken by James Powell, as illustrated in the popular graphic below. Powell examined nearly 14,000 abstracts, searching for explicit rejections of human-caused global warming, finding only 24. We took this approach further, also looking at implicit rejections, no opinions, and implicit/explicit endorsements.
We took a conservative approach in our ratings. For example, a study which takes it for granted that global warming will continue for the foreseeable future could easily be put into the implicit endorsement category; there is no reason to expect global warming to continue indefinitely unless humans are causing it. However, unless an abstract included (either implicit or explicit) language about the cause of the warming, we categorized it as 'no position'.
Note that John Cook also initiated a spinoff from the project with a survey of climate blog participants re-rating a subset of these same abstracts. However, this spinoff is not a part of our research or conclusions.
The Team
A team of Skeptical Science volunteers proceeded to categorize the 12,000 abstracts – the most comprehensive survey of its kind to date. Each paper was rated independently at least twice, with the identity of the other co-rater not known. A dozen team members completed most of the 24,000+ ratings. There was no funding provided for this project; all the work was performed on a purely voluntary basis.
Once we finished the 24,000+ ratings, we went back and checked the abstracts where there were disagreements. If the disagreement about a given paper couldn't be settled by the two initial raters, a third person acted as the tie-breaker.
The volunteers were an internationally diverse group. Team members' home countries included Australia, USA, Canada, UK, New Zealand, Germany, Finland, and Italy.
The Self-Ratings
As an independent test of the measured consensus, we also emailed over 8,500 authors and asked them to rate their own papers using our same categories. The most appropriate expert to rate the level of endorsement of a published paper is the author of the paper, after all. We received responses from 1,200 scientists who rated a total of over 2,100 papers. Unlike our team's ratings that only considered the summary of each paper presented in the abstract, the scientists considered the entire paper in the self-ratings.
The 97% Consensus Results
Based on our abstract ratings, we found that just over 4,000 papers expressed a position on the cause of global warming, 97.1% of which endorsed human-caused global warming. In the self-ratings, nearly 1,400 papers were rated as taking a position, 97.2% of which endorsed human-caused global warming.
We found that about two-thirds of papers didn't express a position on the subject in the abstract, which confirms that we were conservative in our initial abstract ratings. This result isn't surprising for two reasons: 1) most journals have strict word limits for their abstracts, and 2) frankly, every scientist doing climate research knows humans are causing global warming. There's no longer a need to state something so obvious. For example, would you expect every geological paper to note in its abstract that the Earth is a spherical body that orbits the sun?
This result was also predicted by Oreskes (2007), which noted that scientists
"...generally focus their discussions on questions that are still disputed or unanswered rather than on matters about which everyone agrees"
However, according to the author self-ratings, nearly two-thirds of the papers in our survey do express a position on the subject somewhere in the paper.
We also found that the consensus has strengthened gradually over time. The slow rate reflects that there has been little room to grow, because the consensus on human-caused global warming has generally always been over 90% since 1991. Nevertheless, in both the abstract ratings and self-ratings, we found that the consensus has grown to about 98% as of 2011.
Percentage of papers endorsing the consensus among only papers that express a position endorsing or rejecting the consensus. From Cook et al. (2013).
Our results are also consistent with previous research finding a 97% consensus amongst climate experts on the human cause of global warming. Doran and Zimmerman (2009) surveyed Earth scientists, and found that of the 77 scientists responding to their survey who are actively publishing climate science research, 75 (97.4%) agreed that "human activity is a significant contributing factor in changing mean global temperatures." Anderegg et al. (2010) compiled a list of 908 researchers with at least 20 peer-reviewed climate publications. They found that:
"≈97% of self-identified actively publishing climate scientists agree with the tenets of ACC [anthropogenic climate change]"
In our survey, among scientists who expressed a position on AGW in their abstract, 98.4% endorsed the consensus. This is greater than 97% consensus of peer-reviewed papers because endorsement papers had more authors than rejection papers, on average. Thus there is a 97.1% consensus in the peer-reviewed literature, and a 98.4% consensus amongst scientists researching climate change.
Why is this Important?
Several studies have shown that people who correctly perceive the scientific consensus on human-caused global warming are more likely to support government action to curb greenhouse gas emissions. This was most recently shown in McCright et al. (2013), recently published in the journal Climatic Change. People will defer to the judgment of experts, and they trust climate scientists on the subject of global warming.
However, research has also shown that the public is misinformed on the climate consensus. For example, a 2012 poll from US Pew Research Center found less than half of Americans thought that scientists agreed that humans were causing global warming. One contributor to this misperception is false balance in the media, particularly in the US, where most climate stories are "balanced" with a "skeptic" perspective. However, this results in making the 3% seem much larger, like 50%. In trying to achieve "balance", the media has actually created a very unbalanced perception of reality. As a result, people believe scientists are still split about what's causing global warming, and therefore there is not nearly enough public support or motivation to solve the problem.
Such false balance has long been the goal of a dedicated misinformation campaign waged by the fossil fuel industry. Just as one example, in 1991 Western Fuels Association conducted a $510,000 campaign whose primary goal was to "reposition global warming as theory (not fact)." These vested interests have exploited the media desire to appear "balanced."
Open Access for Maximum Transparency
We chose to submit our paper to Environmental Research Letters because it is a well-respected, high-impact journal, but also because it offers the option of making a paper available by open access, meaning that for an up-front fee, the paper can be made free for anybody to download. This was important to us, because we want our results to be as accessible and transparent as possible.
To pay the open access fee, in keeping with the citizen science approach, we asked for donations from Skeptical Science readers. We received over 50 donations in less than 10 hours to fully crowd-fund the $1,600 open access cost.
Human-Caused Global Warming
We fully anticipate that some climate contrarians will respond by saying "we don't dispute that humans cause some global warming." First of all, there are a lot of people who do dispute that there is a consensus that humans cause any global warming. Our paper shows that their position is not supported in the scientific literature.
Second, we did look for papers that quantify the human contribution to global warming, and most are not that specific. However, as noted above, if a paper minimized the human contribution, we classified that as a rejection. For example, if a paper were to say "the sun caused most of the global warming over the past century," that would be included in the less than 3% of papers in the rejection categories.
Many studies simply defer to the expert summary of climate science research put together by the Intergovernmental Panel on Climate Change (IPCC), which states that most of the global warming since the mid-20th century has been caused by humans. According to recent research, that statement is actually too conservative.
Of the papers that specifically examine the human and natural causes of global warming, virtually all conclude that humans are the dominant cause over the past 50 to 100 years.
Net human and natural percent contributions to the observed global surface warming over the past 50-65 years according to Tett et al. 2000 (T00, dark blue), Meehl et al. 2004 (M04, red), Stone et al. 2007 (S07, light green), Lean and Rind 2008 (LR08, purple), Huber and Knutti 2011 (HK11, light blue), Gillett et al. 2012 (G12, orange), Wigley and Santer 2012 (WS12, dark green), and Jones et al. 2013 (J13, pink).
Most studies simply accept this fact and go on to examine the consequences of this human-caused global warming and associated climate change.
Another important point is that once you accept that humans are causing global warming, you must also accept that global warming is still happening; humans cause global warming by increasing the greenhouse effect, and our greenhouse gas emissions just keep accelerating. This ties in to our previous posts noting that global warming is accelerating; but that over the past decade, most of that warming has gone into the oceans (including the oft-neglected deep oceans). If you accept that humans are causing global warming, as over 97% of peer-reviewed scientific papers do, then this conclusion should not be at all controversial. With all this evidence for human-caused global warming, it couldn't simply have just stopped, so the heat must be going somewhere. Scientists have found it in the oceans.
Spread the Word
Awareness of the scientific consensus on human-caused global warming is a key factor in peoples' decisions whether or not to support action to reduce greenhouse gas emissions. However, there is a gap here due to the public's lack of awareness of the consensus. Thus it's critical that we make people aware of these results. To that end, design and advertising firm SJI Associates generously created a website pro-bono, centered around the results of our survey. The website can be viewed at TheConsensusProject.com, and it includes a page where relevant and useful graphics like the one at the top of this post can be shared. You can also follow The Consensus Project on Twitter @ConsensusProj, and on Facebook.
Quite possibly the most important thing to communicate about climate change is that there is a 97% consensus amongst the scientific experts and scientific research that humans are causing global warming. Let's spread the word and close the consensus gap.
Coming tomorrow, details about a feature that will let you test our results by rating the papers directly yourself. The Consensus Project results have also been incorporated into the rebuttals to the myths There is no consensus and IPCC is alarmist.
Otiose: You claim "The funding for climate science has been channeled by many governments since at least the early 90's (especially true in Europe) specifically towards those scientists concerned about climate i.e. probably already predisposed towards concluding that humans were the cause if not the primary cause."
This claims is illogical for a number of reasons. First, you offer no evidence that in fact scientists who choose to study climate are predisposed to concluding that humans cause warming. It can just as well be that people who study the climate see the evidence and come to the same conclusion that CO2 causes warming, the way a biologist concudes humans evolved from other species.
Further, you don't show that publishing papers questioning AGW would ruin a scientist's career. In fact, good research that overturn the consensus would make that scientists a rock star among his peers, a true innovator.
But more importantly, the whole idea that funding somehow causes bias shoud be dismissed out of hand. If we want to accept your argument, then we should also reject any biologists' claims about evolution, or any medical research--after all, most of such research comes from the government. Do engineers have a bias that objects fall at 32 ft/sec squared because they attend publich universities, funded by the government? Likewise, if a car company invents a super effecient car that gets 200 miles to the gallon, and produces the car so that everyone can test it, does that mean we shoudn't believe the car gets 200 miles to the gallon because the funding came from a car company? At the same time, if a private organization which is agains global warming funds scientists who truly produce papers overturning the consensus, do we accept thier finiing or reject them because of the funding?
Given your logic, we should not conduct any inquiry, but should act like medieval thinkers, identifying ourselves with tribes, and dismiss the ideas of other tribes not our own.
I think it's unfortunate that scientists didn't put this forward 40 years ago or whatever as "Global Warming" being an increase in the ocean heat content and hold hard onto that through the decades before discussing all these complex, though important, matters of climate change, global average surface temperature anomalies and so on. They've confused the heck out of the general public by pretty much stating it backwards.
grindupBaker: Yes. While fundamental controller for the climate of any body with an atmophere has always been top-of-atmosphere energy balance, we've been stuck with surface temperatures because that's all we've got for all most of the period of interest. The start dates for various records look something like this:
10,000-1,000BP: Global prooxy record
~1700: Regional instrumental record
~1850-1880: Global instrumental record
~1950?: Energy content from reanalysis
~1990s: Energy content from observations
#102, grindupBaker:
Real science often progresses in ways that seem not to have been the most straightforward, from a backwards-looking perspective. The development of quantum mechanics was far weirder than that of the framework for climate science, yet that was done in thirty years. If you look at the book by Archer & Pierrehumbert that I mentioned before, you can see how the picture fell into place.
"Jim Eager at 07:14 AM on 18 May, 2013
Herrhund wrote: "there are other theories"
Except that there are no other theories.
There are several hypotheses, most of which have already been disproven, such as "it's the sun," or shown to be insignificant, such as cosmic rays, but none have risen to the level of being a cohesive theory supported by the full body of evidence."
What about papers done in other languages than English?
Or paper which are about that exact topic but not having the key words in their abstract?
An example: 'Impact of galactic cosmic rays on Earth’s atmosphere and human health' A.Singh, D. Siingh, R. Singh;
The abstract of this paper does not use the key words - so I believe this research paper was not included. Just naming one example of a research paper that is ignored by this study.
Why are the CERN physicists focusing on cosmic rays when it is correct what Jim Eager said about that there are no theories supported by the full body of evidence?
The cloud experiment is going to release their research results this year, but here is some information for now:
http://home.web.cern.ch/about/experiments/cloud
Another example of a research paper, that could not find a way in this study because it is German.
http://wkserv.met.fu-berlin.de/Beilagen/2013/SO%2001-13%20Klimawandel%20Europa.pdf
This paper shows for example a significant correlation between the middle temperature in Europe and the number of sunspots since 1672.
herrhund - the CLOUD project at CERN is a bad example for your argument, there results are interesting, but don't actually support a strong link between cosmic rays and climate. This was something that was pretty well known already, but it does show that governments are happy to fund climate skeptic scientists if the quality of the proposed experiments was high (which AFAICS they were - the results provided so far have been interesting, but not for the reasons the skeptics hoped).
As for the paper in German, I suspect the reason it wasn't included is that it clearly isn't a journal paper. The fact it was published in January of this year is another good reason.
Well, I think nobody can talk about the cloud project in detail since the results are not released yet - I read an interview with the research team and at the moment the results are being checked and will be presented as soon as they are excepted.
http://science.orf.at/stories/1717291/
herrhund the CLOUD project has published the results of some of their work already, so there is no problem in discussing it. The fact is that it doesn't support your argument and that you simply don't want to discuss it, that is not the same as not being able to discuss it.
I suspect that there are very few following the discussion that would fall for such transparent evasion as introducing an item to support your argument and then saying we can't discuss it because it hasn't been publsihed yet!
Dikran,
the last publication from the cloud project is from august 2011
http://cloud.web.cern.ch/cloud/People/Publications.html
If you would understand German, the interview link I posted is from may 10th with the leading researcher of the cloud project. He is saying that they are going to release new research results, LATER this year - and it is going to be interesting.
herrhund, your evasion is transparent. You have made it abundently clear that you are unwilling to discuss the information the CLOUD project has published, which is a fairly good indication that either you know it doesn't really support your argument or that you don't actually know what it says and are just playing rhetorical games to avoid the fact that those who reject the existence of AGW are only a tiny minority within the scientific community - which is the topic of the discussion.
The CLOUD people were making the same kinds of noises to the general public before the last publication, but the science itself turned out to be a damp squib. I wish they were right, it would be great news, but I am rational enough to know that is highly likely to be wishfull thinking.
Another study that is not included in the consensus project because of missing key words in the abstract:
'Variation of cosmic ray flux and global cloud coverage—a missing link in solar-climate relationships'
(Henrik Svensmark, Eigil Friis-Christensen
Solar-Terrestrial Physics Division, Danish Meteorological Institute, Lyngbyvej 100, DK-2100; published 1997)
What I am trying to say here is, that throwing around with headlines like '97% Consensus on Human-Caused Global Warming in the Peer-Reviewed Literature' is not accurate since they are missing all literature in other languages than English, as well as all research papers which are not using the exact key words. And I can bring up tones of research papers like that - all peer reviewed.
Fine, there is a lot of evidence that the climate change is driven by humans - there is also a lot of evidence that support other hypothesis and theories. There is also something like False consensus effect. existing.
herrhund, there are also many papers not included in the project beacuse of missing keywords that do at least implicitly supprt AGW, I know this because I am one of the authors of one such paper! Pointing out skeptic papers that were not included does not establish there was a bias unless you can show that it disproportionately favours mainstream over skeptic papers. So please stop disrupting the discussion until you can do so.
Note that in the particular case of the Svensmark and Friis-Christensen paper, there is clearly no bias as the database doesn't contain the article by Gierens and Ponater that debunks Svensmark and Friis-Christensen either. Or the one by Jorgenson and Hansen. Funny you didn't mention that (it isn't difficult to check whether comments papers are published pointing out the flaws in existing papers).
Herrhund, your argument is incorrect, unless you have some reason to suggest that English speaking scientists are more biased in favour of AGW than against. You have provided precisely zero evidence to support that contention. Similarly the exact choice of keyword is irrelevant unless you can show that the particular keywords used were biased towards AGW. It is not feasible to surver every single paper ever published, so we have to have some method of generating a representative sample. Pointing out skeptic papers that were missed is not evidence of bias, especially if your examples includes papers that were debunked in the peer reviewed litterature and the debunkings are not in the database either!
Did you check all papers which count to the 32% with position on AGW if they got debunked somehow?
herrhund, stop playing games, I was merely pointing out your lack of self skepticism. The debunkings I mentioned were clearly listed on the web page that you yourself provided, but you failed to mention them. This means that either you saw them and chose not to mention them (which would be dishonest) or that you stopped looking when you found the evidence that supported your argument and didn't bother to check it out (which would suggest that it is you that is biased). Being charitable, I shall assume it is the latter.
Well, okay I will leave - doing some research though.
I don't know if you notices, English is not my first language so yes indeed, sometimes I oversee text which is not written in my mother language for some reason.
However, it does not change the facts I pointed out. And you did not answer my question since you expected me to check my sources too, I thought you checked yours in case some of them were debunked.
herrhund, yes, please do research, but bear in mind that it is generally best to do the research before posting.
Asking whether the database contains all debunkings is yet another attempt at rhetorical evasion on your part. It is irrelevant to the purpose of a study whether the paper is a debunking or not, so there is no need for me to check. The point was that there are plenty of non-skeptic papers that are not included due to the choice of keywords as well, that you were not taking into account. Pointing out the debunking papers was just an easy way of demonstrating that this is the case.
herrhund, two can play that game.
"Sensitivity of a global climate model to an increase of CO2 concentration in the atmosphere" by Manabe and Stouffer (1980) was not included because it was published too early for the survey, even though it clearly affirms anthropogenic global warming (IMO).
"Thermohaline Circulation, the Achilles Heel of Our Climate System: Will Man-Made CO2 Upset the Current Balance?" by Broecker (1997) was also excluded due to lacking the correct search terms although it also affirms AGW.
By its nature, not survey of the contents of scientific papers can be expected to be exhaustive; but any large survey is likely to be representative unless it uses biased search terms. This survey did not.
Of course, you can disprove my assumption that the survey was not biased by its search terms by conducting several different surveys of comparable size using distinct (and independent) search terms; or by one very large survey using a random selection of articles from the entire scientific corpus. But just identifying single papers that were not included proves nothing. For all you know, for each "skeptical" paper not included, there are 100 affirming papers that were not included and of which you are unaware.
herrhund - "What I am trying to say here is, that throwing around with headlines like '97% Consensus on Human-Caused Global Warming in the Peer-Reviewed Literature' is not accurate since they are missing all literature in other languages than English, as well as all research papers which are not using the exact key words."
Indeed, some skeptic papers were missed by those by criteria. But for those same reasons, so were many papers supporting the consensus. This is a sampling procedure, looking at a representative subpopulation and extrapolating to the whole, with small uncertainties due to sample size and how representative a slice of the population was examined.
12,000 papers is a huge sample - political studies with +/-3% accuracy are commonly done with <1000 samples. And I would opine that 'global warming' and 'global climate change' will produce a quite representative slice.
This is a percentage study of a population to measure consensus - and given the data, it's missed some 44 consensus papers for every single skeptic paper not collected by the search criteria. Whether or not it collected your favorite (individual) paper(s) is irrelevant; your argument is invalid.
It's very possible in a decade or two science historians will cite this paper as an example during the period in question measuring the degree to which politization was successful in defending a parituclar view or set of views in the climate field and not a measure of the validity of the science.
[DB] Please note that evidenceless posturing adds only noise to this discussion. Comments containing only noise will be removed.
Say we wanted to introduce a new brand of cat food, "Vibrizzae", and for our advertising campaign, we conducted a survey to find out whether cats preferred our catfood over that of our competitors. So we found twenty households with cats, and asked their owners to put out a bowl of Vibrizzae and a bowl of MouseChunkz for each cat at noon and then return an hour later to see which had been eaten. Eight cats were found to have eaten the Vibrizzae, two had eaten the MouseChunkz and 10 had not touched either (perhaps they were having a nap, or had gone outside hunting, or just weren't hungry). So how could we present our results? The cats that didn't eat either catfood don't really tell us anything about the cat's food preferences, so the obvious thing to analyse the food preferences of the cats that actually ate some of the catfood. We could then reasonably claim "eight out of ten owners, that expressed a preference, said their cats preferred Vibrizzae"; sound familiar?
Now the Vibrizzae Corporation could have said that "40% of owners said their cats preferred Vibrizzae", but that is clearly only a lower bound on the proportion of cats that preferred Vibrizzae as it implicitly assumes that the 50% of cats that didn't eat anything would have eaten MouseChunkz had they eaten anything. Now if you were the CEO of MouseChunkz inc. I can see why you would argue that Vibrizzae should choose the latter advertising slogan, but would that have been fair or reasonable? No, of course not, it would clearly be a daft request, and the CEO of the Vibrizzae Corporation would laugh his arse off if the CEO of MouseChunkz inc had made any request of the sort.
In case anybody has missed the analogy, the skeptics that are claiming that there is not a 97% concensus are making the same argument as the CEO of MouseChunkz inc. and their argument is about as sensible. "In our survey 97 out of 100 papers who expressed a preference, said their scientists preferred anthropogenic climate change" is essentially what the survey says.
Of course, like all analogies, you can almost guarantee that someone will extend it beyond reason to avoid the basic point being made, but at the very least it establishes a precedent for the calculation used. ;o)
The fact that two-thirds of the 12,000 papers reviewed did not explcitly mention endorse "climate change" or "global warming" completely undercuts the denier meme that scientists have adopted pro-AGW positions in order to feed at an imaginary grant trough.
Otiose: "It's very possible in a decade or two science historians will cite this paper as an example during the period in question measuring the degree to which politization was successful in defending a parituclar view or set of views in the climate field and not a measure of the validity of the science."
Yes, Otiose, it is possible. Now, tell me how probable it is, and give me the evidence upon which you base your answer.
herrhund: "Why are the CERN physicists focusing on cosmic rays when it is correct what Jim Eager said about that there are no theories supported by the full body of evidence?"
Herrhund, the cosmic ray theory is not a comprehensive theory of climate change. It is not an alternative to the theory of anthropogenic global warming. If Svensmark turned out to be right about cloud seeding, CO2 would not suddenly stop absorbing/emitting radiation at various pressure-broadened bands in the thermal infrared range. You also fail to understand what's working against Svensmark. That, in fact, strongly suggests that you're not concerned with scientific progress but rather finding anything (no matter how collectively incoherent) that supports your current understanding of the world.
Note that I tweaked the first sentence a bit to add some precision.
Great work, SkS team!
One complaint I have (or perhaps just missed) is a relative lack of discussion of the No Position papers and their self-ratings.
You mentioned at the end of page 4:
"Among self-rated papers not expressing a position on AGW in the abstract, 53.8% were self-rated as endorsing the consensus."
What about the other 46.2%? Needless to say, it would somewhat undermine your conclusion if of the No Position papers that were self-reviewed, the entire remainder rejected the consensus. Now, I'm 99.9% sure that wasn't the case, but unless I missed it, I don't see any further discussion of the self-reviews of the No Position papers.
Could an author perhaps provide some additional insight to go along with that 53.8% figure? Of the remainder, what percentage self-identified as "rejecting" the consensus, and what percentage self-identified as "no position."
My German is not as good as it used to be, but this link herrhund posted looks a lot like statements from a Brazilian 'skeptic' meteorologist (Luiz Carlos Molion): vague claims, supported by vague inductive reasoning. In the German case, some visual correlation between sun spot activity and temperature (not always global). In the Brazilian case, visual correlation between PDO phases and global temps. No quantification of effects, no error bars, no statistical analyses.
To those who want desperately to find some article with graphs and figures saying they are right, that's enough. If you don't have science, an article like that is as good as it gets...
Longhornmaniac8 @129, relevant information is in Table 5. Assuming that no paper rated as endorsing AGW based on the abstract was rated by its authors as rejecting AGW (and vice versa), from that table it can be seen that, of papers rated as having no position, or being undecided on AGW and which were self rated, 551 (41.25) where rated as self rated as affirming AGW, 761 (56.8%) were self rated as neutral or having no position, and 27 (2%) were self rated as rejecting AGW.
The discrepancy is no doubt because some papers rated from the abstract as affirming AGW were self rated as neutral. In fact, approximately 169 (21.4% of affirmation rated papers) must have been rated as affirming AGW by the SkS team, but self rated as neutral. Even though the survey generated some false positives, the ratio of false positives to false negatives is 0.4, indicating a strong conservative bias in the rating method.
Looking at a total comparison of abstract rated to all self rated papers, it is evident that the SkS rating team showed two distinct biases (in aggregate). There was a very strong bias towards rating papers as neutral which applied across the board. There was a weaker bias to rate rejection papers as neutral shown by the larger self rated/abstract rated rejection papers (3.25) than for affirmation papers (1.7).
The former is discussed in the paper, and is largely attributed to the reduced information in the paper, but I am inclined to think is in significant part due to too conservative a methodology. The later is not discussed. Some residual personal bias cannot be entirely excluded, but it is more likely (IMO) to be due to a tendency among rejection papers to hide that rejection deep in the paper in a few lines that are not, in fact supported by the rest of the analysis (as for example, in McLean et al, 2009).
It seems in some opinion pieces that deny the consensus (and at the same time propose that consensus in science is not important) there is a curious inability to understand the basis of the 97%. For example in: http://tucsoncitizen.com/wryheat/2013/05/18/cooking-consensus-on-climate-change/
In which the author writes:
"The actual findings of the survey, in Cook’s own words: “We find that 66.4% of abstracts expressed no position on AGW, 32.6% endorsed AGW, 0.7% rejected AGW and 0.3% were uncertain about the cause of global warming.” So how do you get a 97% consensus out of that?"
Somehow, even though the opinion piece itself expicitly includes the qualifying statement that the 97% figure is for papers addressing the cause of global warming (i.e. 33.6% of all papers surveyed), these "skeptics" are still confused. Hard to believe that something so basic can stump them.
Over at Lucia's Blackboard there is a largely irrational discussion of the paper, but Lucia raises one interesting point. Many of the papers implicitly accepting that >50% of late twentieth century warming is due to anthropogenic causes are papers looking at the consequences of that warming. Contrary to Lucia's assumption, most (65.45%) such papers ended up being classified as neutral, but never-the-less her idea of first classifying papers by the IPCC Working Group to which they belonged, and only classifying WG1 papers by level of support for the concensus. In her opinion, only WG1 papers are relevant to the issue.
Fortunately, the searchable database subclassifies papers as Impacts (equivalent to WG2), Mitigation (equivalent to WG3), and Methods and Paleoclimate (equivalent to WG1). Using that feature it is simple to determine that 92.94% of "WG1" papers that state a position affirm that anthropogenic forcings are responsible for >50% of recent warming; while only 7.06% deny it. So, even using Lucia's classification it is clear a consensus exists.
The problem is, I believe Lucia's classification to biased. She motivates it by talking about acceptance of AGW by biologists, and asking just how relevant that is to assessing the consensus. She ignores the fact that many "impacts" and "mitigation" papers look at the effects of global warming on the hydrological cycle, or frequency of storm events, etc. Assessing these issues requires a large overlap of skills with those required to assess the direct effect of CO2 on temperature (and are often done by the same people). Lucia's classification merely excludes a large number of papers that are relevant, and in a maner which biases the result.
Even biological impact papers are arguably relevant to the concensus, in that the biologists must at least make themselves familiar with the detailed predictions to write such papers. However, even accepting Lucia's point on this detailed classification, she has not shown that excluding them will have a significant impact on the survey results.
Tom Curtis @ 131,
Thanks for the reply. I see what you're doing there, used it as inspiration to come up with the following values:
Assuming (as you mentioned) there was a false positive on 169 of the abstracts, this means that 622 (791-169) of the papers were listed as endorsing the consensus in both the abstract and self reviews.
This provides us with 1342, the total number of self-reviews endorsing the consensus (622+[.538*1339]).
Using the same assumptions you made, 27 of the papers identified as neutral in the abstracts were self-identified as rejecting the consensus (although this assumes as well that there were no abstracts identified as rejecting the consensus that were self-identified as no position). 27/1339=2.02%.
This leaves us with the remainder of 592, which were identified as netural in both the abstracts and self-reviews. 592/1339=44.2%.
So:
53.8% of No Position abstracts self-review as endorsing the consensus;
44.2% of No Position abstracts self-review as No Position;
2.02% of No Position abstracts self-review as rejecting the consensus.
We can use these numbers to slightly adjust the abstract totals to provide a more comprehensive analysis.
37.2% (4447 [3896+720-169 we know were misidentified in the abstract]) were identified as endorsing the consensus in either the abstract review or self-review.
61.6% (7352 [7930-747+169]) were identified as No Position in either the abstract review or the self-review.
1.2% (145 [78+40+27]) were identified as either rejecting the consensus or uncertain in either the abstract review or self-review.
Of those papers having a position, 96.8% were identified by either the abstract review or self-review as endorsing the consensus, roughly the same as the numbers used by the authors, but a fun side track nonetheless!
Cheers!
Herrhund. There was an interesting review of the CLOUD results at RealClimate which gives some indication of why the research is interesting and worth doing. However, it also outlines succinctly the steps required for the results to even relevant to the question of global climate change. Do you believe new results from CERN are going to advance any of these steps further?
There is 100% consensus from the denialist camp on the validity of consensus in scientific matters of peer reviewed papers especially with this peer reviewed paper. Does their opinionated only consensus even negate their own premise? It is a perfect example of circular thinking. Sad. Bert
Longhornmaniac8 @134, I believe you've got the idea, but you have your figures wrong. When normalizing the original abstract ratings to allow for detected bias, I get the following for all papers rated:
Total: 11944 (100%)
Affirm: 7349 (61.53%)
Neutral: 4334 (36.29%)
Unsure: 22 (0.18%)
Reject: 239 (2.0%)
Ignoring neutral papers, that gives us:
Total: 7610 (100%)
Affirm: 7349 (96.57%)
Unsure: 22 (0.29%)
Reject: 239 (3.14%)
So, even adjusting for bias as best we are able with publicly accessible figure, the 97% concensus figure looks very solid, although it is possible that the original abstract rating understated rejections (1.2 vs 3.1%). We should not make the mistake of thinking that normalized figures are superior to the original abstract ratings. They no doubt contain their own biases, and some dubious projections. The figure for "unsure" papers is based on projections from just 5 out of a 1000 papers.
The normalized figures do not supplant the original figures, therefore, but help flesh out how robust those figures are. Various critics of this paper have pointed out a number of possible causes of error (many of which are imaginary). They have not, however, tended to flesh out the criticisms with numerical values. When you do, as you can see, the original figures continue to look very good.
Herrhund, first, GCR induced cloud nucleation climate forcing is very far from being an established theory supported by the full body of evidence. But more important, as DSL pointed out, any GCR effect, regardless of sign or magnitude, would in no way negate the radiative physics of CO2 and other greenhouse gases, nor negate the observed fact that burning fossil carbon has increased atmospheric CO2 by ~42%, thereby forcing Earth's energy budget out of balance. Any GCR effect would be entirely separate and in addition to the known anthropogenic enhanced greenhouse forcing, and not in any way be an alternative hypothesis. In short, it would neither endorse nor refute the AGW hypothesis.
I think the distinction between accept/reject AGW (options 1, 2 &3) and >50% influence (option 1) is blurred at times in the paper. The paper begins;
but then strays in the body of the paper.
A careful reading can tease this distinction apart, and one must also be careful to note that a large proportion of the rated abstracts/papers were neutral. It wouldn't matter so much if this was not a study aimed at public perceptions. As Joe Public aren't always careful readers they could come away with the impression that 97% of all the abstracts/studies endorse the 'consensus' that humans are primarily responsible for global warming over the last 50 - 100 years. This distorted reading, helped along by SkS emphasis here and there, is echoed in the press and blogosphere. Eg,
http://www.accuweather.com/en/weather-blogs/climatechange/new-study-reaffirms-overwhelmi/12560235
LINK
http://www.australasianscience.com.au/news/may-2013/consensus-humans-cause-climate-change.html
http://www.smh.com.au/environment/climate-change/not-much-climate-change-doubt-science-says-20130515-2jmup.html
These statements either state or imply that 97% of rated abstracts (or the subset of those venturing an opinion) agree with the proposition that human activity is responsible for most/all of recent global warming, a misapprehension not undiminished by the full context of the articles.
Is there a breakdown of results for each of the 7 ratings for the whole period in the study or supplementary material? I couldn't find any, which is a disappointing omission for a peer-reviewed survey.
The messaging seems to be successful, but at some cost to scientific rigour, IMO. Advocacy and objective science make uneasy bedfellows.
[RH] Fixed link that was breaking page formatting.
Dana @ here,
I hope I express Lucia's views properly:
Lucia thinks that if reviewers altered their approach to rating during the first phase of the rating period as a result a result of discussing and refining rating criteria, then this interferes with the notion of independent rating.
I'm not convinced either way, based on what the study methodology purports, but it certainly fails a double-blind test.
This begins to sound something like the hockey stick arguments. "Skeptics" claiming that this study is wrong for one reason, and another wrong for another reason, etc. But there are now over a dozen multiproxy reconstructions that come up with essentially the same conclusion as Mann's original work.
Now we have the consensus issue. Here we have a much larger sampling of data that is coming up with almost exactly the same number as several previous research papers. But there is always this and that nitpic.
Exactly how many times do you have to do research coming to the same conclusion using different methods before "skeptics" finally admit that maybe the answer is correct?
It seems to me this is exactly why we begin to call "skeptics" deniers.
Barry, how exactly can "double-blind" applied to this methodology? The conventional meaning is that subject doesnt know whether getting placebo or treatment, and nor does the assessor of the subject response. I'm scratching my head to see how you apply this paper ratings.
Exactly, Rob. Just as the E.P.A. is not required to reprove the existence of the atom every time it approaches a scientific question, scientists need not state a position on climate change in every paper they publish.
This is why the National Academy of Science took the unusial step of referring to the warming of the Earth and the human causation of it as "settled fact" (pages 44 and 45, here).
Kudos to you all for the effort!
I am wondering something : did you in any way measure the correlation between the abstract rating and the self rating? The paper mentions only how many papers were rated as endorsing or rejecting or no position under each rating method. It would be fun to see if there are any patterns in the rating of the abstract, which is admittedly subjective, vis-a-vis the self rating....
Rob,
I don't know which posts you are referring to, but a counter to your point is the "right answer, wrong method" response. This is a particularly apt caution considering the emphasis given by the author/s of this paper to the importance of solidifying public perceptions.
(I'm still mulling over the methodology and haven't yet arrived at an opinon)
scaddenp.
It does not apply. I mentioned that as but one familiar standard, a strict one. Is there a standard that the methodology for independence follows in this paper? Or is it a unique methodology? Is it normal practise for surveyors to discuss and amend their approach during the initial ratings procedure in this kind of review, or is it unusual? Does the methodology weaken the results? I don't know the answers to these questions.
barry @ 139,
I think the distinction between accept/reject AGW (options 1, 2 &3) and >50% influence (option 1) is blurred at times in the paper.
Papers that implied humans were having a minimal impact without saying so explicitly, or explicitly minimised or rejected the notion that humans are causing global warming, or explicitly stated that humans are causing less than half of global warming, were classified as 5, 6, and 7, respecitvely, so I think that the reporting you are objecting to actually reflects the findings correctly.
one must also be careful to note that a large proportion of the rated abstracts/papers were neutral.
Did you actually try the rating exercise yourself? If not, I suggest you do so, because then you'll see why they were "neutral" and what that really means. 50% of my sample were "neutral" according to the rules of the classification, but in every case it was purely because they were looking at impacts without addressing why global warming was occuring as it was outside the scope of their study. Indeed, the very first sentence of the introduction of one of my "neutral" papers (rated by reading the abstract in isolation) explicitly stated how much temperature was due to rise globally over the coming century based on greenhouse gas emissions and cited the IPCC as the reference!
For the purposes of this study it is fair to ignore neutral papers when quantifying the consensus, but it would be a mistake to assume that because they are neutral that is somehow evidence against the consensus. In fact, the reverse would be true — as the science matures, fewer and fewer papers would feel the need to mention "humans are causing global warming" because it's taken for granted that everyone accepts that, just as papers in the field of biology aren't going to be stating that they accept evolution unless they are explicitly about evolution.
Roger D @ 132,
Hard to believe that something so basic can stump them.
Indeed.
I think of the work leading up to the supporting/rejecting test as a filtering exercise, designed to weed out the irrelevant papers.
We start with the title of the paper: "Quantifying the consensus on anthropogenic global warming in the scientific literature"
That means that the first filter we are going to apply is one that will weed out everything that is not "the scientific literature". In this case, the ISI Web of Science was chosen to represent "the scientific literature".
This will include a lot of early papers that are not really relevant to modern scientific thinking, so the next filter is to limit the time period to 1991-2011.
This is still going to give an enormous quantity of literature to review, most of which will be irrelevant to the question at hand, so the next filter is to use the topics "global warming" and "global climate change". This should not affect the final result provided there is no systematic difference in the use of those keywords between papers that endorse the consensus and those that reject it.
Now the first three filters don't really require any manual effort to apply, but unfortunately there's no avoiding the next one — weeding out papers that are irrelevant to the question at hand but happen to use those keywords. That requires human involvement.
But the result of that final filtering step is a set of papers that actually does address the question and allows the degree of consensus to be quantified, just as the paper purported to do.
Now, they could report the percentage of papers that endorse the consensus and the percentage of papers that reject the consensus as percentages of various totals:
1. All papers that address the question at hand. (97.1% endorse, 1.9% reject.)
2. All papers that passed the keyword and date range filtering steps. (32.6% endorse, 0.7% reject.)
3. All scientific papers. (~0% endorse, ~0% reject.)
The first one is obviously relevant because it directly addresses the question at hand.
The second is also somewhat interesting, because we expect that as the science matures, fewer and fewer authors will feel the need to state a position on the subject because it will be conventional wisdom, and it's interesting to know how much effort the authors put in and to quantify the size of the work in the field.
The third is obviously useless.
Confusing the second with the first is, as you say, hard to believe.
Why stop at papers with those keywords? What is so special about a paper that fails to address the question that happens to use those keywords vs a paper that fails to address the question that happens not to? After all, the purpose of those keywords was simply to make the problem tractable by reducing the number of papers that need manual classification without, hopefully, affecting the final results. The percentages in #2 can be made as small as you like by adding even more irrelevant papers, but the percentages in #1 should be invariant provided the assumption holds that there is no systematic difference in the use of those keywords between papers that endorse the consensus and those that reject it.
Anybody who nevertheless chooses to use #2 but fails to report the "rejection" figure as 0.7% really has no excuse.
BTW, there is one filtering step that really does make a difference to the outcome, and that's the first one, but since that's the whole point of the paper I guess I'll let it slide. :-)
Barry, the aim of the methodology is create an inscutable rating of the papers. It is not an investigation of bias - it aims to avoid it. Surely discussion and comparison of ratings and raters to produce a reliable rating of the papers is actually important to this.Deniers are in a tiz of John's surveys because they are not confident that their ratings would not show bias. However, this paper is actually about measuring the consensus.
Eli's just put up a cross comparison from his own smaller study here.