Frequently Asked Questions

The following answers are to questions regarding the paper “Quantifying the Consensus on Anthropogenic Global Warming in the Scientific Literature,” published in the journal Environmental Research Letters on May 15, 2013.

Methodology Questions

Where can I download your data?

The following data related to our research is available for download:

• Details of each paper and ratings based on the papers’ abstract (Year, Paper Title, Journal, Authors, Category rating (based on abstract), Endorsement level (based on abstract))
• Ratings by the authors of the papers (Year, Abstract Endorsement Level, Self-Rated Endorsement Level)
• First and second ratings by our team. Ratings are ordered sequentially. E.g., in order that original ratings were made (Article Id #, Original endorsement rating, Original category rating, Endorsement rating after consultation stage, Category rating after consultation stage)
• Data of 1000 "no position" abstracts that were reexamined for expressions of uncertainty about AGW (Article Id #, Expression of uncertainty on AGW. 0 = no position expressed on AGW. 1 = expression of uncertainty)
• The survey protocol used by the rating team
• All the articles listed by Id number (Article Id #, Year of Publication and Paper Title)

Who contributed to The Consensus Project?

The Consensus Project was a citizen-science effort that took over a year from beginning of the project until submission of the paper to the journal Environmental Research Letters. 24 volunteers contributed to the rating of the papers. Half of the raters completed 97.4% of the ratings. All of the major contributors to the project are named as co-authors or thanked by name in the Acknowledgements of the full paper.

In order to invite scientists to self-rate the level of endorsement of their own papers, 40 volunteers helped collect over 8,000 scientists’ emails. A total of 1,200 scientists responded to the invitation with over 2,000 papers each receiving a rating from the paper’s author.

It was very important to the volunteers at The Consensus Project that our published paper be freely available to the public (unfortunately, many peer-reviewed papers are hidden behind a pay-wall). For this reason, we chose the high-impact journal Environmental Research Letters, which charged a $1,600 fee to enable the paper to be open-access. This fee was crowd-funded by donations from Skeptical Science readers. The funds were raised in 9 hours.

The importance of raising awareness of the scientific consensus on climate change cannot be overstated. Typically, the general public think around 50% of climate scientists agree that humans are causing global warming. The Consensus Project has shown that the reality is 97%. To help “close the consensus gap”, the website TheConsensusProject.com was created pro-bono by the New York-based design and advertising agency, SJI Associates.

How did you independently check your results?

Nobody is more qualified to judge a paper’s intent than the actual scientists who authored the paper. To provide an independent measure of the level of consensus, we asked the scientists who authored the climate papers to rate the level of endorsement of their own papers. Among all papers that were self-rated as expressing a position on human-caused warming, 97.2% endorsed the consensus. This result is consistent with our abstract ratings, which found a 97.1% consensus.

We adopted several methodologies to eliminate the potential of bias in our abstract ratings. We developed a strict methodology (see the Supplementary Material for more details) specifying how to categorise each abstract. In addition, each abstract was rated by at least two separate raters, with any conflicts resolved by a third reviewer.

The entire database of 12,464 papers is available in the Supplementary Material. We have also published all our abstract ratings, which are also available via a search form. However, we have not published individual self-ratings from the authors of the papers as the “self-rating survey” was conducted under the promise of confidentiality for all participants.

We have also created an Interactive Rating System, encouraging people to rate the papers themselves and compare their ratings to ours. We welcome criticism of our work and we expect that further scrutiny will improve the reliability of our results. By encourage more interaction, we hope people will come to appreciate the diversity and richness of climate research.

Who did you email to categorise their own papers?

We obtained emails from 8,547 authors of the climate papers matching our Web of Science search. These were typically either the corresponding author or the lead author of the paper. Not all journals provide email addresses of the authors. In many of those cases, addresses were found via the authors’ listed affiliations or searching the internet. Email addresses were found for 60 to 70% of papers published between 1995 and 1999, 70 to >80% for papers published between 2000 and 2005, for more than 90% in the years 2006 to 2008, and for all papers in the sample published in the years 2009 to 2011.

An initial email was sent to the 8.547 authors, inviting them to categorise their own published papers. After a delay, a second email was sent to all unresponsive authors. In total, we received 1,200 responses. While this indicates a relatively low 14% response rate, this is very likely an underestimate of the actual response rate. The mailing system employed (using the University of Queensland mail server) failed to report bounced emails. As emails were collected from papers dating as far back as 1991, it is expected that a proportion of the email addresses would be out of date by late 2012 when the emails were sent. As expected, response rates were lowest in the early part of the period, as shown in Table S2 from the Supplementary Information.

What is the significance of the papers that express no position on human-caused global warming?

Naomi Oreskes predicted in 2007 that as human-caused global warming became settled science, fewer papers would see the need to explicitly endorse the consensus. For example, no research papers on geography currently need to state that the Earth is round. Our results confirm this prediction: As the field progresses, scientists feel less and less need to waste the valuable real estate of the paper’s abstract with an affirmation of settled science.

Moreover, most of papers that expressed "no position" in the abstract went on to endorse the consensus in the full paper. We determined this by asking scientists to rate the level of endorsement of their own papers - a way of rating the full paper rather than just the abstract. More than half of the papers that were rated as "no position" based on their abstract were self-rated as endorsing the consensus.

Did your paper include all climate papers?

Our analysis of 12,464 abstracts is the most comprehensive analysis of climate research to date. Nevertheless, a common comment we received from the scientists who rated their own papers was that we didn’t include all their climate research. Some even sent us their publication record and CV!

It’s certainly true that there is a great deal more published climate research than the 12,464 abstracts matching the search terms “global climate change” or “global warming”. For example, a search of the Web of Science for “climate change” articles from 1991 to 2011 yields 43,650 papers while a search for “climate” results in over 128,656 papers.

It is possible that a broader search would find more papers rejecting human-caused global warming. However, it is expected that an expanded search would also produce a great deal more papers endorsing the consensus. This is an interesting area for future research. A broader search would allow analysis of how consensus has evolved going further back into the past, to examine when the consensus formed in the scientific literature.

How did the level of consensus differ across different research areas?

The 12,464 climate papers we analysed covered a diverse range of topics (readers are strongly encouraged to use our Interactive Rating System to experience first hand the breadth and depth of climate research). Scientists were categorised as writing research on Impacts, Paleoclimate, Mitigation or Methods. The consensus levels among the scientists across these different categories of research were all 97% or higher:

The issue of consensus across different areas of research is something we plan to investigate further in future research. A 2009 survey of Earth scientists by Peter Doran and Maggie Zimmerman found that as the expertise in climate research increased, agreement that humans were causing global warming grew stronger. An interesting research question is how the level of consensus changes among published research that is more relevant to climate science (as opposed, for instance, to papers focusing on mitigation technologies).

How broad is the consensus?

The crowd-sourcing nature of The Consensus Project enabled us to perform an analysis an order of magnitude greater than previous analyses. We identified 10,188 scientists who had authored climate abstracts endorsing human-caused global warming. These scientists came from at least 74 different countries. We identified nationality from email addresses and given that emails were not obtained for all scientists, it is expected that the number of countries is an underestimate. This indicates the scientific consensus involves a large community of scientists scattered all over the world.

Does your result change if impact papers are excluded?

Out of the 11,944 papers in our analysis, 5780 were categorised as Impact papers. How does excluding these papers affect the final results? Among all non-Impact papers that state a position on human-caused global warming, 97.8% endorse the consensus (2661 out of 2722 papers). Linear regression over the 21 year period finds a statistically significant increasing trend (p = 0.047). Consequently, the chief results of Cook et al of an overwhelming consensus greater than 97%, strengthening from 1991 to 2011, still holds if impact papers are excluded from the analysis.

Philosophical Questions

Why does consensus matter?

When a representative sample of the US public are asked how many climate scientists agree that humans are causing global warming, the average answer is around 50%. There is a significant gap between public perception and the reality of 97% consensus. This “consensus gap” has real-world consequences. People who think that climate experts disagree about human-caused global warming are less likely to support policies that address climate change.

The consensus gap has persisted for over two decades due to deliberate efforts to cast doubt on the consensus. In 1991, Western Fuels Association conducted a $510 000 campaign whose primary goal was to ‘reposition global warming as theory (not fact)’. A key strategy involved constructing the impression of active scientific debate using dissenting scientists as spokesmen. Similarly, a memo from communications strategist Frank Luntz leaked in 2002 advised Republicans:

“Should the public come to believe that the scientific issues are settled, their views about global warming will change accordingly. Therefore, you need to continue to make the lack of scientific certainty a primary issue in the debate.”

Closing the consensus gap in the public perception of a scientific consensus is an important step towards meaningful climate action. For this reason, opponents to climate action has sought to widen the consensus gap for over two decades.

Isn’t science decided by evidence?

Absolutely! There is a quote by John Reisman that aptly sums up this sentiment:

“Science isn’t a democracy. It’s a dictatorship. Evidence does the dictating.”

That humans are causing global warming has already been established by many lines of evidence. A number of independent measurements all find a human fingerprint in climate change. Our study establishes that the scientists agree that humans are causing global warming and that their agreement is expressed in the most robust venue for scientific debate – in the peer-reviewed literature.

Consensus doesn’t prove human-caused global warming. Instead, the body of evidence supporting human-caused global warming has led to a scientific consensus.

What other indicators of consensus are there?

The seminal study in this area was done by Naomi Oreskes who surveyed 928 papers published between 1993 and 2003 whose abstracts matched the search term "global climate change". She found zero papers rejecting the consensus position that humans are causing global warming. We expanded her search by adding another decade of papers and including the search term "global warming", which swelled the number of papers to over 12,000. Our analysis confirmed Oreskes' result - among the papers matching her search parameters, zero rejected the consensus.

There have also been two surveys of climate scientist in recent years. One study by Peter Doran and Maggie Zimmerman found that the greater the expertise in climate research, the higher the agreement that humans were causing global warming. The other study led by William Anderegg surveyed public statements on climate change by 908 climate scientists. Both of these studies found a 97% consensus. Our paper mirrors the methodology of Anderegg’s study by examining published statements of climate scientists. We found that among scientists publishing climate papers that state a position on human-caused global warming, over 97% endorse the consensus.

Repeated surveys of scientists found that scientific agreement about AGW steadily increased from 1996 to 2009. This is reflected in the increasingly definitive statements issued by the Intergovernmental Panel on Climate Change on the attribution of recent global warming. Our analysis also finds that the consensus is strengthening with an accelerating number of scientists endorsing human-caused global warming in the peer-reviewed literature.

A non-peer-reviewed analysis also examines the level of consensus in the peer-reviewed literature, examining “global warming” or “global climate change” papers published from 1991 to 9 November 2012. This analysis was actually the first stage of The Consensus Project, conducted by Jim Powell in collaboration with John Cook and Dana Nuccitelli. Powell’s analysis found 24 explicit rejections of human-caused global warming among 13,950 articles. This is consistent with the results in our paper, which provides peer-reviewed confirmation of Powell’s analysis.