Arguments
Measure the climate consensus yourself with our Interactive Rating System
Posted on 17 May 2013 by John Cook
The Consensus Project was a long, ambitious effort by many volunteers, lasting 12 months from beginning to submission of our paper to peer-reviewed journal Environmental Research Letters. The project involved citizen science from start to finish - from the rating of the abstracts to the collection of scientists' emails to crowd-funding the journal fee to make the paper free to the public. It was an enormous collaborative effort that the entire Skeptical Science community contributed to. The effort has resulted in strong media interest including a tweet from President Obama.
We want our results to be transparent and replicable, so that anybody can quantify the scientific consensus on human-caused global warming for themselves. Thus we've created an interactive rating system that lets Skeptical Science readers rate the abstracts from The Consensus Project. You can then compare your ratings to the results from Quantifying The Consensus. Note that your ratings are private - no specific ratings will be publicly attributed to individuals.
All papers receive two types of ratings - the category of research and the level of endorsement of anthropogenic global warming (AGW). Here are general definitions of each category and endorsement level, although we've also provided a more detailed set of guidelines.
Level of Endorsement
- Explicit Endorsement of AGW with quantification
- Explicit Endorsement of AGW without quantification
- Implicit Endorsement of AGW
- Neutral
- Implicit Rejection of AGW
- Explicit Rejection of AGW without quantification
- Explicit Rejection of AGW with quantification
Categories
- Impacts (Effects and impacts of climate change on the environment, ecosystems or humanity)
- Methods (Focus on measurements and modeling methods, or basic climate science not included in the other categories)
- Mitigation (Research into lowering CO2 emissions or atmospheric CO2 levels)
- Not Related To Climate (Social science, education, research about people’s
views on climate) - Opinion (Not peer-reviewed articles)
- Paleoclimate (Examining climate during pre-industrial times)
The purpose of the interactive rating system is to replicate our experience of reading and categorising peer-reviewed climate research, to gain a deeper insight into our Quantifying The Consensus paper by participating in the process yourself. Check out the diversity of climate research on offer and try for yourself the categorisation of the papers' abstracts. All papers from our analysis are available for rating except the ones that had no abstract (47 papers). Please share your thoughts in this comments thread on the rating system, the categorisation guidelines, the climate research and other thoughts related to our Quantifying The Consensus paper.
Depending on how popular this feature is, I hope to continue adding features to the interactive rating system. Some possible features include visualisations of your ratings and a leaderboard of most productive raters.
Very good. If someone thinks the ratings were not adequate, they can see for themselves.
Oh my.
That's a terrible idea. I could get addicted to such a "game". (I find myself having to do _this_ lot of ratings so I can move to the next page just to see what the next interesting/ odd/ peculiar/ fascinating assortment of papers is going to be. The occasional downright weird item is just the cherry on the sundae.)
I had some trouble using it. When I moused over the 4th or 5th paper title, the abstract would appear so far up in the page that it did not appear in my screen.
I use Google Chrome 26.0.1410.64 m
Alexandre @3, I had the same problem but fixed it by reducing the zoom back to 100%. You may need to go a little lower.
Love it, love the ongoing improvements.
A suggestion - how about a category for articles relating to the effects of factors other than GHG emissions on climate? I came across an article on clouds that was hard to categorize.
Another abstract was an environmental impact study of engineered water recirculation for fish farming, and one of the impacts was "GWP (global warming potential)." I wasn't sure if it was safe to assume that the GWP was due to GHG emissions related to generating the mechanical power to recirculate the water, or if it could be something else like albedo or CO2 uptake effects of tinkering with bodies of water and/or land. (I suppose if it were CO2 uptake effects, that would be an implicit endorsement of AGW, because if CO2 uptake affects climate, than so do emissions.)
Another possible article topic that could be hard to categorize would be climate effects of sulfate/aerosol emissions (as by-products as opposed to deliberate geoengineering, which would fit nicely in mitigation).
I suppose the best place for this type of article right now is "methods," in the sense of the category description "basic climate science not included in the other categories." But using "methods" as a catch-all category in this manner might obscure the statistics about articles that really get into the nitty-gritty details of data selection, gathering, and analysis.
Hard to categorise? Yup. I studiously kept on checking that I was doing the ratings correctly. And worried that I was getting it wrong in some way or another. Now I've checked mine against the project and every single ratings number I gave was the same as theirs.
But the categories? Not hopelessly jumbled, but distinctly unwonderful in places. I'll now be using my previous scores to inform my future categorisations.
Only done a handful, but surprisingly, also 100% in agreement. Hard do find something to quibble about.
Good job.
To observe the obvious, there's quite a lot of buzz on this. One thing I noticed was that the people attacking the article, are mostly not really attacking the article; they are attacking the secondary sources reporting on the article, on points that are addressed in the article itself. And, they are reduced to quibbling over semantics without at all touching the observation that researchers rejecting AGW are a very, very small minority.
The "Peer review process was corrupted" zombie appears to be a bit more animated than normal. Yeah, either there has been a multi-national, multi-decade, coordinated effort to keep dissenting views from being published (nevermind the fame that could be had for successfully overturning established science, and being the journal in which it was published), or it is very difficult to write a dissenting article without the article containing egregious flaws.
Thanks a lot for the search possibility.
I tried to check the total number of papers in Endorsement Level '8. Undecided', but got no matches. Endorsement Level 4 'No position' gives 8269 which is much higher than the 7930 mentioned in the Cook et al paper.
Are the level 8 papers per accident incorporated in the level 4 papers?
JosHag @8, the total number of endorsement papers on the database exceeds the numbers mentioned in the paper as well. Combined there is an excess of 335 papers. Interestingly in the consensus project paper, it says:
There were a total of 335 "not climate-related" and "no abstract" papers, suggesting these have been retained in the data base we are searching.
@Tom Curtis #9, the total number of papers in the search option is 12280 (enter SQL wildcard % at 'Search Term') and before I start a search it says that 12,464 papers are present. So, you are probably right that (almost) all the papers have been retained in the database. The difference between the two numbers is 184, almost the same as the 186 you mention. A coincidence?
JosHaq @10, not knowing how to search for wildcards, I merely searched for "e" with essentially the same results (12279 total abstracts). The difference between that and the 12,465 papers recovered by the ISI search according to the paper would indeed be the 186 non-peer reviewed papers. (Or at least that seems very likely.) I have no explanation for the difference between the wildcard search and my "e" search (unless there is an abstract with no "e"s), nor why the difference in the total papers between search engine and paper.
I can't imagine an abstract with no e's, unless someone was having a bet. A paper with no abstract and no e's in the title might be the case.
Kevin C @12, well picked. The "missing" paper from an "e" search turns out to be:
Agricultural Impacts Of Global Warming - Discussion
Authors: Innes, R; Kane, S (1995)
Journal: American Journal Of Agricultural Economics
Category: No Abstract
Endorsement Level: 4. No Position