Arguments
An estimate of the consensus project paper search coverage
Posted on 10 June 2013 by Ari Jokimäki
As part of my involvement in the consensus project (TCP) that recently published its results, I looked into some aspects of the data which were not part of the final paper. One thing I did was look into what proportion of the literature was covered by the project.
Here's the description of the search from the paper:
"In March 2012, we searched the ISI Web of Science for papers published from 1991–2011 using topic searches for ‘global warming’ or ‘global climate change’. Article type was restricted to ‘article’, excluding books, discussions, proceedings papers and other document types. The search was updated in May 2012 with papers added to the Web of Science up to that date."
This resulted in 12,465 papers, but after eliminating papers that were non-peer-reviewed, not related to climate, and papers without abstracts, the resulting number of papers was 11,944.
In order to check the completeness of the search, we should compare the search results to some other known sample. To me, the obvious sample for comparison is found from IPCC fourth assessment report (AR4) reference lists because I think they cover the subject reasonably well. However, it should be noted that IPCC reference lists don't contain all the papers on the subject, but they are only a subset just like the sample in TCP. The comparison between the TCP sample and IPCC reference lists presented below only shows if TCP paper search did not cover the subject well.
I made a cross-comparison between the TCP sample and the reference lists of AR4. I didn't go over all AR4 chapters, though, but only few selected ones. In TCP, papers were categorized to different subject areas, so I took equivalent chapters for each subject area from AR4 for comparison. For methods, I selected Working group I chapter 9 "Understanding and Attributing Climate Change". Obvious choice for paleoclimate was Working group I chapter 6 "Palaeoclimate". I used Working group II chapter 1 "Assessment of observed changes and responses in natural and managed systems" for impacts. For mitigation, I used Working group III chapter 7 "Industry" and chapter 8 "Agriculture" (two chapters in order to keep somewhat similar paper count to other comparisons).
The reference lists of AR4 chapters have some entries that are out of scope for TCP. Such entries are non-peer-reviewed documents (for example reports, websites, books), comment papers (comments and replies), and papers out of TCP timeframe (1991-2012). These were excluded from the comparison and from AR4 paper count. Table below shows the results of the comparison. Rows of the table are: category in TCP, equivalent AR4 chapter(s), number of relevant entries in AR4, number of papers found in TCP of the relevant AR4 entries, and the coverage percentage (and its standard error) of TCP paper search. Last column gives the overall estimate calculated by adding all the four different subject areas together.
| Category | methods | paleoclimate | impacts | mitigation | overall |
| AR4 chapter | WG1 CH9 | WG1 CH6 | WG2 CH1 | WG3 CH7+8 | overall |
| Paper count | 493 | 542 | 570 | 318 | 1923 |
| Found | 53 | 26 | 68 | 21 | 168 |
| Coverage % | 11 ± 2 | 5 ± 1 | 12 ± 2 | 7 ± 2 | 8.7 ± 0.7 |
The overall coverage percentage is estimated to be 8.7 %, which means that to acheive complete coverage we would have to have looked at 140,000 papers! The search found more papers from methods and impacts subject areas. At least for methods (containing basic climate science papers) this is expected, but it would make more sense if methods papers would have larger coverage percentage than impacts papers. But it may be that the authors of impacts papers are more inclined to mention the phenomenon causing the impacts they are studying, whereas for climate scientists it might go without saying that they are studying climate change related issues and they are perhaps more concentrated on studying the little details of the issue.
Somewhat surprising is low coverage percentage for paleoclimate papers. Perhaps they just don’t mention global climate change or global warming that much. Mitigation is a bit higher than paleoclimate, which might be understandable as mitigation is done because of global warming, so mitigation papers are perhaps expected to use the term more often than paleoclimate papers. It also makes some sense that impacts papers have higher coverage than mitigation papers, but I didn’t expect the gap between them to be this large (before seeing the numbers I considered them as somewhat equal in this sense).
The results of this comparison can also be used in other way. We can use the numbers above to estimate the total number of all published global climate change and global warming related papers between 1991 and 2012. The reasoning is this: TCP paper search found 8.7 % of papers referenced in AR4. Total paper count from TCP paper search is 11,944. If 11944 represents 8.7 % of papers, then total number of papers must be (11,944 * 100 %) / (8.7 %) = 136,693.
I calculated numbers similarly for all subject areas. Additionally, I estimated total numbers for endorsement and rejection papers between 1991 and 2012. Table below shows the results of these. Rows of the table are: category, number of papers in TCP in the category, coverage % calculated above, estimated total number of papers as described above, number of endorsement papers in TCP in the category, estimated total number of endorsement papers, number of rejection papers in TCP in the category, estimated total number of rejection papers.
| Category | methods | paleoclimate | impacts | mitigation | overall |
| TCP paper count | 1991 | 785 | 5780 | 3386 | 11942 |
| Coverage % | 11 | 5 | 12 | 7 | 8.7 |
| Total paper count | 18520 | 16364 | 48450 | 51274 | 136693 |
| TCP endorsements | 581 | 169 | 1235 | 1912 | 3897 |
| Total endorsements | 5404 | 3523 | 10352 | 28953 | 44607 |
| TCP rejections | 53 | 4 | 17 | 3 | 77 |
| Total rejections | 493 | 83 | 143 | 45 | 881 |
As mentioned above, the estimate for all published papers relating to global climate change and global warming is about 137,000 between 1991 and 2012. Reading one of them each day would take 375 years. If you would be a climate scientist wanting to read all papers relating to global climate change and global warming (in order to keep up with the trade), and if your career would last, say, 50 years, you would need to read 7 papers each day. That’s doable, isn’t it?
As we all know, there are lots and lots more endorsement papers than rejection papers, but interesting thing here might be the total number of rejection papers, which I estimate to be 881. There are actually quite many of them. Perhaps one of them is the one that turns around the whole of climate science. I wouldn’t bet on it, though.
Your comment fit well with my mass balance work on glaciers, "impacts papers are more inclined to mention the phenomenon causing the impacts they are studying, whereas for climate scientists it might go without saying that they are studying climate change related issues and they are perhaps more concentrated on studying the little details of the issue." All of my results indicate the impacts of global warming on glaciers, but the focus in word choice in the abstract will be more on negative mass balance, glacier retreat due to warming in the region. Since we are studying a locality that is the focus, and so it sometimes goes unsaid that it is all driven by global warming. .
"Perhaps one of them is the one that turns around the whole of climate science."
Then it would need to be a thick one, since it needs to overthrow a sizeable chunk of the known, and settled, science.
Or, of course, the explanation is dispersed over a number of papers.
Either way, both cases are testable, either look into the thick research papers and see if they provide an alternative explanation which covers all the known detected anomalities. Or see if there are number of papers that does the same, being referred by each other.
After all, ~900 papers should not be that hard to read through. That's less than 3 years with 1 paper per day and I'm pretty sure you can automate this.
What are the odds of pseudosceptics picking up this task?
Mspelto, thanks for the insider view. I think there are lot of papers in each category, where things are left unsaid just because the scope of the study.
Lanfear, that last comment was made tongue at least halfway in the cheek. We don't yet have the ~900 papers in our hands but it would need some digging to find out them all. We would need to rate every climate paper there is in order to find them. Having rated a few papers during this project, I know how tiring the rating process can be, so if we would like to rate large amount of papers, we would need lot more raters.
To use consensus to justify any scientific position is to stand on very shaky ground. We should abandon this argument. The evidence is very strong and speaks for itself. If the consensus was that climate change is nonsense, would this make it any more true.
William:
The existence of the scientific consensus on AGW is not being used to justify the position to other climate scientists. It's being used to make it clear to members of lay public, who don't have the skills and knowledge to understand the evidence directly, that there really isn't a debate still going on about AGW within the community of experts, contrary to the insistence of the deniers.
Genuine skeptics who are not themselves experts on climate science will acknowledge and respect the judgement of those who are. On another thread I linked to the text of a talk by Texas State Climatologist John Nielsen-Gammon at last year's AGU meeting, titled Scientific Meta-Literacy, I think it's germane to your comment as well.
william @4, maladapted @5, the argument that there is a concensus of climate scientists that AGW is happening has not been used by SkS, and should not be used to anyone. The argument used by SkS is simply that fake "skeptics" claim that there is no consensus, but on the contrary, evidence shows that there is. In turn, the claim by fake "skeptics" that there is no consensus is used as the premise in a number of invalid arguments such as - "There is no consensus, therefore there is reasonable scientific doubt about AGW", and "There is no consensus, therefore both sides of the evidence should be presented equally to the public", and "There is no consensus, therefore it is too soon to initiate any policy on climate change".
Consensus is not a measure of scientific validity but is the only rational basis for determining policy.
Ari@3
"that last comment was made tongue at least halfway in the cheek."
Yes, and from a strictly scientific perspective we could do the actual study also.
However I do have a strong sense that this study would only confirm what the current science says, ie. that the AGW is real, so what it would cause among the pseudoskeptics is another wave of Lewandowsky-grade complains and conspiracy-theories.
That's why I would like to see that someone from the pseudoskeptic side would pick this up, since, as I noted, there are a couple of shortcuts that will help you narrow down the search. Not to mention that there is (or at least they like to claim so) a lot of 'skeptics' who can help sifting through the papers.
Funny thing that you came up with a number close to 900, wasn't this the number of 'peer reviewed' papers that poptech listed :)
I took a peek to that poptech list once, but I very quickly saw (as would anyone who has gotten familiar with the body of scientific literature on the subject) that it wasn't very good effort. It seems to me that just about any paper could be included to that list when looked at some specific angle. If he would have taken real effort to do it, the list might have been useful at least in some sense. Well, I guess it's also useful now, but only in propaganda sense.
As a volunteer reviewer, I found very few papers taking a strong position either way. Most that I reviewed, referred to climate change, without taking a stance as to its cause. I wonder if choosing the words "climate change" in an abstract is the best metric for this work.
Daniel, it is the standard term for the area of research (e.g. International Panel on Climate Change). The fact that most papers don't take an explicit stance on AGW is itself a useful finding (at least in terms of informing the public debate on climate change). For a start it refutes the myth that climatologists are promoting AGW in order to secure greater research funding for themselves. If that were true, all papers would take a stance on AGW somehow. However scientists are not actually like that, they follow where the science leads, sure they have particular interests, but if reality doesn't fit their current understanding of the physics, they are sensible enough to know that it would be a career limiting move to reject reality. It also highlights the fact that most climatologists are not actually working directly on attributing warming to AGW, neither of the two climate change related projects I have been involved with were directly concerned with determining the causes of climate change.
Dikran,
You seem to be echoing my concern in your last sentence. If a paper mentions climate change, but makes no mention of the cause, how can it be considered to support AGW? Even if the myth were true, many of the papers would still not likely take a stance, as most of these papers were not written by climatologists. The term "climate change" was not the issue in my post, but its selection as criteria for the paper.
Daniel, I suggest you read the paper, the criterion for determining whether a paper supports AGW are stated as clearly as one could reasonably expect. A paper on statistical downscaling (an area of climatology in which I have worked) is not a paper that seeks to determine whether climate change is anthropogenic or not, but it would not be unusual for such a paper to motivate the need for the study by mentioning in the abstract that the majority of climate change has ocurred due to fossil fuel use. In your view, would such a paper endorse AGW, and at what level according to the stated criterion for the TCP?
Daniel, the direction of research can be an indication of the paper's understanding of mainstream climate science. A team of herpetologists decides to study the long-term migration of a frog species. The team finds that the frog species is moving northward by about three kilometers per decade. The team writes the migration up and, in the discussion section, speculates about the future of the movement. The team assumes current mainstream estimates of future warming for the region, noting that the species may end up being caught between a rock and a hard place, since there is a natural barrier a few km to the north. That speculation is represented in the abstract. This paper explicitly endorses the mainstream theory (AGW), since the mainstream theory is the only one taken into account. The mainstream theory (is this what you refer to as a "myth"?) is anthropogenic global warming, and implicit in that theory is the continuation of warming as long as the greenhouse effect continues to be enhahnced.
True, it would be very difficult for such a paper to go against the mainstream view. After all, to do so would be to explicitly reject the mainstream theory of climate, and a reason would need to be provided. As science naturally becomes more interdisciplinary, trust is a major issue. The herpetologists must trust climate modeling to give them a realistic basis for useful speculation. In that way, the use of the mainstream view is a vote of confidence for the mainstream view. It's not simply "well, we were forced to use it." The herpetologists didn't have to use it. They could have simply relied on an extrapolation of their own data. Such an act would be less useful, even if it were tied to a physical mechanism (e.g. according to local data, it's getting warmer in this region).
The herpetologists would be especially driven toward speculation if they had long-term data, and the recent data suggested highly unusual activity. There's absolutely nothing wrong with such speculation. It reflects the concern the scientists have for their research areas. I can't count the number of times people have essentially tried to argue (toward me) that nature is conservative and any unusual behavior or "wild" speculation must be the result of scientists trying to make an extra buck or keep their jobs. No. If the basic inputs change (e.g. rising system energy storage), every element of the system changes.
By the way, what were you referring to as a myth?
Daniel, the consensus project did not use 'climate change' as criteria for anything. Preliminary sample was built using search phrases 'global climate change' and 'global warming'. And of course, simple mention of climate change (or specific search phrases used) doesn't suggest that paper supports AGW, hence a large proportion of neutral papers was found in the project.