Arguments
Watts' New Paper - Analysis and Critique
Posted on 2 August 2012 by dana1981, Kevin C
"An area and distance weighted analysis of the impacts of station exposure on the U.S. Historical Climatology Network temperatures and temperature trends"
Paper authors: A. Watts, E. Jones, S. McIntyre and E. R. Christy
In an unpublished paper, Watts et al. raise new questions about the adjustments applied to the U.S. Historical Climatology Network (USHCN) station data (which also form part of the GHCN global dataset). Ultimately the paper concludes "that reported 1979-2008 U.S. temperature trends are spuriously doubled." However, this conclusion is not supported by the analysis in the paper itself. Here we offer preliminary constructive criticism, noting some issues we have identified with the paper in its current form, which we suggest the authors address prior to submittal to a journal. As it currently stands, the issues we discuss below appear to entirely compromise the conclusions of the paper.
The Underlying Problem
In reaching the conclusion that the adjustments applied to the USHCN data spuriously double the actual trend, the authors rely on the difference between the NCDC homogenised data (adjusted to remove non-climate influences, discussed in detail below) and the raw data as calculated by Watts et al. The conclusion therefore relies on an assumption that the NCDC adjustments are not physically warranted. They do not demonstrate this in the paper. They also do not demonstrate that their own ‘raw’ trends are homogeneous.
Ultimately Watts et al. fail to account for changing time of observations, that instruments change, or that weather stations are sometimes relocated, causing them to wrongly conclude that uncorrected data are much better than data that takes all this into account.
Changing Time of Observations
The purpose of the paper is to determine whether artificial heat sources have biased the USHCN data. However, accounting for urban heat sources is not the only adjustment which must be made to the raw temperature data. Accounting for the time of observations (TOB), for example, is a major adjustment which must be made to the raw data (i.e. see Schaal et al. 1977 and Karl et al. 1986).
For example, if observations are taken and maximum-minimum thermometers reset in the early morning, near the time of minimum temperature, a particularly cold night may be double-counted, once for the preceding day and once for the current day. Conversely, with afternoon observations, particularly hot days will be counted twice for the same reason. Hence, maximum and minimum temperatures measured for a day ending in the afternoon tend to be warmer on average than those measured for a day ending in the early morning, with the size of the difference varying from place to place.
Unlike most countries, the United States does not have a standard observation time for most of its observing network. There has been a systematic tendency over time for American stations to shift from evening to morning observations, resulting in an artificial cooling of temperature data at the stations affected, as noted by Karl et al. 1986. In a lecture, Karl noted:
"There is practically no time of observation bias in urban-based stations which have taken their measurements punctually always at the same time, while in the rural stations the times of observation have changed. The change has usually happened from the afternoon to the morning. This causes a cooling bias in the data of the rural stations. Therefore one must correct for the time of observation bias before one tries to determine the effect of the urban heat island"
Note in Watts Figure 16, by far the largest adjustments (in the warming direction) are for rural stations, which is to be expected if TOB is introducing a cool bias at those stations, as Karl discusses.
Instruments Change, Stations Move
As Zeke Hausfather has also discussed, the biggest network-wide inhomogeneity in the US record is due to the systematic shift from manually-read liquid-in-glass thermometers placed in a louvred screen (referred to in the U.S. as a Cotton Region Shelter and elsewhere as a Stevenson screen) to automated probes (MMTS) in cylindrical plastic shelters across large parts of the network in the mid- to late-1980s. This widespread equipment change caused an artificial cooling in the record due to differences in the behaviour of the sensors and the sheltering of the instruments. This is discussed in a number of papers, for example Menne et al. 2009 and 2010, and like TOB does not appear to be accounted for by Watts et al.
Additionally, the Watts paper does not show how or whether the raw data were adjusted to account for issues such as sites closing or moving from one location to another. The movement of a site to a location with a slightly different mean climatology will also result in spurious changes to the data. The Watts paper provides no details as to how or whether this was accounted for, or how the raw data were anomalised.
Quite simply, the data are homogenised for a reason. Watts et al. are making the case that the raw data are a ‘ground truth’ against which the homogenisations should be judged. Not only is this unsupported in the literature, the results in this paper do nothing to demonstrate that. It is simply wrong to assume that all the trends in raw data are correct, or that differences between raw and adjusted data are solely due to urban heat influences. However, these wrong assumptions are the basis of the Watts conclusion regarding the 'spurious doubling' of the warming trend.
The Amplification Factor
The conclusion regarding the lower surface temperature warming trend is also at odds with the satellite temperature data. Over the continental USA (CONUS), satellites show a 0.24°C per decade warming trend over the timeframe in question. According to Klotzbach et al. (2010), which the Watts paper references, there should be an amplification factor of ~1.1 between surface and lower troposphere temperatures over land (greater atmospheric warming having to do with water vapor amplification). Thus if the satellite measurements were correct, we would expect to see a surface temperature trend of close to 0.22°C per decade for the CONUS; instead, the Watts paper claims the trend is much lower at 0.155°C per decade.
This suggests that either the satellites are biased high, which is rather implausible (i.e. see Mears et al. 2011 which suggests they are biased low), or the Watts results are biased low. The Watts paper tries to explain the discrepancy by claiming that the amplification factor over land ranges from 1.1 to 1.4 in various climate models, but does not provide a source to support this claim, which does not appear to be correct (this may be a reasonable range for global amplification factors, but not for land-only).
A discussion between Gavin Schmidt and Steve McIntyre on this subject led to the conclusion that the land-only amplification factor falls in the range of 0.78 to 1.23 (average over all global land areas), with a model mean close to 1 (using a script developed by McIntyre on 24 different models). Note that McIntyre is a co-author of Watts et al., but has only helped with the statistical analysis and did not comment on the whole paper before Watts made it public. We suggest that he share his land-only amplification factor discussion with his co-authors.
Another important consideration is that the amplification factor also varies by latitude. For example Vinnikov et al. (2005) found that at the CONUS latitude (approximately 40°, on average), models predict an amplification factor of approximately 1 (see their Figure 9). Note that this is the amplification factor over both land and ocean at this latitude. Since the amplification factor over land is less than that over the oceans, this suggests that the amplification factor over the CONUS land may even be less than 1.
Combining the latitude and land-only status of the CONUS, the amplification factor may very well be less than 1, but a range of values significantly lower than the 1.1 to 1.4 range used in the Watts paper would be reasonable.
Note also that as discussed above, the satellite data (like all data) are imperfect and are not a 'gold standard'. They are a useful tool for comparison in this study, but the satellite trends should not be assumed to be perfect measurements.
More Apples and Oranges
Watts et al. compare the best sited Class 1 and 2 stations (using their categorisation) to the total homogenised network. Strictly speaking, this is comparing apples and oranges; Watts' data are an inhomogeneous sub-sample of the network compared to a homogeneous total network. In practice, this methodological error doesn’t make much difference, since the homogenisation applied by NCDC produces uniform trends in all of the various classes.
However the use of a smaller network has disadvantages. When taking a gridded average of a sparse network, the impact of inhomogeneities is likely amplified and the overall uncertainty of variability and change in the timeseries increases.
The Class 1 & 2 sites used by Watts in this context represent just 20% of all the total CONUS network. The comparison of the raw Class 1 & 2 sites with the same network of homogenised data in Watts' own Figure 18 indicates likely inhomogeneities in that raw data. Watts et al. argue that the raw and adjusted Class 1 & 2 trends in Figures 18a and 18b are so different because "well sited stations are adjusted upward to match the already-adjusted poor stations," but this is simply not how the homogenization process is done. In reality the difference is likely due to the biases we have discussed above, indicating that the Watts raw data is inhomogeneous and influenced by these non-climate effects.
Determining whether or not the Class 1 & 2 raw data is homogeneous is therefore a key requirement of a revised manuscript. And since the Class 1 & 2 sites have been selected for good exposure, Watts et al. would need to show the cause of any statistical discontinuities that they find. This work has already been done by NCDC in the Menne at al. papers, which show influences from the range of factors discussed above, and not just urban influence.
The Watts final conclusion that adjusted temperature trends are 'spuriously doubled' (0.155°C vs. 0.309°C per decade raw vs. adjusted data) relies on a simple assumption — that the raw data must be correct and the homogenised data incorrect. There is no a priori basis for this assumption and it is unsupported by the literature.
Adjustments Make Little Difference Globally
While Watts et al. identify possible issues concerning the adjustments applied to station temperature records, it wisely makes no attempt to assess the global impact of the adjustments, which are beyond the scope of the work. Nonetheless, this is a significant question from a public interest perspective.
In order to answer this question, we willl try and estimate the maximum possible impact of station adjustments on the instrumental temperature record. To minimise the warming signal, we will use the simplest method for calculating a global temperature average - the CRU method, which is known to yield poor coverage at high latitudes and hence underestimate recent warming. Further, we'll assume that entirety of the data adjustments are wrong (ignoring proven bias corrections such as TOB). If we calculate land temperature averages from both the raw and adjusted data, we can see how much difference the adjustments make. The result is shown in the figure below (red and green lines).
Just to be absolutely sure, we can do a further calculation using just rural unadjusted data (using the GHCN station classifications) - the blue line.
While the adjustments do make a difference, the difference is small compared to the overall warming signal since 1979. Using a more sophisticated temperature calculation reduces this difference. Furthermore, we are only looking at land temperatures - 30% of the planet. Including the remaining 70% of the planet (the oceans which, if not precisely rural, are certainly not urban!) dramatically reduces the remaining impact of the GHCN adjustments. Indeed, comparison of warming trends over the oceans and over large inland lakes (including the North American Great Lakes) shows a high degree of consistency with terrestrial trends. Warming over the oceans and lakes is presumably not due to urbanisation.
The entire CRU-type calculation requires 65 lines of python code (by comparison, a modern airliner requires upwards of a million lines of code to fly). The code is available below.
Show code
Many others have done this comparison, including Caerbannog and Zeke Hausfather. Fawcett et al (2012) provide a comprehensive assessment of the sensitivity of Australian temperature trends to network and homogenisation choices, including comparison with an unhomogenised gridded analysis of Australian temperatures. Furthermore, the BEST project has obtained a similar result with a different, independent implementation of the station homogenization algorithm. It would be surprising if an independent approach were to yield a similar but incorrect result by chance.
Do the overall adjustments make a difference? Yes. Are they justified? Yes, according to the body of scientific literature. Watts raises a scientific issue, but one which only affects part of the adjustment. Does it matter? Not very much. Even if the entirety of the adjustments were wrong, we still see unprecedented warming over the past 40 years. And there is certainly not a factor of two difference between global warming trends in the raw and adjusted data.
Constructive Criticisms
It's worth noting that Peter Thorne of NCDC was interviewed by Andrew Revkin, and discussed three papers which NCDC has recently published (see here, here, here). In the first of those linked papers, they actually concluded that there likely remains a residual cool bias in the adjusted data, and that the adjusted data are consistent with reanalysis data (detailed in the third linked paper). Watts et al. do not address these papers. Ironically Watts responded to that interview by saying that Thorne needs to get out into the real world, but it is Watts et al. who have not accounted for real world effects like TOB, station movement, instrument changes, etc.
In its current form, the Watts paper contains little in the way of useful analysis. There are too many potential sources of bias which are not accounted for, too many apples-to-oranges comparisons, and they cannot draw any conclusions about urban heat influences until their data are homogenized and other non-climate influences are removed.
The primary conclusion of the paper, aside from not being supported by the analysis, is simply implausible. The CONUS surface warming trend proposed by the Watts paper appears to be inconsistent with the satellite observations, and overall global trends in raw data do not differ dramatically from those in the adjusted data. Comparing raw to adjusted data globally shows a rather small difference in long-term trends; far smaller than a factor of two.
The flaws we have identified entirely compromise the conclusions of the paper. Ultimately Watts et al. assume that all adjustments are 'spurious' unless due to urban heat influences, when in fact most of their identified discrepancy likely boils down to important adjustments for instrumental changes, TOB, and other influences they have not accounted for in their analysis. Watts et al. attempt to justify their assumption by asserting "well sited stations are adjusted upward to match the already-adjusted poor stations," but this is simply not how the homogenization process is done.
Fortunately McIntyre has acknowledged that TOB must be considered in their analysis, as has Watts, which is a good start, but they must also account for the other biases noted above in order to draw any valid conclusions about urban heat influences.
In conclusion, Watts et al. of course deserve the right to try to make their case in the peer-reviewed literature, however implausible that case appears to be. Therefore, we hope they will consider addressing the important concerns detailed above before they submit the paper to a journal. Otherwise we suspect the paper will not fare well in the peer review process. With said caveats carefully addressed and the conclusions amended if and where necessary, the paper has the potential to be a useful contribution to the climate science literature.
Evan Jones @87 states:
For much the same reason that a general gets the credit for winning (or losing) a battle. Although I have refined the hypothesis somewhat and slogged through much mud, he (Anthony Watts) is the one that found the mud in the first place. I am just the infantry. And he has also done much heavy lifting of his own in this, as well.
Yes indeed. Anthony Watts has made a career of finding the mud an wallowing in it.
I'm afraid I still don't understand.
If you draw a trend from 1998-2008 the 'heat sinks' react one way, but if you draw the trend from 1998-2015 they react differently?
How can a heat sink react to a trend line? A trend line isn't a physical thing. It's an exploratory tool. There is no way a heat sink can be reacting to temps a decade ago.
Tristan:
Think "smoke and mirrors".
John, I don't believe Evan is trying to deceive.
I believe he is trying to do something without any of the multi-disciplinary background generally required to do such a thing (he's stated that he's self-taught) and therefore doesn't recognise that he's come-a-cropper. I admire his efforts, but he needs to realise that his interpretation of the data has lead to some unphysical conclusions. The response to that is not "invent new physics", which seems to be what he has done, but rather "figure out why my sampling is spitting out these results".
1999-2008 is too short a time to obtain significant differences with a truncated data set without strong statistical analysis. Eyeballing a graph does not count. You will be strongly criticized for too short a period of time.
We do that strong statistical analysis. The divergene shows in our subsets as well as in our full sets. Actually, the graphs look much the same — until you drop in the trendlines. That is because the divergence is gradual over time, not a result of jumps in the data. The bar graphs show it "all at once", and therefore more starkly. Some are sown in the figures, all of it will be archived, and review is welcome. As far as I am concerned, the archive will be considered to be a living document, at least by me. There are always a few more stations to be found (perhaps 1/3 of them unperturbed and therefore "valid", if the past is any guide).
There is also the added advantage that a much larger number of stations are unperturbed for that period. For example, a station that moved in 1995 would not be eligible for the full set but could be included in the 1999 - 2008 set. So the shorter interval is compensated for by the larger sample size. And, as I say, since we are not in any way trying to extrapolate from that interval that AGW is not occurring, the 30-year rule does not apply.
We use the only two intervals available to us (reasons in pervious comment). We address the CRN vs. COOP issue. I can answer criticisms regarding not using a longer set. In order to assess microsite, one would have to split it all into warming, cooling, and flat intervals, being careful to pick the correct start and end points (always a reality MoE and always a point of argument).
FWIW, regarding the flat trend since CRN went official, I consider the "pause" to be an artifact of CO2 pushing up and negative PDO pushing down. But the period we picked was 1979-2008, which is nearly all positive PDO. That was the point, though. We want to measure the effects of warming on stations. To do that, one would naturally want to select a period of maximum warming. Both CO2 and positive PDO piled into one. We also include a 1979-1998 version. Warming/Cooling is the primary indicator, and the greater the trend, the better one can generally measure effects upon it. And I dug up a nice distinct cooling period (shorter but more sample sixe) to observe the effects of cooling. Also primary. A non-divergence in a flat trend is secondary, although important because it is consistent with and important to the hypothesis.
This is not cherrypicking because we are not in any way saying that our findings for these intervals are in any way typical of the whole. In a sense, these intervals are selected precisely because they grant a maximum play for heat sink effect on the sample to be measured, and are therefore — not — typical of the whole, by design. But it important to note this and not claim any of these intervals as "typical".
Our sample size is lower than our 2012 set. We don't have the quite same subsets (but using the archived sheet, you could run them up). Our results for microsite are a bit warmer than in 2012. Anomalizing the data was a large factor in this, as was dropping TOBS-biased stations. Moves were pretty much a push, so far as I could tell.
Evan Jones @82... "That is exactly what our "purposes" were, and that is exactly what we did. And I have been addressing the resulting (exraordinarily valuable) independent review ever since. It takes time."
This strikes me as a post hoc rationalization. Your example would be a researcher sending out pre-press copies to colleagues, not a public press release stating your earth-shattering research is about to be published.
Perhaps the problem here is that you're trying to produce a research paper with a specific conclusion that the surface station data is wrong. You even say as much @81 where you state:
"I claim...that [US adjustments are wrong] — with very strong evidence — but WITHOUT the conspiracy theories. I do not think this is fraud, merely an error. An understandable one, one I might easily have made myself."
You're starting from a conclusion that you clearly could not prove before and now have spent 4 years and many thousands of hours attempting to locate something that very likely is incorrect. If there was "very strong evidence" then it wouldn't take 4 years and thousands of hours!
Given this record you have of filling up comment threads to no purpose, can you make clear your purpose here? Are you just after a bit of a chat? Are you here to announce the imminent pre-release of Watts et al.(unsubmitted) for a second time? Do you wish to share some specific aspect of its content with us denizens here at SkS (& if so, we would benefit from knowing what)? Or are you just trolling it?
My purpose here is as stated. To elicit independent review. To answer all questions and field valid questions. When the boyz on the panel present the snappy questions, I'd just as soon have the snappy answers. That is the benefit I accrue.
Your benefit is that you get to know the current state of the paper. You share in those snappy answers and can come up with snappy rebuttals. Define the weak points. Be a constructive part of the process. I find always learns more from one's critics than from one's clique. A sword that cuts both ways, but consistent with scientific method.
When I corresponded with Doc. VV, whom I highly esteem, I got to know how/why homogenization can bomb. He got to know we were for real and is aware, at least, of a potential red flag. (though he is putting his stock in step jumps). We both got what we were after. We both profited from the exchange.
Evan Jones @107... An internet comments forum is a perfectly awful and inefficient way to have a fully substantive discussion on a research paper.
My suggestion would be for you to send John Cook a full copy of your paper and ask if the SkS team will review it. If accepted, we would review it privately and send back our comments.
Tristan:
Yeah, Evan appears to be a straight shooter. It's a shame that he's working with people who are not.
Evan:
Can you tell us what "game", Anthony Watts is going to change with this paper?
Evan Jones @107.
Okay. Your stated purpose here at SkS? "To elicit independent review. To answer all questions and field valid questions."
So let us have a sight of what the dickens it is we are supposedly reviewing and asking questions about!! While the suggestion of Rob Honeycutt @108 would be "consistent with (the) scientific method," I'm sure even on planets like Wattsupia or Climateetcia that indulge in trial-by-numpty, the folk are allowed a sight of the object under discussion.
Evan Jones,
Tamino and Ramsdorf have both put out blogs that show there are no cooling (or flat) periods in the global record for the past 50 years. There cannot possibly be a statisticly significant cooling or flat period in the US record which is noisier. You are fooling yourself. Your claims will not stand an unbiased review. Fortunately, you have Watts new "journal" to publish in and do not need peer review.
This discussion is a waste of time. I have given you feedback and you assure me that you have covered the bases without showing me any data or analysis. Your claim at 99 of 0.00000% chance of overlap cannot be true. Such data is un-natural for climate. No-one has that many significant figures. You must find someone who is qualified to analyze your data correctly.
I am no longer going to respond to your posts. Please go somewhere else and waste their time.
While we wax eloquently on comment threads such as this, thanks to AGW, the Earth’s climate system is going to hell in a handbasket at an accelerating and alarming rate.
In an editorial published today, Marcia McNutt, Editor-in-Chief Science Journals, eloquently states our predicament in a single paragraph:
The beyond-two-degree inferno, Editorial by Marcia McNutt, Editor-in-Chief Science Journals, July 3, 2015
Yes indeed. Anthony Watts has made a career of finding the mud an wallowing in it.
He found this mud. Makes very good wallowing. I've been doing that for the last five years, so I should know.
If you draw a trend from 1998-2008 the 'heat sinks' react one way, but if you draw the trend from 1998-2015 they react differently?
The trend from ~2001 to 2015 is flat. Therefore, there will be no divergence over time for that period. What we want to examine is if there is a divergence during a strong cooling period and if so, how much that divergence is. Including a long string of flat data on the end muddies the signal and places 2008 near the middle, which washes out the slope.
For that reason, we also supply 1979-2008 data in order to show a warming-only period.
I repeat, we are not trying to say that our 1999-2008 series is representative of the US or global longterm trend. We are only using it to demonstrate the effect of heat sink during a period of significant cooling.
How can a heat sink react to a trend line? A trend line isn't a physical thing. It's an exploratory tool. There is no way a heat sink can be reacting to temps a decade ago.
A heat sink's effect on any datapoint is relevant in relation to its given place in a given series. Start point, end pont, trend. If a relatively cooling period occurs near the start of the time series, the trendline is increased. If it occurs duing the end of the time series, it decreases the trendline.
If a blip occurs smack in the middle of a time series, it doesn't affect the trend a hoot in hell no matter how high or how low it is. But a step change has the greatest effect in the middle of a time series and the smallest at either end.
It's not about a sensor, or even a datapoint. It's about what point the datapoint occurs in the series, be it running warm or cool.
During a warming phase, the heat sink's temperature rises dispropotionately. It is a function of that difference over time that is the spurious amount added to the sensor reading from the start to end point. If it is a cooling trend, the effect works in reverse.
[PS] Hypothesizing about an unseen future paper is fruitless and frankly offtopic. I would strongly suggest that no further discussion happen on this subject until there is an actual paper to discuss.
Tamino and Ramsdorf have both put out blogs that show there are no cooling (or flat) periods in the global record for the past 50 years.
There is perceptible Tmean cooling in the US from 1999-2008.
Class 1\2 (Raw+MMTS adjustment): -0.135C/decade
Class 3\4\5 (Raw + MMTS adjustment): -0.309C/decade
Class 1\2 (NOAA-adjusted): -0.232C/decade
Class 3\4\5 (NOAA-adjusted): -0.398C/decade
As you can see, the cooling is exaggerated by poor microsite and made worse by adjustment. Same happens the other way around, which is most of the time.
[PS] Hypothesizing about an unseen future paper is fruitless and frankly offtopic. I would strongly suggest that no further discussion happen on this subject until there is an actual paper to discuss.
I don't see why. But as you wish.
[RH] ***Note to all.*** This conversation has to stop until an actual paper is provided. All follow up comments will be deleted until such time Evan produces a paper for us to read.