Arguments
Climategate and the Freedom of Information (FOI) requests
What the science says...
The Independent Climate Change Email Review found the CRU scientists were unhelpful and unsympathetic to information requesters and at times broke FoI laws. However, CRU is a small research unit with limited resources, and they perceived the requesters were not acting in good faith. The same inquiry found the rigour and honesty of the scientists are not in doubt, and their behaviour did not prejudice the advice given to policymakers.
Climate Myth...
Freedom of Information (FOI) requests were ignored
"The emails suggest that the authors co-operated (perhaps the word is “conspired”) to prevent data from being made available to other researchers through either data archiving requests or through the Freedom of Information Acts of both the U.S. and the UK." (Pajamas Media)
Exhibit No. 1 of the climate conspiracy theory is a collection of emails stolen from the Climatic Research Unit (CRU) of the University of East Anglia (UEA), which appeared on the internet in November 2009. Though some of these "Climategate" emails can sound damning when quoted out of context, several inquiries have cleared the scientists. The most comprehensive inquiry was the Independent Climate Change Email Review.
One allegation arising from the emails (and arguably the only instance where there is actually a case to be answered) is that Freedom of Information requests received by CRU were wrongly denied. Meanwhile, defenders of CRU “have suggested that a number of these FoIA requests were inappropriate or frivolous.” [10.2]
Below I have reproduced some of the emails often quoted in support of these allegations (all were written by Phil Jones):
7/5/2004: Many of us in the paleo field get requests from skeptics (mainly a guy called Steve McIntyre in Canada) asking us for series. Mike and I are not sending anything, partly because we don’t have some of the series he wants, also partly as we’ve got the data through contacts like you, but mostly because he’ll distort and misuse them. Despite this, Mike and I would like to make as many of the series we’ve used in the [Reviews of Geophysics] plots available from the CRU web page.
2/2/2005: [D]on’t leave stuff lying around on ftp sites — you never know who is trawling them. The two MMs have been after the CRU station data for years. If they ever hear there is a Freedom of Information Act now in the UK, I think I’ll delete the file rather than send to anyone. Does your similar act in the US force you to respond to enquiries within 20 days? - our does! […] Tom Wigley has sent me a worried email when he heard about it—thought people could ask him for his model code. He has retired officially from UEA so he can hide behind that.
21/2/2005: I’m getting hassled by a couple of people to release the CRU station temperature data. Don’t any of you three tell anybody that the UK has a Freedom of Information Act!
27/4/2005: I got this email from McIntyre a few days ago. As far as I’m concerned he has the data — sent ages ago. I’ll tell him this, but that’s all — no code. If I can find it, it is likely to be hundreds of lines of uncommented fortran ! I recall the program did a lot more than just average the series. I know why he can’t replicate the results early on — it is because there was a variance correction for fewer series.
29/5/2008: Can you delete any emails you may have had with Keith re AR4? Keith will do likewise. […] Can you email Gene and get him to do the same? […] We will be getting Caspar to do likewise.
3/12/2008: When the FOI requests began here, the FOI person said we had to abide by the requests. It took a couple of half hour sessions — one at a screen, to convince them otherwise showing them what CA was all about. Once they became aware of the types of people we were dealing with, everyone at UEA […] became very supportive. […] The inadvertent email I sent last month has led to a Data Protection Act request sent by a certain Canadian, saying that the email maligned his scientific credibility with his peers! If he pays 10 pounds (which he hasn’t yet) I am supposed to go through my emails and he can get anything I’ve written about him. About 2 months ago I deleted loads of emails, so have very little — if anything at all.
10/12/2008: Haven’t got a reply from the FOI person here at UEA. So I’m not entirely confident the numbers are correct. One way of checking would be to look on CA, but I’m not doing that. I did get an email from the FOI person here early yesterday to tell me I shouldn’t be deleting emails — unless this was ‘normal’ deleting to keep emails manageable! […] According to the FOI Commissioner’s Office, IPCC is an international organisation, so is above any national FOI. Even if UEA holds anything about IPCC, we are not obliged to pass it on, unless it has anything to do with our core business — and it doesn’t. I’m sounding like Sir Humphrey here! McIntyre often gets others to do the requesting, but requests and responses all get posted up on CA regardless of who sends them.
The general allegation is that CRU incorrectly denied FoI requests. In particular, the Review focused on the question of whether UEA’s formal processes for dealing with FoI requests were “fair and impartial”.
The Review Team interviewed the relevant UEA and CRU staff, as well as representatives of the Information Commissioner’s Office (ICO). UEA’s FoI process is centred around their Information Policy & Compliance Manager (IPCM). In the two years after current laws came into effect at the start of 2005, no requests for information were logged with the IPCM, though we know from the emails that there were such requests. We know from the IPCM log that CRU received four requests in 2007, two in 2008, and one in the first half of 2009 (four were fully granted and three rejected).
Then came the storm. Between 24 July and 28 July, CRU received no less than 60 FoI requests, and 10 more between 31 July and 14 August. The requesters demanded access to both raw temperature station data and any related confidentiality agreements. The Review found evidence that this was an organized campaign (one request asked for information “involving the following countries: [insert 5 or so countries that are different from ones already requested]”). The Review says “such orchestrated campaigns [have] literally overwhelming impacts on small research units.”
The Review found there was “insufficient priority given from the UEA centre to motivating staff and to prompting continuing education” about their legal requirements under FoI law. Similarly, they found “a lack of engagement by core CRU team”, as well as “a tendency to assume that no action was required until precedents had been set”. Some of the emails suggest a “lack of sympathy with the requesters” and “a tendency to answer the wrong question or to give a partial answer.” [10.5]
“There seems clear incitement to delete e-mails, although we have seen no evidence of any attempt to delete information in respect of a request already made.” (The former is legal but not the latter.) The email dated 3/12/2008 included “a clear statement that e-mails had been deleted […] It seems likely that many of these ‘deleted’ e-mails subsequently became public following the unauthorized release from the backup server.” [10.5]
The Review found that the IPCM “may have lacked […] the authority to challenge the assertions of senior professors” and “the UEA senior staff need to take more explicit responsibility for these processes”. He told the Review he felt “very much the bull’s eye at the centre of the target”. He explicitly denied that he “became very supportive” as suggested by Jones. The 10/12/2008 email provides “evidence that the IPCM did try to warn Prof. Jones about deliberate deletion of information”. [10.5]
In general, “[t]he Review found an ethos of minimal compliance (and at times non-compliance) by the CRU with both the letter and the spirit of the FoIA and EIR. We believe that this must change”. The Review also made it clear that CRU did not receive enough support from UEA management, and made recommendations to the university on how it should handle future information requests. It also recommended to the ICO that it engage more with universities and clarify how FoI law applies to research.
However, as Steve Easterbrook commented, the Review “never really acknowledges the problems a small research unit (varying between 3.5 to 5 FTE staff over the last decade) would have in finding the resources and funding to be an early adopter in open data and public communication, while somehow managing to do cutting edge research in its area of expertise too.” The Review does point out that in the years since CRU was founded climate science has developed from “a relatively obscure area of science […] into an area of great political and public concern.”
The Review concluded:
[W]e find that a fundamental lack of engagement by the CRU team with their obligations under FoIA/EIR, both prior to 2005 and subsequently, led to an overly defensive approach that set the stage for the subsequent mass of FoIA/EIR requests in July and August 2009. We recognize that there was deep suspicion within CRU, as to the motives of those making detailed requests. Nevertheless, the requirements of the legislation for release of information are clear and early action would likely have prevented much subsequent grief. [10.6]
As Phil Jones has admitted, CRU did the wrong thing with regard to Freedom of Information requests. However, they clearly perceived that the requests were not being made in good faith. The Review apparently made no attempt to investigate the motivations of the requesters.
But all this must be considered in the context of the Review's general findings (summarised here): although the scientists failed to display the proper degree of openness, their rigour and honesty are not in doubt, and their behaviour did not prejudice the advice given to policymakers. Despite being heralded as “the final nail in the coffin of anthropogenic global warming”, Climategate has not even invalidated CRU's results, let alone the conclusions of the climate science community. In any case, the entire work of CRU comprises only a small part of the large body of evidence for anthropogenic global warming. That mountain of evidence cannot be explained away by the behaviour of a few individuals.
Last updated on 4 November 2016 by James Wight. View Archives
Hello. I don't know if this belongs in another thread, but I wanted to follow-up on the "skeptics'" line of attack that "climate researchers hide all their work and just expect us to believe it". The above posts were helpful in explaining why some data should not be released to the public. But none of the comments addressed opening up the code itself to scrutiny? Are there valid reasons why these would be held under wraps (which I perceive, perhaps incorrectly) was done by CRU? Are climate models generally "open" or "held secret"? Thanks in advance for any forthcoming replies.
Actually, pretty much all of the data (raw or otherwise) and model code is openly available.
The raw data:
ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/v2
ftp://ftp.ncdc.noaa.gov/pub/data/ushcn/
ftp://ftp.ncdc.noaa.gov/pub/data/ushcn/v2/monthly/
http://dss.ucar.edu/datasets/ds570.0/
http://www.antarctica.ac.uk/met/READER
http://eca.knmi.nl/
http://www.zamg.ac.at/histalp/content/view/35/1
http://amsu.cira.colostate.edu/
Link to SORCE
http://daac.gsfc.nasa.gov/atdd
http://oceancolor.gsfc.nasa.gov/
http://www.psmsl.org/
http://wgms.ch/
http://www.argo.net/
http://icoads.noaa.gov/
http://aeronet.gsfc.nasa.gov/
http://aoncadis.ucar.edu/home.htm
http://climexp.knmi.nl/start.cgi?someone@somewhere
http://dapper.pmel.noaa.gov/dchart/
http://ingrid.ldgo.columbia.edu/
http://daac.gsfc.nasa.gov/giovanni/
http://www.pacificclimate.org/tools/select
http://gcmd.nasa.gov/
http://www.clivar.org/data/global.php
http://www.ncdc.noaa.gov/oa/ncdc.html
http://www.ipcc-data.org/maps/
http://climatedataguide.ucar.edu/
http://cdiac.ornl.gov/
http://www.cru.uea.ac.uk/cru/data/
http://www.hadobs.org/
Next, the processed data:
http://data.giss.nasa.gov/gistemp
http://clearclimatecode.org/
http://hadobs.metoffice.com/hadcrut4/index.html
http://www.ncdc.noaa.gov/cmb-faq/anomalies.php#anomalies
http://ds.data.jma.go.jp/tcc/tcc/products/gwp/temp/ann_wld.html
http://www.berkeleyearth.org/
http://vortex.nsstc.uah.edu/data/msu/
http://www.ssmi.com/msu/msu_data_description.html
http://www.star.nesdis.noaa.gov/smcd/emb/mscat/mscatmain.htm
ftp://eclipse.ncdc.noaa.gov/pub/OI-daily-v2/
http://www.cpc.noaa.gov/products/stratosphere/temperature/
http://arctic.atmos.uiuc.edu/cryosphere/
http://nsidc.org/data/seaice_index/
http://www.ijis.iarc.uaf.edu/en/home/seaice_extent.htm
https://seaice.uni-bremen.de/sea-ice-concentration/
http://arctic-roos.org/
http://ocean.dmi.dk/arctic/icecover.uk.php
http://www.univie.ac.at/theoret-met/research/raobcore/
http://hadobs.metoffice.com/hadat/
http://weather.uwyo.edu/upperair/sounding.html
http://www.ncdc.noaa.gov/oa/climate/ratpac/
http://www.ccrc.unsw.edu.au/staff/profiles/sherwood/radproj/index.html
http://cdiac.ornl.gov/trends/temp/sterin/sterin.html
http://cdiac.ornl.gov/trends/temp/angell/angell.html
http://isccp.giss.nasa.gov/products/onlineData.html
http://eosweb.larc.nasa.gov/project/ceres/table_ceres.html
http://sealevel.colorado.edu/
http://ibis.grdl.noaa.gov/SAT/SeaLevelRise/index.php
http://dataipsl.ipsl.jussieu.fr/AEROCOM/
http://gacp.giss.nasa.gov/
http://www.esrl.noaa.gov/gmd/aggi/
http://www.esrl.noaa.gov/gmd/ccgg/trends/
http://gaw.kishou.go.jp/wdcgg/
http://airs.jpl.nasa.gov/AIRS_CO2_Data/
http://www.usap-data.org/entry/NSF-ANT04-40414/2009-09-12_11-10-10/
http://climate.rutgers.edu/snowcover/index.php
http://glims.colorado.edu/glacierdata/
http://www.nodc.noaa.gov/OC5/3M_HEAT_CONTENT/
http://oceans.pmel.noaa.gov/
http://cdiac.ornl.gov/oceans/
http://gosic.org/ios/MATRICES/ECV/ecv-matrix.htm
http://www.ncdc.noaa.gov/bams-state-of-the-climate/2009-time-series/
Now, the model code:
http://www.giss.nasa.gov/tools/modelE/
ftp://ftp.giss.nasa.gov/pub/modelE/
http://simplex.giss.nasa.gov/snapshots/
http://www.cesm.ucar.edu/models/
http://www.ccsm.ucar.edu/
http://www.ccsm.ucar.edu/models/ccsm3.0/
http://www.cgd.ucar.edu/cms/ccm3/source.shtml
http://edgcm.columbia.edu/
http://www.mi.uni-hamburg.de/Projekte.209.0.html?&L=3
http://www.mi.uni-hamburg.de/SAM.6074.0.html?&L=3
http://www.mi.uni-hamburg.de/PUMA.215.0.html?&L=3
http://www.mi.uni-hamburg.de/Planet-Simul.216.0.html?&L=3
http://www.nemo-ocean.eu/
http://www.gfdl.noaa.gov/fms
http://mitgcm.org/
https://github.com/E3SM-Project
http://rtweb.aer.com/rrtm_frame.html
http://www.sciencemag.org/cgi/content/full/317/5846/1866d/DC1
http://www.pnas.org/content/suppl/2009/12/07/0907765106.DCSupplemental
http://geoflop.uchicago.edu/forecast/docs/Projects/modtran.html
http://geoflop.uchicago.edu/forecast/docs/models.html
http://www.fnu.zmaw.de/FUND.5679.0.html
http://www.pbl.nl/en/themasites/fair/index.html
http://nordhaus.econ.yale.edu/DICE2007.htm
http://nordhaus.econ.yale.edu/RICEModelDiscussionasofSeptember30.htm
https://github.com/rodrigo-caballero/CliMT
http://climdyn.misu.su.se/climt/
http://starship.python.net/crew/jsaenz/pyclimate/
http://www-pcmdi.llnl.gov/software-portal/cdat
http://www.gps.caltech.edu/~tapio/imputation
http://holocene.meteo.psu.edu/Mann/tools/MTM-SVD/
http://www.atmos.ucla.edu/tcd/ssa/
http://holocene.meteo.psu.edu/Mann/tools/MTM-RED/
http://www.cgd.ucar.edu/cas/wigley/magicc/
Source code for GISTEMP is here:
https://data.giss.nasa.gov/gistemp/sources_v3/
https://data.giss.nasa.gov/gistemp/news/
https://data.giss.nasa.gov/gistemp/faq/
https://data.giss.nasa.gov/gistemp/
https://simplex.giss.nasa.gov/snapshots/
Related links:
https://data.giss.nasa.gov/gistemp/faq/
https://data.giss.nasa.gov/gistemp/faq/#q209
https://podaac.jpl.nasa.gov/
https://daac.gsfc.nasa.gov/
https://earthdata.nasa.gov/about/daacs
http://www.wmo.int/pages/prog/wcp/wcdmp/index_en.php
http://berkeleyearth.org/summary-of-findings/
http://berkeleyearth.org/faq/
https://www.climate.gov/news-features/understanding-climate/climate-change-global-temperature
https://www.climate.gov/maps-data/primer/climate-data-primer
https://www.ncdc.noaa.gov/monitoring-references/faq/anomalies.php
https://www.ncdc.noaa.gov/ghcnm/v3.php?section=quality_assurance
https://www.ncdc.noaa.gov/ghcnm/v3.php?section=homogeneity_adjustment
https://www.ncdc.noaa.gov/crn/
https://www.ncdc.noaa.gov/crn/measurements.html
https://agupubs.onlinelibrary.wiley.com/doi/full/10.1029/2009JD013094
https://agupubs.onlinelibrary.wiley.com/doi/full/10.1029/2011JD016761
https://agupubs.onlinelibrary.wiley.com/doi/full/10.1002/2015GL067640
https://www.clim-past.net/8/89/2012/
https://www.carbonbrief.org/explainer-how-data-adjustments-affect-global-temperature-records
Global surface temperature records use station temperature data for long-term climate studies. For station data to be useful for these studies, it is essential that measurements are consistent in where, how and when they were taken. Jumps unrelated to temperature, introduced by station moves or equipment updates, need to be eliminated. The current procedure also applies an automated system that uses systematic comparisons with neighboring stations to deal with artificial changes, which ensures that the Urban Heat Island effect is not influencing the temperature trends. In the same fashion that a chef turns raw ingredients into a fine meal, scientists turn raw data into a highly accurate and reliable long-term temperature record.
Although adjustments to land temperature data do have larger consequences in certain regions, such as in the United States and Africa, these tend to average out in the global land surface record.
Note that the Muir Russell Commission was able to do a full global reconstruction from the raw data linked to from the above page, without any code, in a mere 2 days (when asked, they replied "any competent researcher" could have done the same).
The Auditors over at McIntyre's Climate Audit have been struggling with their "audit" reconstruction for many years now.
The "any competent researcher" bit comes from here:
The "2 days" bit comes from here:
The report itself is here.
Er, wow. Thanks Daniel, especially for giving such a reply on Christmas Day.
And for contrast, deniers favourite temperature series is UAH. They are not reknowned for providing quick access to their code - does anyone know if code for latest version is available yet?
Do note that scientists in all fields tend to hold tight to raw data till they have had a chance to publish it properly. I think this may have been source of accusations over paleo proxies some time in the distant past.
scaddenp — for the layman, what is "wrong" (or "inferior" or whatever) with the UAH data vs. "better" or "more credible" temperature data?
Thanks in advance.
UAH is preferred by pseudo-skeptics because it has the lowest rate of warming of the available temperature series and is published by long-time climate skeptics, Spencer and Christie. Note that in past, pseudo-skeptics had been very keen on RSS for the same reason.
I dont think anyone is going to really rush in an say UAH is wrong or inferior. Like RSS, it is a record for tropospheric temperature rather than surface temperature derived from satellite MSU readings. It also has a troubled history - see here for details. It would have more credibility if the algorithms used were properly published as RSS does. The latest version is the joker in pack compared to other temperature series and appears to be also diverging from radiosonde readings. (eg see here). Time will tell.
AFT @11 ,
the UAH data have been "wrong" since their beginning. Basically, UAH is measuring the wrong thing.
My understanding is that the satellites were set up with the expectation that their data would be a useful addition to the surface temperature readings (which are limited in coverage ~ though still decidedly adequate, statistically speaking). Also an addition to the (at that time) very limited deeper ocean temperature readings. It ought to have been a good thing (especially for overall global coverage) . . . but turned out not to be so.
As mentioned above, UAH has had a great deal of trouble in getting their act together ~ the delays in recognizing teething problems, the delays in correcting the data, the delays in acknowledging (to politicians) the inadequacies of the UAH data.
Essentially, the UAH term "TLT" (The Lower Troposphere) is no such animal. It is a beautiful name, TLT, but misleading. Not really comparable to surface temperatures ~ the surface where we humans and the general biosphere are living [excepting the abyssal creatures below the oceanic thermocline].
UAH "TLT" has some stratospheric contamination, and contamination from mid and upper troposphere. I have heard the TLT described as being very useful if you are interested in conditions at the level of the uppermost part of Mt Everest. In other words, of little relevance to our biosphere (except for high-flying migrating geese).
But the satellites are already up there, and may as well be made some use of. But it is a pity that the UAH group has chosen to misuse the data.
AFT,
In addition to the issues that Scaddenp mentioned, the UAH temperature record is of the upper atmosphere and not the surface. The TLT (temperature of the lower troposphere) is centered arond 10,000 meters up in the air. Are we interested in the surface temperature, where we live, or the temperature where airplanes fly? The other products from UAH are even higher.
UAH deals poorly with interference from the stratosphere which is well known to be cooling (scientists predicted decades ago that carbon dioxide pollution would cool the stratosphere).
It is much more difficult to measure the temperature of the atmosphere than the temperature of the surface. They use microwave emissions, not thermometers. There is a very complex model to convert the measurements into temperature. The UAH data do not agree with RSS and other evaluations of the same data. Their computer code undergoes major changes regularly (currently version 6). UAH rarely finds errors in their code and has to be corrected by other researchers. UAH does not provide a complete copy of their code to anyone. As Daniel Bailey showed, most everyone else does provide code.
By contrast, surface records are easily evaluated to generate a record (a record at least three times longer than satalite records). There have been no major changes or issues for at least 30 years in the surface record. (UAH made major changes two years ago). BEST, financed by the Koch brothers, did a major reevaluation of the surface data using very different methods of other scientists and found that existing records were correct.
The UAH record conflicts with balloon measurements. Should we believe version 6 of a computer model or actual thermometers? The scientists at RSS (an alternate satalite record) say that the surface record is better than the satalite record. The UAH record differs substantially from the other satalite records.
In my opinion, scientists would discard the UAH record because of its long standing severe problems. It is only kept around because deniers like to cite it. Scientists bend over backwards to avoid being accused of being biased.
I do not understand above comments as I found on UAH website : "The satellite-based instruments measure the temperature of the atmosphere from the surface up to an altitude of about eight kilometers above sea level."
https://www.nsstc.uah.edu/aosc/
1. Satellite sensors measure brightness, not temperatures. Temperatures can be inferred from brightness, but there are numerous "corrections" and "adjustments" to the raw data that must take place prior to these inferred numbers being considered reliable. The corrections to the satellite data vastly outweigh the minor changes to the surface station data during the homogenization process.
2. Data series span multiple generations of orbital platforms. A tremendous amount of "corrections" and "adjustments" to the data are needed for these time series to become long enough to achieve statistical significance.
3. The one data channel that some favor among all the satellite data channels is that of the TLT. This is nominally of the lower troposphere. The TLT channel is a synthetic (derived) product, and not a measured product. Further, it is not a measurement of the surface (where people live), but of the lower troposphere (where airplanes fly). Thus, it CANNOT be used to compare to surface temperatures.
4. The known uncertainties in the satellite trend, as estimated by the record providers, are five times the known uncertainties in the thermometer record trend.
5. Thermometer measurements from ground-based and radiosonde instrument packages are still the gold standard. Note that the radiosonde temperature series goes back to 1958, so it's a longer and more robust series than is the satellite record. It shows continued warming of the lower troposphere.
In summary:
1. Satellites don't measure temperatures, they measure brightness
2. Satellites don't measure the surface temperatures, where people live
3. Satellites measure brightness of the air thousands of feet above the surface, where birds and airplanes fly
4. Satellites convert brightness to temperatures via computer models
5. The known uncertainties in the satellite trend, as estimated by the record providers, are five times the known uncertainties in the thermometer record trend.
http://www.ua.nws.noaa.gov/factsheet.htm
Roque ,
If I understand Daniel's comment correctly :- you should pay little or no attention to the satellite record ~ it measures the air temperature (at best) at high altitude . . . say like the uppermost part of Mount Everest . . . which is of small relevance to the important global surface warming [land and ocean] where humans, plants, animals and fish, are living.
Worse, the satellite record is sometimes quoted with the intention of deceiving the uninformed citizen. If you follow the satellite record, you will notice that it tends to lag the surface temperatures by a number of months ~ so it adds little to the more accurate surface temperature records. Overall, the satellites have been disappointing / borderline useless.
I would consider the satellite termperature record a useful supplement to the surface temperature record, but only as a supplement to it.
Other instrument packages on the satellite platforms are a great deal more useful.
To address the specific quote that Rogue provides in #17:
The quote should be interpreted as an indication that the single "temperature" value provided by the UAH model calculations (based on satellete-measured atmospheric radiation emissions - AKA brightness) are dependent on atmospheric conditions over the layer from the surface to roughly 8km. The value is not equally-weighted for all heights within that range. Spencer's web site shows the weighting for the various model values they produce: