Climate money and adjustments: keeping things in perspective

Kent Clibze has been trying to get hold of documents that record the ‘rationale, methodology and discussions’ relating to temperature adjustments carried out by NOAA.

NOAA in turn has informed the FOI requester it needs money to comply with the request:

money

That’s right – NOAA whose annual budget request exceeded $5.5 billion dollars in 2015, is asking for hundreds of thousands of dollars from a private citizen to provide information. On their colourful website NOAA declares it will accomplish lots of good things with their budget ‘while maintaining strong fiscal discipline’. Perhaps this is how they do it.

The rejection letter proclaims the request amounts to work searching for information going back 30 years, as the organization collected temperature data and slathered layer upon layer of adjustment and quality-control.

Messages can’t be given because, we learn, ‘very few if any letters, phone logs, memos, and other communications on this subject would be available’. ‘Historic archived emails’ cannot be had as they are ‘expensive to access and analyze’. In fact we are told almost anything would be be too much. The schizophrenic NOAA proudly states it has been a steward of temperature data ‘for decades’ it has accumulated so much information it would be impossible to find records pertaining to temperature adjustments among them.

If ‘stewardship’ means collecting data and throwing it randomly in the backroom, sure, decades of such accumulation would be difficult to dig through. In case you had doubts ‘thrown-in-the-backroom’ is not how national agencies archive temperature and climate data this should dissuade you:

14015679426_b76495ea32_z 14015740106_c50c0c7373_z 14035560111_4dd74ce23e_z 14035614612_764173ba23_b

From top to bottom, these are climate data archives at Mozambique, El Salvador, Paraguay and Saudi Arabia respectively. It smacks of hypocrisy for NOAA which is undoubtedly the largest and best-funded climate organization in the world to be asking for money to produce records. In effect, NOAA’s letter claims their records are the electronic equivalent of Saudi Arabia’s paper records.

The problem is worse: a clear trail of why each adjustment was adopted, the supporting evidence and relevant authorities’ signing off on them, has to be on file. This is reproducible data science 101. It is inconceivable an organization like NOAA would have functioned in an ad-hoc manner w.r.t one of their public products – the global average temperature record. Procedures must be in place.

The only conclusion is Clizbe’s request has been unfairly turned down.

Advertisements

15 comments

  1. Warren Pearce (@WarrenPearce)

    Thanks for the post. You say NOAA are asking for money from a private citizen, yet from their letter it appears they have derived the $262,000 figure from a ‘commercial requester’ fee category. That doesn’t seem to make sense?

  2. Shub Niggurath

    Yes, for some reason NOAA has made the determination Clizbe has to be charged commercial data request handling rates.

    The federal register code 15 CFR Part 4.11(c) (i) determines ‘commercial use’ as:

    Commercial use request means a request from or on behalf of a person who seeks information for a use or purpose that furthers his or her commercial, trade, or profit interests, which can include furthering those interests through litigation.

    This implies NOAA considers Mr Clizbe’s request is designed to enrich himself with the data provided to him

    !

  3. omanuel

    Implications for a “Global Ministry of Truth” would be horrific for mankind.

    Evidence for a “Global Ministry of Truth” are overwhelming in false, but world-wide, Standard Models of:

    1. Earth’s Climate
    2. Atomic Nuclei
    3. Ordinary Stars
    4. Big Bang Cosmology

    The UN’s IPCC ses to be the “Global Ministry of Truth” for #1, climate.

    The UN’s IAEA (International Atomic Energy Agency) seems to be the “Global Ministry of Truth” for #2, nuclear and theoretical physics.

    See: Organizational Chart of the IAEA*
    https://www.iaea.org/sites/default/files/orgchart_0.pdf

    *This includes the Abdus Salam International Centre for Theoretical Physics (ICTP), legally referred to as the “International Centre for Theoretical Physics”, is operated as a joint programme by UNESCO and the Agency. Administration is carried out by UNESCO on behalf of both organizations.

  4. manicbeancounter

    Shub.
    I do not think that an audit trail exists. The reason is that there are two types of homogenization adjustments.

    First is the location adjustments that everyone keeps on going on about – for instance for TOBS, urban heat island effect and site relocation. We are finding out in countless places that the adjustments are totally unrelated to this factor. Homogenization here implies identifying a possible bias – such as the urbanization of the area – and making adjustments to trend on the basis of (a) nearby data free of those biases (b) some index of the effects under similar conditions.

    Second are the adjustments necessary to create a global surface temperature anomaly. Temperature readings can appear chaotic at the local level, and the weather stations are not evenly dispersed. For these, and other reasons, adjustments are made to actual data to both cleanse the data of random factors that occur and use the individual stations as grid reference points. This is on the basis on these random factors not appearing in other nearby weather stations.

    For example, in the Southern African data that Euan Mearns and I looked at (on March 5th and 15th) there were two pairs of data sets for the inter-war period that caught my attention.
    Data for Antananarivo (central Madagascar) and Harare (Zimbabwe) was retained in the GISS Homogenised data set, but temperature peaks between 1914 and 1928 were largely adjusted out. The data for Kimberley and Capetown (respectively with warming peaks in 1933 & 1934) was deleted from the GISS Homogenised database, despite having long and complete sets of data.

    Due to the chaotic nature of climate, all these peaks might have similar causes, but with lagged effects. Alternatively they might be random factors causing a coincidence on decadal time scales at regional or sub-regional levels. In either case adjusting out these factors may have the perverse effect of amplifying, or even creating, patterns, rather than separating trend from noise.
    What strikes me is that data deletion is as much a part of the process of creating this homogenised pattern as the adjustments. The implication is that retained data has a pattern that is only tangentially related to the original data. At a data point in a grid, it also has derived data patterns from over a wide area. So the $262,000 may be quite good value in terms the volume of stuff received. But if it is good value, it will be incomprehensible. If it is poor value, the explanations will not stack up to the adjustments made.
    This method might be internally quite coherent, with strong justifications. The problem is with the question of what constitutes noise in data. For example, suppose that in a growing city, the real, but unknown, trend is zero over the last 40 years. The UHI effect is 0.5C, so data shows a level of warming that is consistent with both the global average and some temperature stations in the locality. I would predict it would then be accepted as being correct, and will be then used to “homogenise” unbiased stations that show no warming.

  5. Shub Niggurath

    manic, my point was not about whether NOAA have a scientific and rational justification for the series of adjustments have been carried but about whether they have a clear audit trail of the process.

    NOAA should have a data description document with indexed declarations chronologically listed, with the internal decision-making process along with the final calls. Like this:

    [1] adjustment ‘X’ is deemed required to accomodate artifact ‘Y’ (like TOBS) on date: dd/mm/yyyy, approved by So-and-so. Basis: Genius scientist et al 1991.

    Obviously the code is itself a catalog of the changes, but without the supporting reasoning, information is lost.

    I agree it may be hard to adjudicate what changes are required as part of the calculation of a gridded average and those superimposed afterward. This is easily sorted by declaring code at an arbitrary start point as base code, and then documenting changes from that point on.

    Otherwise, if a flawed data munging decision was taken, say in the mid-80s and the code making the change incorportated without documentation of justification, later users might not question it, and error becomes part of the unquestioned picture.

  6. David Blake

    Hi all,

    NOAA seem to be blocking access to old datasets that were previously available on their website. For example I had this bookmarked (an Archive.org backup of the extreme temp/precipitation records), but now it can’t be accessed either on the NOAA webpage, or on Archive.org.

    On the NOAA site searching for “globalextremes” brings up the rather Orwelian message:

    “www.ncdc.noaa.gov/oa/climate/globalextremes.html
    We would like to show you a description here but the site won’t allow us. ”

    On archive.org, there’s a message:

    “Page cannot be crawled or displayed due to robots.txt.”

    Linking to this file: robots.txt, part of which reads:

    “Disallow: /temp-and-precip/*.php?
    Disallow: /temp-and-precip/alaska/*/
    Disallow: /temp-and-precip/climatological-rankings/?
    Disallow: /temp-and-precip/climatological-rankings/download.xml
    Disallow: /temp-and-precip/drought/nadm/nadm-maps.php/?
    Disallow: /temp-and-precip/global-temps/*/
    Disallow: /temp-and-precip/msu/*/
    Disallow: /temp-and-precip/national-temperature-index/*?
    Disallow: /temp-and-precip/time-series/?
    Disallow: /temp-and-precip/us-weekly/”

    I suspect they don’t want people looking up the old data …. Get paranoid!

  7. Pingback: NOAA Refuse To Publish Methodology For Temperature Adjustments | NOT A LOT OF PEOPLE KNOW THAT
  8. Joe Public

    Methinks the costs accrue from the part requesting ” …documents that record the ….. ‘discussions’.

    If simply requesting ” …documents that record the ‘rationale & methodology’, in theory there should be no issue.

    In fact, that info really ought to be in the public domain anyway.

  9. Tom O

    Shub, you wrote –

    Commercial use request means a request from or on behalf of a person who seeks information for a use or purpose that furthers his or her commercial, trade, or profit interests, which can include furthering those interests through litigation.

    This implies NOAA considers Mr Clizbe’s request is designed to enrich himself with the data provided to him

    I disagree with your analysis. the key portion is the last comma delimited portion of the sentence – furthering interest through litigation. The fear is that the data will be used in a law suit, which should tell you how shaky the rational for the adjustments really were.

    Second, having worked for a government agency in the past – approximately 15 to 18 years ago, for the Bureau of Reclamation to be exact, take your pick of the pictures – with the exception of the Saudi one – and you have how the data from various restoration projects done on dams was stored. That is then scanned into electronic files at some point, but the files are merely jpg type scans of the paper – not OCR or word searchable. Perhaps in the last 15 years data handling has gotten better, but I doubt by much – paper documents scanned into even PDF formats are only recently becoming truly searchable without actually looking at the damn things. Don’t give the government too much credit for maintaining files, regardless of the size of the budget.

  10. Shub Niggurath

    Tom O, the clause qualifies an interest as ‘commercial’ if the item sought (information) would further the profit or trade, via litigation. While NOAA may well be afraid of litigation, like any governmental body or a private corporation, it cannot, directly, cite the fear of litigation itself as a cause for withholding the requested item, in this case information.

    Ironically what this means is NOAA implicitly admits, though inadvertently, that the withheld information, if obtained, can be lucrative to those who get a hold of it! This is what I suggested. This is obviously a ridiculous suggestion from NOAA. Even if Clibze manages to get his requested information, sues NOAA on the basis of the obtained information, and wins, it is US citizens and beyond that would benefit. Why and how would Mr Clibze personally financially benefit?

    NOAA application of this clause is a transparent and gigantic fob-off. It’s them telling ‘we can do whatever we want and make up the rules as we go. Do what you can.’

    As to the temperature archives, what Mr Clibze requested was a listing of the adjustments carried out by NOAA and the rationale behind each of the steps, not raw temperature recordings themselves. I posted the pictures to show the state of affairs in the weather bureaus across some of these countries, compared to whom data is not only digitized and archived from a much longer period of time, but storage conditions and the resources available are undoubtedly vastly superior at NOAA. For them to make a claim that retrieving data on adjustments would be difficult because things go back many years is a bit of a stretch. Sure, data resides across many legacy systems in places like NOAA, but it will not be thrown around irredeemably like in Saudi Arabia’s ‘bonfire method’ of storage and therefore be much easier to pull than what they claim.

  11. Pingback: Weekly Climate and Energy News Roundup #197 | Watts Up With That?