The farce has tipped over into the gulch
About a year back, John Cook and his Skepticalscience team members put out a paper finding a 97% ‘consensus’ in climate literature. An unstoppable media juggernaut trailed its wake. Analysis and discussion followed. Among critical voices, Richard Tol summarized his findings in a live manuscript. But available data was not enough to examine the paper’s fundamentals. The long quest for the full raw data began.
In a series of requests to the authors, the journal and the university, Tol contacted everyone concerned. Cook refused to release the data. Matters escalated to University of Queensland Deputy Vice-Chancellor of Research Max Lu and portions of data were reluctantly released.
Further problems were immediately evident — they were tweeted within the day. Some discrepancy was to be expected but the new material suggested gaps of significantly larger magnitude. It appeared the system used was not reliable:
Abstract and paper ratings agree in 37% of cases. Chance agreement is 32%. Cohen’s Kappa is 8%. h/t @shubclimate
— Richard Tol (@RichardTol) July 8, 2013
Tol continued his efforts and more data was released. However, meta-data required for validation continued to be held back. Tol eventually published his paper without such data.
Closely on its heels, blogger Brandon Shollenberger announced he had additional data. At this juncture, legal threats arrived from the University of Queensland. The university has threatened to sue Shollenberger for breach of copyright, warning him to not release or analyse the data.
Precious secret research data
The university has claimed confidentiality issues and copyright over the data. However, Cook gifted paper authorship to eight volunteers who rated high numbers of papers. Twelve others were named in the acknowledgements for ‘collecting email addresses and rating abstracts’. Their names were public. As Anthony Watts notes, the paper was published under a Creative Commons license.
In his study, Cook emailed authors of papers asking for ratings on a consensus scale. He also used several volunteers to rate climate abstracts. The reliability problems came to light with the first release. The match between scientists ratings and the volunteers was poor. Were similar issues lurking behind the paper’s headline conclusion?
It is meta-data that would answer such questions Shollenberger has now come into possession. This implies serious weaknesses that could come to light with analysis as the reason for the threat of lawsuits.
The oft-repeated claim of 97% consensus appears to be serious trouble.