The Consensus Project

A lot of hard work went into it, no doubt. The mountain has laboured and brought forth a mouse.

Many venues have already written about the Cook et al Consensus project. The thing is, you can completely accept every single finding in the paper. Yet, it falls flat.

What did the authors find? First, that about 32% of climate papers expressed a position on the cause of global warming. Fine. Second, of the papers that expressed a position, 97.1% ‘endorsed’ human-caused global warming. Accepted again.

Put two and two together. What does it tell us? That about 30% of climate papers ‘endorsed’ human-caused global warming.

This, after counting up every climate paper over the past  twenty-two years – more than eleven thousand of them.

Nice ‘consensus’ you’ve got there guys.

Lewandowsky et al 2013: surveying Peter to report on Paul

In 2012, Stephen Lewandowsky and co-authors submitted a paper to the journal Psychological Science, generating widespread publicity. Here, I address a simple issue/question that has hovered around the paper from the time it made its appearance. The issue is at the heart of Lewandowsky’s first ‘Moon Hoax’ paper and the in-limbo second paper in Frontiers in Psychology.

The ‘Moon Hoax’ paper (a.k.a LOG12, LOG13 etc) draws a number of conclusions about climate skeptics (called ‘deniers’). A major portion of the data and analysis is devoted to ‘rejection of climate science’. The paper’s title advertises its findings about ‘deniers’.

So the question is: how did Lewandowsky and co-authors study climate skeptics?

The paper draft (pdf) stated simply that authors ‘approached’ 5 skeptic blogs to post a survey, but ‘none did’. This led to a hunt to find who exactly these bloggers were (Lewandowsky wouldn’t tell). Lewandowsky spread significant amounts of distraction and smoke on the matter, raising hue and cry that he did email skeptical bloggers:

First out of the gate was the accusation that I might not have contacted the 5 “skeptic” bloggers, none of whom posted links to my survey. Astute readers might wonder why I would mention this in the Method section, if I hadn’t contacted anyone.

What matters however, is not whether or not Lewandowsky contacted skeptics but what came of such contact. The whole point of contacting the bloggers was to get surveys posted on their websites to ensure skeptic participation. This never took place. Through the noise, the question of non-sampling of skeptics remained unresolved‡.

As a way of providing answer, the paper itself appeared in final form about a month back. When examined, the authors appear to have settled on a remarkable method of addressing the defect. In the supplementary information, Lewandowsky et al (LOG13) make a startling claim. They state the blogs that did carry their survey have a broad readership ‘as evidenced by the comment streams’:

All of the blogs that carried the link to the survey broadly endorsed the scientific consensus on climate change. As evidenced by the comment streams, however, their readership was broad and encompassed a wide range of view on climate change.

The authors claim to have analysed reader comments at one venue to determine this. They state:

To illustrate, a content analysis of 1067 comments from unique visitors to http://www.skepticalscience.com, conducted by the proprietor of the blog, revealed that around 20% (N = 222) held clearly “skeptical” views, with the remainder (N = 845) endorsing the scientific consensus.

Extrapolating, the authors infer further that close to eighty-thousand skeptics saw Lewandowsky’s survey on Skepticalscience alone (see below). Owing to such broad readership, enough skeptics are said to have been exposed to the survey.

Readers of climate blogs will at once see several things that are off. However, these are the assertions forming the basis on which Lewandowsky et al 2013 rests.

Analysis

To start, the authors’ premises are accepted. It is deemed that comment streams can be analysed to determine whether a blog has a broad readership, or a more polarized one.

Comments on six blogs where Lewandowsky et al’s survey was posted were analysed. Commenter names and comment counts were obtained from web pages using R scripts. Following the authors’ method, this was carried out for the entire month the survey was posted. For each blog, duplicates were removed.

Commenters were classified as (a) skeptic, (b) ‘warmist’ (c) ‘non-skeptic’ (d) lukewarmer, (e) neutral, or (e) indeterminate. Regulars whose orientations are familiar (e.g., dana1981 – ‘warmist’) were tagged first. Those with insufficient information to classify, and infrequent posters with singleton comments were tagged ‘indeterminate’†.

The results are presented below. A total of 614 commenters contributed 4976 comments to six blogs in the month the survey was posted (range: 2 – 2387 comments/blog). An estimated 111 commenters posted across blogs, with 504 unique commenter aliases from all blogs.

The results show a skewed commenter profile. As a whole, there are 59 skeptical commenters, amounting to about 9.5% of total. Individually, skeptics range from 5-11% of commenters between blogs, with one venue (Hot Topic) showing 19% skeptics. Closer examination shows this to be made up by just 10 commenters. Non-skeptics are close to 80%, i.e., 480 of 614. Neutral posters are 9%, and indeterminate 3%. Of the 59, more than half are from comments posted at one blog (Deltoid).

The same pattern can been seen to repeat by blog:

The marked difference in comment number between the blogs obscures underlying similarities. When commenter proportions are made equal, these become plain:

From the data above it is evident these blogs are not places where readership is “broad” or encompasses a wide range of views on climate. To the contrary, these are highly polarized, partisan blogs serving their cliques. One half of the blogs hosted comments from all of 6 skeptical commenters in total (Scott Mandia, A Few Things Ill Considered, and Bickmore’s Climate Asylum).

The non-surveyed Skepticalscience.com

What about Skepticalscience’s comment stream? Lewandowsky et al state that John Cook at his website analyzed 1067 comments to identify 222 skeptics and use this to buttress claims of broad readership in survey blogs. One wonders how Cook got the fantastic figures! When commenters for Sept 2010 are analysed, there are 36 skeptical voices of a total 286. Cook’s estimates are inflated six times over. In reality skeptics form 12.58% of commenters for that month, and a mere 0.03 fraction of John Cook’s 1067 unique commenters.  These results verify with independent analysis performed by A.Scott.

Furthermore, close to 90% of commenting viewers are not skeptics. Contrary to Lewandowsky et al, Skepticalscience is not a place where readership is “broad and encompasses a wide range of view on climate”. In fact Skepticalscience exactly matches Deltoid, a virulently anti-skeptic website, in commenter profile.

Importantly however, John Cook never posted the survey at Skepticalscience (see here and here). In the face of this false claim, the authors’ post-hoc exercise of computing skeptic exposure becomes counterfeit.

How would the picture have been had Lewandowsky et al actually obtained survey exposure with a skeptical audience? As a comparative exercise, I pulled comment counts from widely read skeptical blogs Wattsupwiththat, Bishop Hill, Joanne Nova and Climate Audit for the same period. Traffic figures provided by Anthony Watts indicate close to 3 million visits in August 2010. The results ought to be eye-opening:

Conclusion:

A number of things can now be confirmed. The authors of Lewandowsky et al 2013 did not survey skeptical blogs. The websites that carried the survey have neither a broad readership, nor represented skeptical readers and commenters. The authors did not survey any readers at the website Skepticalscience, but represent their data and findings as though they did. Lastly, the authors’ calculations in assessing survey exposure, which they base on the same Skepticalscience, are shown to be wrong.

With the above, conclusions drawn about skeptics by Lewandowsky et al by sampling a population of readers and commenters who are not skeptic can be termed invalid. At best the study’s skeptic-related analysis is meaningless, arising from non-representative sampling. At worst the possibility of false conclusions owing to flawed survey exposure arises. The above data combined with Lewandowsky et al 2013 survey results, in fact, show one possible outcome of displaying loaded questions relating to climate skeptics to a non-skeptical audience. Conclusions about non-skeptical ‘pro-science’ commenters and their psychology are probably more appropriate.

Notes:

‡ The list of surveyed blogs (from Lewandowsky et al 2013 SI):

Skepticalscience – http://www.skepticalscience.com
Tamino – Open Mind http://tamino.wordpress.com
Climate Asylum – http://bbickmore.wordpress.com
Climate change task force – http://www.trunity.net/uuuno/blogs/
A few things ill considered – http://scienceblogs.com/illconsidered/
Global Warming: Man or Myth? – http://profmandia.wordpress.com/
Deltoid – http://scienceblogs.com/deltoid/
Hot Topic – http://hot-topic.co.nz/

Note that (a) there is no record of Skepticalscience having posted the survey, and (b) the Climate Change Task Force entry is available on the Waybackmachine (for e.g., here)

† Batch Google searches (e.g., http://google.siliconglobe.co.uk/) and keyword searches on scraped HTML blog posts were used to search for commenter output. Multiple entries were frequently required for each commenter to be satisfactorily classified. Wherever possible (which was so in almost all instances), results during August and Sept 2010 were employed. Comments supportive of consensus, critical of ‘deniers’ and ‘skeptics’ and/or unequivocally appreciative of article (e.g., “great post, now I can use this in my arguments with deniers”) were classified as coming from ‘warmists’. Comments approving of main thrust of a ‘warmist’ blog post, but with no further information available were classified as ‘ns’ – not skeptic. Commenters questioning basic premises of blog post, being addressed to by ‘denier’, ‘denial’ etc, whose stance could be verified by similar mode of behaviour in other threads, were classified as ‘skeptics’. In most instances they were easily recognized. Those, in whom no determination could be made, owing to various factors, were classified as ‘indeterminate’. Commenters explicitly professing acceptance of consensus but posing relatively minor question, etc – classified as lukewamers. Entries required reading at least two different comments for almost every commenter, except in instances commenter orientation was known from prior experience. Certainly there will be errors to a degree, and subjectivity is involved. It is unavoidable that infrequent (and singleton) commenters, and those with non-unique names (‘tom’, ‘john’) are resistant to classification. Validation of method was available when blogger A.Scott arrived at similar results working independently on portions of the data.

This article was published at WUWT.

Lew and Cook: recursion in the climate ghetto

One of the main indicators of the ‘ghetto-ization’ of climate blogging is a complete lack of response to criticism one encounters. John Cook’s Skepticalscience is a prime example in this regard. These people won’t respond to criticism even if their lives depended on it.

But, on occasion, they will, If they think such criticism might reach important ears, or if they feel there might be blowback

Cook is currently in the middle of one such episode. He has had to respond to Bishop Hill revealing (via Barry Woods’ work), that he and his fellow author Lewandowsky identified Richard Betts, Chief of Climate Impacts of the Met Office UK, as a ‘conspiracist’.

Cook claims on his blog that Betts is not a conspiracist. Betts’ comment on the other hand, made it into the main data table in the supplementary information of his peer-reviewed academic paper.

How does he explain this?

The paper’s methods are quite clear on what was done.

1. Authors define ‘recursive hypothesis’ - “…any potentially conspiracist ideation that pertained to the article itself or its author, unsubstantiated and potentially conspiracist allegations pertaining to the article’s methodology, intended purpose, or analysis “
2. Authors use Google searches, Alexa rankings and direct site visits and gather recursive hypotheses
3. Authors excerpt ‘blog posts’ that published recursive theories into a master table with, and this is key, ‘each excerpt representing a mention of the recursive theory’

Examine point#3 again, just in case. The authors claim that “all recorded instances” of recursive theories are in their supplementary data table. Betts’ qualifies under a specific conspiracist idea – ‘didn’t email deniers’.

The thread with Betts’ comment focused entirely on Lewandowsky’s data and has over one hundred comments. The table contains 9 entries for “didn’t email deniers” excerpted from the thread. All excerpted comments meet the authors’ criteria for ‘recursion’, i.e., express some judgement about Lewandowsky’s method, purpose, analysis or motive. This includes Richard Betts’ comment.

Lewandowsky and Cook now claim

1. “we are certainly not claiming that [Betts] is a conspiracy theorist”
2. Betts’ name being in the table “attests to the thoroughness of  daily Google search”
3. the supplementary table just represents “raw data”. 

None of the above can be correct.  It is not possible for the table to just be “raw data”, as their own description of method shows. The comment selection does not reflect on the thoroughness of Google search; rather it does on the faithful identification of comments/posts with recursive conspiracist ideas as defined. As a result, this does imply that Betts is a conspiracy theorist.

If we accept that Betts is not a ‘conspiracist theorist’, then the same would apply to other contributors found by the authors’ searches as well. The Betts comment is qualitatively no different from the others.

It would be interesting to see how Lewandowsky and his co-authors show this not to be true.

Expecting standards of academic probity from Lew psychology

On his blog Skepticalscience, esteemed doctoral fellow John Cook writes of a commenter’s reaction to his colleague Lewandowsky’s as yet unpublished paper:

“LOG12 was fundemenatlly [sic] flawed from the start, and throughout. It offered no valuable insight or understanding as a result. It is clear to any rational outside observer it had one purpose – to be used to promote the authors advocacy of catastrophic anthropogenic global warming – and to demean and denigrate those who do not believe as he does. The fact this paper has never been published, as Lewandowsky’s repeatedly claims, confirms this finding.”

Cook laughs at this commenter as a ‘conspiracist’ for thinking non-publication confirmed how bad the paper really was.

It will be interesting to see whether this commenter resists the “Something Must Be Wrong” urge when LOG12 is published or continue to assert that the research is “a fraud”.

No, Cook.  That’s not the ‘something must be wrong’ urge, that’s the ‘any serious academic would see right through this’ type of wishful thinking.

Thinking your colleague’s paper didn’t get published, because of how bad it is, is placing faith in the academic peer-review process, yet. Where one hopes reviewers and editors would see questions and criticism raised about the paper. Your commenters and critics come from a place where higher standards reside.

Why is John Cook not responding?

This relates to Skepticalscience‘s John Cook who has co-authored a academic peer-reviewed paper in the Frontiers in Psychology journal with Stephen Lewandowsky.

There are two simple, yet serious questions about his paper. Question number one: where is the ethics approval section of the paper?

Now, I might be mistaken. The section could be in the paper. After all, the paper is 57 pages long and ethics review section could be hidden someplace. On top of it, I am a ‘denier’. So I might be not seeing what’s there.

On to question number two. Why are there what appear to be fabrications and falsifications in the paper?

Again, this has to be clearly understood. People make all sorts of mistakes in research. The kind of errors that are considered serious enough to constitute scientific misconduct are hard to pin down. As a shortcut, the US NSF for instance makes the determination that any act that constitutes fabrication, falsification or plagiarism, would qualify.

The kind of usage of comment material Stephen Lewandowsky, Cook and others appear to have employed in their paper seems to fall squarely in the falsification and fabrication territory. Brandon Shollenberger’s post is published at a prominent outlet WUWT.

Shollenberger’s evidence doesn’t rely on interpretive grounds to support this conclusion: the excerpted quotes and the full quotes with their context are provided in the open.

Cook usually does not answer to criticism. But this is about a scientific publication in the public domain. Of the questions above, the second one, is serious. It requires a response.

Quote fabrications from Australian academic climate communicators Lewandowsky and Cook

Stephan Lewandowsky and John Cook are two extremely well-respected academics. The former operates out of University of Western Australia and the latter, from University of Queensland in addition.

This post by Brandon Shollenberger shows Lewandowsky and Cook to have fabricated quotes from material left by commenters on blogs. The quotes were then used to imply entirely different meanings from what the original comments intended, in their peer-reviewed climate communication paper in the prestigious Frontiers in Psychology stable.

How is this possible? The peer-review process usually has a strong track-record of trapping errors such as the above.

Lewandowsky and Cook’s paper, a draft of which is available, also lacks description of an institutional review process or approval for their study. Usually reviewers are prompt in asking for such things and rarely, if ever can anyone get past them. Even the lowliest of studies involving human subjects entails an ethics board or an institutional review board examination.

How did this happen? Maybe, there is a simple explanation. Perhaps Lewandowsky and Cook obtained institutional review via proper channels but failed to mention them in their paper.

Skepticalscience and John Cook’s vaccine misinformation

John Cook is doing a fellowship at the University of Queensland. He has recently released a presentation that spreads vaccine misinformation on the propagandist website Skepticalscience

The Cook co-authored presentation includes a slide about a Centers for Disease Control (CDC) flyer on flu vaccines. It claims that attempting to correct a ‘myth’ can familiarize readers to the myth.

To illustrate, Cook and co-authors show the flyer which lists ‘myths’ about influenza vaccines and provides correct information beneath each item. Item #3 is “The side effects are worse than the flu”, which is countered by “The worst side effect you’re likely to get with injectable vaccine is a sore arm”

From the CDC flyer on influenza vaccines

Cook and authors believe this approach is wrong. Instead, they insist that only correct information (‘facts’) be provided. The authors advocate saying: “–The vaccine is safe! The worst side effect would be a sore arm.”

Presentation slide by Cook and co-authors on message about flu vaccine

This is outright false information. There are several adverse effects associated with an injectable flu vaccine, the least harmful of which is a sore arm. Far worse adverse events are possible, ranging from fever to severe allergic reactions. Fortunately, these are rare. What vaccinees need is to be informed in an appropriate manner about the probabilities of these events.

Cook gets rid of key phrases, adds new words and an exclamation point to the CDC’s careful language. An optimistic statement about probabilities has been mutated into a false categorical declaration about absolute risk from a vaccine.  What one hears now sounds like a burger sales pitch by a marketing team. It is the Susan Joy Hassol brand of ‘science communication’.

The live-attenuated vaccine administered as a nasal spray can result in a mild flu-like illness. In a given season, it is quite possible and likely that individuals note their mild symptoms following the vaccine and contrast them with unvaccinated members who show no visible illness. The origin of the misconception likely lies in this specific context. The CDC text addresses these points.

Cook’s approach, instead, is to label the whole thing as just a ‘myth’.

Yeah. That should clear up all confusion.

Postscript: Cook and Lewandowsky’s conclusion that repeating a so-called myth will reinforce it, comes from their colleague Nobert Schwarz’s presumptuous “Skurnik, I., Yoon, C., & Schwarz, N. (2007). Education about flu can reduce intentions to get a vaccination”. We learn, from a Schwarz review article in Journal of Experimental Social Psychology, that Skurnik et al showed subjects two leaflets one being the CDC flyer and the other a simple ‘facts’ sheet. Those who saw the CDC flyer misremembered more myths as facts on a true/false questionnaire, after a time delay. How such a study design can reflect either the intentions or the true state of knowledge of potential vaccine recipients than rather the test-taking strategies of experimental subjects is beyond me.

Incidentally, ‘Skurnik, Yoon and Schwarz (2007)’ is…unpublished.

Consider what Cook does on his website. He takes a purported ‘myth’ which is usually a caricatural simplification of an original question and start off confidently pretending that there is a clear-cut refutation. The refutation is constituted by an  answer that is often over-simplified to the point of falseness. When all messy questions that arise from reality are ‘myths’, all answers are simple.

Cook should keep his shenanigans to climate and stop spreading false information on vaccines and matters of public health and safety.

Skepticalscience.com quote surgery on Pat Michaels

I read with amusement Skepticalscience’s latest in their lineup of posts on climate scientists whose views diverge from the consensus. First it was the ‘Michaels Mischiefs’ series. Now he’s been turned into a ‘serial deleter’. Michaels’ probably getting off easy – John Christy is stuck with ‘Christy’s Crocks’. I guess if you run a website, you can call people whatever names you want.

Skepticalscience – Rewriting History

“…resist the temptation to reply to [trolls].
Instead, do what the troll hates most — simply remove the comment.”
— John Cook

The recent censorship episode at the skepticalscience.com brings an often overlooked aspect to the forefront. The target of deletion Prof Roger Pielke Sr, runs a blog. The actions of Skepticalscience were revealed because he posted them there.

What if a scientist or a lay person, interacted with websites like Skepticalscience and did not have a blog?