Why the Cook paper is bunk: Part II

In the Cook group paper, the degree of acceptance of a ‘consensus’ in climate literature is measured. Here, I address a simple question that has hovered around the paper from the time it made its appearance‡, the ‘implicit endorsers’ of anthropogenic warming. (for e.g., see here and here for discussion)

Shown below in the top panel, is total papers Cook et al classified in their project.   An abrupt increase in climate-related papers is seen after 2005. From the bottom panel, it is evident that papers that don’t state a position on anthropogenic global warming make up most of the rise.

compv

Figure 1

Now, Cook and colleagues have spread the message wide that 97% of a ‘large number of scientific abstracts’ support anthropogenic global warming (examples are, herehere, here and here). From the University of Queensland’s press release:

About 97 per cent of 4000 international scientific papers analysed in a University of Queensland-led study were rated as endorsing human-caused global warming.

How this happened is known: a large number of papers not stating a position on AGW were classified as ‘implicitly’ accepting the orthodox climate position.

What is the risk an abstract is classified as an implicit endorser? Of the 7 major categories, 4 are based on explicit statements in abstracts and these items are not susceptible. Category ‘5’ is for abstracts that imply rejection of AGW and is less likely to be mistaken as well. It is the remaining large number of papers with no stated position on anthropogenic warming, that are at risk

From their data, it can be determined that roughly close to a third of at risk abstracts were classified as ‘implicit endorsers’ (median 27%, range: 19-43%).

Now, turn to an another aspect of the study. Every paper got two ratings from two persons and there was an error rate. In estimating how this error acts, it is evident the considerations noted earlier apply again. Abstracts with explicit statements are less likely to be erroneously classified. Papers rejecting the orthodox position are less likely to be interpreted across the divide. The same neutral papers identified above would likely be most affected by error in volunteers’ classification.

With the above two aspects, examine the data shown below (Figure 2):

Rplot04 copy

Figure 2

In Figure 2, the left panel shows the fraction of  papers with no stated position that got classified as ‘implicit endorsers’. On the right is Cook’s error rate (0.33) applied to papers that are most susceptible. Do the two look similar?

Indeed the two quantities track close to another, especially before 2005. Their correlation (Figure 3) is statistically significant (p<<0.05, Spearman). 

corr

Figure 3

Is the similarity between implicit endorsements and the error fraction (Figure 4) a co-incidence? It is possible. But it is pointing to a basic observation: the implicit endorsement category is nothing but the error in the classification exercise.

With the exercise undertaken by Cook et al, if a handful of raters are given a mass of neutral abstracts, about a third would be classified as “implicit endorsers”, regardless.

imperr

Figure 4

This explanation reconciles several observations. It accounts for the fact that total ‘endorsements’ and no position (category ‘4’) abstracts seem inversely related:. The ‘implicits’, which form the bulk of the endorsements, interact with the no position abstracts in reciprocal fashion during classification: these are just categories where one can be mistaken for the other. Cook’s convoluted explanation is  wrong: papers of these putative groups do not interact reciprocally in the real world.

Furthermore, it explains the steady proportions of categories observed (Figure 5). Which is more likely? That thousands of scientists working in hundreds of disparate fields write an ever-increasing number of scientific papers that somehow show a near-constant fraction of papers ‘implicitly endorsing’ an orthodox position? Or, that a handful of volunteers classify papers with a method that affects their results in a roughly uniform manner?

endorse - Copy

Figure 5

The ‘implicit endorse’ category Cook’s group invented, illustrates devilish intricacies that can arise in classification studies. Papers were added to the category merely because a predetermined rating system suggested it to volunteers, who then went looking for it. It serves as a paradigm that illustrates how researchers can imprint methodological and observer biases on material they set out to study.

‡ Glancing at this table  is not essential but very useful
† (i.e., papers shown in blue in bottom panel, Figure 1, minus ‘implict’ rejectors)
Cook’s team has refused to release the error discrepancy data to date.

Advertisements

20 comments

  1. willard (@nevaudit)

    > But it is pointing to a basic observation: the implicit endorsement category is nothing but the error in the classification exercise.

    I’m not sure how this can be characterized as a basic observation. To minimize the editorial content, one might prefer to call it the conjecture.

    One does not simply observe an error.

    ***

    > If a handful of raters are given a mass of neutral abstracts, about a third would be classified as “implicit endorsers”.

    Conversely, if a handful of raters were giver a mass of ABSTRACTS that should be classified as implicit endorsements (say because we asked the authors first), what would be the error rate?

    Counterfactual thinking is always interesting.

  2. Shub Niggurath

    There is nothing called ‘implicit endorsement’. It is a fabrication. Either, all neutrals are endorsements. Or all are just neutrals.

    It is like dividing humanity into ‘good people’ and ‘bad people’.

  3. willard (@nevaudit)

    > It is like dividing humanity into ‘good people’ and ‘bad people’.

    Endorsing is not a moral attribution, so the analogy crashes down as it is raised. This does not prevent it from serving a function: backhanding a self-righteous shot. Not that I’m assuming that something like functions exist in nature. Let us mind our host’s sensibilities.

    ***

    > There is nothing called ‘implicit endorsement’.

    At Wott’s, Richard Tol quoted something from a review by John Cook, and then said:

    Jonathan Koomey wrote that. John Cook wrote a positive review, highlighting the above phrase.

    http://wottsupwiththatblog.wordpress.com/2013/06/10/richard-tols-fourth-draft/comment-page-1/#comment-1216

    Here’s what John Cook said:

    Koomey also suggests a higher-level approach that he’s found particularly effective: […]

    Does that mean John Cook endorsed Koomey’s higher-level approach? It seems that Richard Tol’s point, to make any sense at all, must presume so, since nothing like an explicit endorsement can be read in Cook’s review. Notice the understatements, emphasized in bold.

    Language is a wonderful thing. It is a social art. If something like “language” exists, of course.

  4. tlitb1

    “Does that mean John Cook endorsed Koomey’s higher-level approach?”

    Oh gosh, I see the horns of a dilemma – you can only either undermine RIchard Tol’s whole analogy *or* approve of the whole of Cooks’s tree-hut gangs definition of “implicit endorsement”!

    Yikes! That’s a poser … If you care about Richard Tol’s approach 😉

    In that “review” of Koomey by Cook I think we know Cook does something like endorsing one of Koomey’s quoted paragraphs by cleverly reading between the lines from this rather nauseating introduction:

    “You’ll have to indulge me if I excerpt the section titled “There’s an app for that”:”

    My stomach heaved there – indicating to me that Cook was endorsing and doing more – since all I saw in the passage is that it essentially promotes Cooks app. We “indulge” Cook and say – “yes you endorse that.”

    Cook’s introduction to the second quoted paragraph left my stomach unturned. In fact when I deliberately let myself think of it considering any clinical implication in a “scientific” way, it left my mind unpersuaded by its passiveness:

    “Koomey also suggests a higher-level approach that he’s found particularly effective:”

    Does Cook buy everything Koomey suggests? – I could accept that from the love-in atmosphere of the whole piece, but if we were to pretend it was a real review from a critical mind then we would be excused if we are left scratching our heads and asking “And…?”

    Tol is wrong by his implication. Cook doesn’t endorse Koomey in that passage. He wrote it too lazily. I admit this is coming from a grammatically challenged person but I think I am right.

    Now back to what Koomey said:

    “Unless the speaker is an expert in the field, their opinions should be given no more weight than any other uninformed observer.”

    I say we have yet to see if Cook endorses that sentiment – in fact i think he actively depends on not doing – and that is why we see more climate coat-tailing haematologists making up the “97%” 😉

  5. tlitb1

    For my own satisfaction:

    I want to modify

    “He wrote it too lazily.”

    To:

    “Cook wrote his introduction to Koomey’s second quoted passage too lazily.”

  6. Shub Niggurath

    “Endorsing is not a moral attribution, so the analogy crashes down as it is raised.”

    If I saw a statement made in haste. What you just did is a common attribute in hurried debate where you are rushing to not see the point.

    Classifying people into good and bad is facile because everyone is good, or everyone is bad. The attributes that can differentiate the two exist in all of us. “Implicit endorsement is similar because all the abstracts contain something on global warming or global climate change. It only takes the rater’s imagination to see ‘implicit’ support. With enough of a stretch, you can see it in every item. As tlitb wrote, the raters were confidently generous in not seeing the connection because they were sure there would be enough (thousands) where seeing ‘implicit endorsement’ would be easy.

  7. willard (@nevaudit)

    > It only takes the rater’s imagination to see ‘implicit’ support.

    It takes more than imagination, but there’s something like that going on, as tltib1 just shows. It might be tough to read ABSTRACTS without some kind of background assumptions. To same goes with the practice of science, in fact. That AGW became the background assumption of so much research tells something about its endorsement level.

    The endorsement level seems higher when looking at the self-rating of PAPERS, anyway. It might be tough to argue that raters over interpreted when the results show conservativeness. THAT must have surprised auditors who tried to hack the whole enterprise.

    So far we’ve seen post hoc stats nits, self-righteous politics, and speculative correlations caricaturing language. When should we expect constructive criticisms? You know, something like a specification what would satisfy contrarians, formal tests, approved protocols, and all that jazz.

    One day, perhaps.

  8. Shub Niggurath

    “That AGW became the background assumption of so much research tells something about its endorsement level.”

    No it doesn’t. If that is the case, do you believe the so-called neutral abstracts do not have AGW in the background assumption? Why read and analyse anything at all then? The mere fact that thousands of abstracts came up when you searched for global warming should be enough to reach your conclusion.

    If AGW became the background assumption of say cockroach reproduction, it does not mean much. What matters is the consensus or dissension of those who matter. Those working in detection and attribution, atmospheric sciences, climate modelers etc. Cook’s group had that. They had hammer their result down everyone’s heads, so they went and analysed thousands and thousands of, irrelevant and marginally relevant abstracts.

    The paper studied something other than what you think it did. Its conclusions are intuitively appealing.

    In a better world, Cook would corralled a bunch of abstracts, analysed their intrinsic properties, then designed the survey questions, handed them over to an external group to apply, and collected results. We know what he actually did, and the answer is: none of the above.

  9. Eli Rabett

    That merely kicks the can down the road to look how they picked their “neutral” group. None of the above is still a reasonable survey.

    As MT said consensus as commonly understood is not the process by which science is decided. But consensus is the evidence that the decision has happened. THAT is what Cooke, et al demonstrated yet again. Your family lost Shub.

    FWIW implicit endorsement is spending months or years of your life studying an effect that global warming might have in the future. The publication is a signal of that implicit endorsement.

  10. Shub Niggurath

    Yah, do you imagine I disagree, Rabett?

    All the ‘neutral’ papers should be classified as implicits. With that, close to 99% of the 11944 papers would be ‘endorsing’. That would have been a correct approach. The problem I am talking about then would be more apparent – anything of which 99% leans toward one side, carries no information. The survey would have been correct, but useless.

    Or you take the opposite tack and be rigourous: only abstracts which clearly say ‘AGW is happening blah’, or otherwise, would be admitted into a category, and everything else ‘no position’. That would’ve been a good idea too. But that would have meant Cook et al took Oreskes’ results and worsened it. ‘Cause they found skeptical papers when she didn’t. The survey would have then been correct and useless as well.

    Surveys like Cook’s appear simple. The real problems lie where the rubber hits the road. Ask the volunteer raters who were gifted authorship because they survey thousands of abstracts. The silence from them indicates they know these issues only too well.

  11. willard (@nevaudit)

    > What matters is the consensus or dissension of those who matter.

    This ad hominem misconstrues what to endorse means. Accepting it would shortcut the discussion anyway.

    ***

    Shub & Richard might need to contact the authors that self-rated their PAPERS (with our emphasis):

    We emailed 8547 authors an invitation to rate their own papers and received 1200 responses (a 14% response rate). After excluding papers that were not peer-reviewed, not climate-related or had no abstract, 2142 papers received self-ratings from 1189 authors. The self-rated levels of endorsement are shown in table 4. Among self-rated papers that stated a position on AGW, 97.2% endorsed the consensus. Among self-rated papers not expressing a position on AGW in the abstract, 53.8% were self-rated as endorsing the consensus. Among respondents who authored a paper expressing a view on AGW, 96.4% endorsed the consensus.

    http://iopscience.iop.org/1748-9326/8/2/024024/article

    Yup.

    THAT must have surprised auditors who tried to hack the whole enterprise.

  12. Shub Niggurath

    willard, you are expert in writing cryptic comments which make it appear as though you are making a point, while being completely obscure to the person what the point is.

    Would a survey of 100 cockroach researchers about anthropogenic global warming satisfy you? Do cockroach researchers have the necessary expertise to assess global warming?

    It shouldn’t satisfy you and they don’t have the necessary expertise.

    An ‘implicit endorsement’ is the non-rejection of a popular idea by non-experts.

    At best, it is a shibboleth. Every one pays a bit of homage. At worst, it is a bandwagon effect. Whatever it is, it is not an ‘endorsement’ – I cannot endorse or reject something I don’t know much about.

    When you measure these, you are not measuring the considered opinions of scientists who have given thought to the matter. The scientific literature is no less, or more immune than the lay press in reflecting popular trends and in the usage of buzzwords.

    Do you see any evidence of any care in handling these issues in the paper? The authors are glib.

  13. willard (@nevaudit)

    > I cannot endorse or reject something I don’t know much about.

    Of course you can. You do that every day. Your knowledge of reality is quite limited, and yet we hope you accept that you are a real being, living a real life, among real people, using real objects, entertaining real scientific theories, etc.

    Besides, authors did self-rate their own PAPERS as implicitly endorsing AGW:

    Among self-rated papers not expressing a position on AGW in the abstract, 53.8% were self-rated as endorsing the consensus. Among respondents who authored a paper expressing a view on AGW, 96.4% endorsed the consensus.

    http://iopscience.iop.org/1748-9326/8/2/024024/article

    Unless you claim the authors themselves have absolutely no idea what “to endorse” means, you have no case, dear Shub.

    The best you could claim is that to endorse might not be the best word to express what has been carried out in Cook & al 2013. But even then, that would not mean that the word is improper in that setting anyway. In other words, your statistical analysis won’t protect you from your linguistic prejudices.

    Hope this helps,

    w

  14. Shub Niggurath

    Your obsession with the self-ratings is very fetching, but meaningless. Self-ratings carry their own biases. One cannot be sure what question they are answering. Even these percent figures are butchered by Cook.

    If you are so convinced self-ratings help your case, ask Cook to release the data.

  15. willard (@nevaudit)

    > Your obsession with the self-ratings is very fetching, but meaningless. Self-ratings carry their own biases.

    Only if bias implies meaninglessness in some way, which would reduce Richard’s comment and Shub’s op-ed to absurdity, among other things.

    Comparing the self-ratings with the ratings show that the ratings were conservative. That authors self-rated into 3 shows that to endorse means something, contrary to Shub’s claim to the contrary. This fact refutes Shub’s conjecture.

    Jaque mate.

    Data squirrels won’t save you.