The IPCC and grey literature

‘Vast majority’, according to Urban Dictionary is, ” possibly the most over-used, tired and tautological phrases ever to have survived in the English language”.

But that is how many references in the IPCC report, an ‘open letter from scientists’ told us, were from ‘peer-reviewed scientific journals’.

The reference list of the AR4 contains about 18,000 citations, the vast majority of which were published in peer-reviewed scientific journals.

This letter was signed by such climate luminaries as Michael Oppenheimer, Kevin Trenberth, Jonathan Overpeck, Gerald Meehl and Stephen Schneider.

The mind-boggling propaganda website reproduced the claim:

The IPCC cites 18,000 references in the AR4; the vast majority of these are peer-reviewed scientific journal papers.

Any non-amnesiac would remember vividly that these are not the only occasions, the club of ‘peer-review’ was brandished at anyone asking questions about the IPCC. You can look here, for example, to refresh your memories.

Even post-Climatagate, the ‘peer-review’ fetish-mongering hadn’t stopped. Strain your eyes and watch the video below, you might see what Mark Steyn thought was surely there:

Nothing to worry about, folks. “We’ll go down the path and see what happens in peer-reviewed studies,” said Ed airily. “Those are the key words here, Stuart. ‘Peer-reviewed studies.'”

Hang on. Could you say that again more slowly so I can write it down? Not to worry. Ed said it every 12 seconds, as if it were the magic charm that could make all the bad publicity go away. He wore an open-necked shirt, and, although I don’t have a 76″ inch HDTV, I wouldn’t have been surprised to find a talismanic peer-reviewed amulet nestling in his chest hair for additional protection.

Such tirades however, were praised by a famous climate scientist.

The problem with ‘peer-review’ and the IPCC

The problem is that we now know — the overbearing drivel about peer-reviewed scientific journals forming the basis for the IPCC —  is simply not true. A basic quality assurance of scientific claims provided by peer review—just did not exist for a significant chunk of the IPCC’s material.

Bibliometric analysis of the third assessment report by Andreas Bjustrom showed the following:

IPCC TAR consists of 14,000 references, of which 62 % are journal references (see Table 1).

Which means 38% of the references are from the non-peer-reviewed ‘grey literature’ category. The situation is the exact opposite of the claims

IPCC Audit

Things are no different with the version released in 2007. A project led by Donna Laframboise, analysing of the IPCC fourth assessement report showed the following:

5,587 references in the IPCC report were not peer-reviewed. Among these documents are press releases, newspaper and magazine articles, discussion papers, MA and PhD theses, working papers, and advocacy literature published by environmental groups.

What was more—in 21 out of 44 chapters in the IPCC report, 40% or more of the references cited were not peer-reviewed.

The picture gets worse the closer one looks. Only 39% of the references cited in the entire chapter on Latin America which contained the notorious Amazongate WWF quote, are from the scientific peer-reviewed literature.

What do we make of this?

The inference is quite clear – the IPCC, working as it were, in areas with sketchy and threadbare primary research, has served a vehicle and a trojan horse for non-substantive, potentially exaggerated claims originating in scientifically dubious publications to queue-jump and attain institutional status. This of course is nothing more than judgement one reaches, applying the same standard the IPCC boasted it had attained, on its own material.

A focused examination reveals a bit more about the specific nature of material employed by the IPCC which contained most of its ‘erroneous’ exaggerated conclusions – Glaciergate, Amazongate, Africagate and so forth.

All these errors arose from material sourced from one specific type of grey literature – environmental pressure group literature.

One can perhaps look the other way as the IPCC employs government report-derived data or quotes newspaper articles to refer to specific events.

But what excuse can the IPCC offer to sourcing scientific material from reports of organizations whose stated missions is to enhance public perception of risk and danger from specific environmental issues?

A bizarre situation

There can be none, but yet there is a great reluctance from the IPCC to give up its addiction to such sources.

For instance, after taking quite the beating over the IPCC sourcing material from World Wildlife Fund (WWF) reports, RK Pachauri said in May,

“The media and several other people have completely misunderstood the need for using non-peer-reviewed literature.  The loose term that is used is ‘grey literature’ as though it was some form of grey muddied water flowing down the drains,”

“But I’d like to highlight what non-peer-reviewed literature constitutes: reports from the reports from the International Energy Agency, the OECD, the UN Food and Agriculture Organization, the World Bank, the Asian Development Bank… and some NGOs – there are some highly prestigious NGOS that are doing detailed academic work, and you cannot ignore this.”

The propagandists wrote in February this year:
Reports by non-governmental organizations like the WWF can be used (as in the Himalaya glacier and Amazon forest cases) but any information from them needs to be carefully checked (this guideline was not followed in the former case).
It is indeed a bizarre situation, but one that goes to the heart of what is wrong. The IPCC’s ties with environmental advocacy cannot be severed easily. On the one hand, you have environmental activitsts protesting with peer-reviewed papers as placards, and on the other hand, you have scientists unwilling to let go of reports from activist organizations.

Protesting with peer review

In a nutshell, the IPCC’s ‘grey literature problem’ can be reduced to its connection with literature from pressure groups and advocacy fronts.
It was into this mess that the Interacademy Council waded into, accepting the charge of offering recommendations for reform. How it performed is the question we will turn to next.


  • Besides the irony of calling me a “famous climate scientist”, let me point out that I brought up Ed Begley’s performance to raise a question more than anything:

    “Now the question is, is this the way we ought to communicate? As filmmaker/activist I guess everything goes, and it may be effective indeed (Begley did get his main point across), but I do have a feeling that it’s not effective or even counterproductive in the long run for a scientist to go at it the way Ed Begley does.”

  • Yes, Bart, I know you talked about Ed Begley and praised him, in order to raise the question “Is is the way we ought to communicate?” However, you go on to give your answer,

    “OTOH, isn’t it time we get out of the defense (yeah, we made a small mistake; we’ll try to do better next time; please don’t hit us) and tell the public how it is? Call a liar a liar?”

    and then

    “I’m of two minds on this one, …”

    So, your contention that you brought up Begley’s ‘performance’ to mainly raise a question is not exactly accurate.

    And while we are on the subject of science communication, what do we make of advocacy through the media by the IPCC, with respect to ??

    It was all done very in a non-aggresive tone, in the print-media, it set up the scene to enable the righteous moral tone climate advocacy and environmental groups adopted. But it had no substance behind it.

  • And just to complete the picture on Pachauri, peer review and ‘grey literature” … while the IPCC has “rules” regarding the use of non-peer-reviewed material [if cited, they will appear in the references, but must be clearly marked as “unpublished”], of the 5,587 non-peer-reviewed references, I was able to find only 6 which were designated as such.

    Until relatively recently Pachauri was wont to brandish peer-review like a crucifix before vampires. Considering the evolution of his own views, I find it somewhat laughable that he’s suggesting that the media have “misunderstood”. See for details! Watch his mouth as his feet march right in!