Last time, I examined the issue of data availability in climate science in the context of Phil Jones’ paper on the urban heat island in Nature. The case of the Jones paper is simple — data supporting conclusions of this important paper are not available and there are serious doubts whether such data was present at the time the paper was written. As first author, Jones has however categorically stated he does not intend to correct the situation or address it in any fashion.
It is notable that Nature magazine’s editorial policy states:
After publication, readers who encounter refusal by the authors to comply with these policies should contact the chief editor of the journal (or the chief biology/chief physical sciences editors in the case of Nature). In cases where editors are unable to resolve a complaint, the journal may refer the matter to the authors’ funding institution and/or publish a formal statement of correction, attached online to the publication, stating that readers have been unable to obtain necessary materials to replicate the findings.
In the present situation, after many years and attempts by several parties, Nature has reported a direct admission from the paper’s author Phil Jones that making data available is ‘impossible’. In some ways, this is quite unprecedented. A request that Nature publish a formal statement of correction therefore seems logical.
It has to be pointed out, perhaps much to the chagrin of climate science activists, that reproducibility of published studies is an acute concern in science.
Cancer research and the case of Anil Potti and Duke University
The case of cancer researcher Anil Potti at Duke University is illustrative of the consequences of full data availability. In 2006 Anil Potti published results of using gene microarray data to identify ‘signatures’ of chemosensitivities to anti-cancer drugs. His group published their work in the prestigious journal, Nature Medicine. They claimed to have developed a method to identify patients who would respond better to certain drugs. The paper was hailed as a breakthrough in oncology research.
The story takes a familiar turn at this point. Biostatisticians at the MD Anderson Cancer Center Keith Baggerly and Kevin Coombes attempted replication of the findings. Incomplete data and software code availability made their efforts difficult – they had to ‘recreate what was done, rather than just retest the model’. Eventually, Baggerly and Coombes employed forensic bioinformatics, working backwards to reconstruct prediction models Potti used. By November 2007, in their letter to Nature Medicine Coombes, Wang and Baggerly pointed out errors in Potti’s software code and column-data which was made available. They concluded:
We do not believe that any of the errors we found were intentional. We believe that the paper demonstrates a breakdown that results from the complexity of many bioinformatics analyses. This complexity requires extensive double-checking and documentation to ensure both data validity and analysis reproducibility. We believe that this situation may be improved by an approach that allows a complete, auditable trail of data handling and statistical analysis.
This produced no visible effect. What was more – Potti and allied researchers published more papers with the same method. Baggerly found further errors in these papers as well, but the journals refused to publish the findings. Matters at Duke had meanwhile evolved to the next stage – clinical trials on patients, employing Potti’s cancer genomic signature tests got underway. The MD Anderson team now changed their approach; Baggerly had a standalone paper ready, summarizing his findings. A prominent biological journal reportedly rejected their paper because “it was too negative”. In November 2009, they eventually published this paper in a statistics journal, the Annals of Applied Statistics.
As Baggerly and Coombes published their findings, Duke University initiated an internal review of Potti’s methods and data, and stopped the clinical trials. Surprisingly, the university concluded that its review found Potti’s conclusions were “confirmed”, but chose to keep the report confidential. The trials were restarted. The hidden nature of the review caused an uproar.
Duke administrators accomplished something monumental: they triggered a public expression of outrage from biostatisticians. In a first such action in anyone’s memory, 33 top-level biostatisticians wrote a letter [to NCI Director Harold Varmus] urging a public inquiry into the Potti scandal.
Duke remained unmoved. About 4 months later (May 2010), a FOI request submitted to the National Cancer Institute, finally made the Duke report public. Strangely enough, in contrast to the university’s claims the redacted report seemed to state that it could not confirm Potti’s results.
In the end, revelations that Potti had falsely stated or implied that he was a Rhodes scholar in grant applications resulted in a final suspension of the clinical trials. Writing in Oncology Times magazine, Rubiya Tuma said that the oncology community was “shocked” at the falsification regarding a well-known scholarship as the Rhodes. In November last year, Anil Potti resigned from Duke University. His papers in Nature Medicine and the Journal of Clinical Oncology have been retracted.
Some termed the whole episode ‘Pottigate‘.
Keith Baggerly estimated their team spent 15,000 hours trying to recreate Potti’s methods from published descriptions. What is interesting is that Baggerly and colleagues stumbled upon a recurring problem in science – inadequate description of methods and reluctance to share code. In their paper in Annals of Applied Statistics, Baggerly and Coombes noted:
High-throughput biological assays such as microarrays let us ask very detailed questions about how diseases operate, and promise to let us personalize therapy. Data processing, however, is often not described well enough to allow for exact reproduction of the results, …
Unfortunately, poor documentation can shift from an inconvenience to an active danger when it obscures not just methods but errors.
They were more pointed in their conclusions.
In the case of reproducibility in general, journals and funding agencies already require that raw data (e.g., CEL files) be made available. We see it as unavoidable that complete scripts will also eventually be required.
Speaking about descriptions of methods, David F Ransohoff, professor of cancer epidemiology at University of North Carolina said,
“If you look at the really big picture— and this is the key point—the entire purpose of methods sections in science articles is to let someone else reproduce what you did,” “That is really why it is there. So I can see what you’ve done, build on that, or, if I want, see if it is right or wrong. And what has happened as studies have gotten more complex, is that is harder to do. But we, as a scientific field, may have to decide if the solution to that is to say that we are not going to try anymore, or to try to figure out how we can preserve that goal, which is a very important goal in science.”
From cancer to climate science
Involved statistical errors can have real consequences. Joyce Shoffner, a cancer patient who was enrolled in the Potti Duke University trial felt “betrayed”, reported the Raleigh News Observer. Shoffner had received a drug based on tests that suggested it would work very well against her tumor, but this was not borne out in her clinical course. Shoffner volunteered: — “There needs to be some kind of auditor of the data”.
For anyone following the climate debate, the parallels between examination of Potti’s results by biostatisticians Kieth Baggerly et al and examination of Michael Mann’s work by Steve McIntyre and Ross McKitrick are glaring. There are startling parallels between how the University of Albany investigated Doug Keenan’s fraud charge against Jones’ co-author Wei-Chyung Wang and Duke University’s internal investigations of Potti as well. The blog Duke.Fact.Checker recalled the story thus:
For four years, some entrenched people at Duke tried to discredit these challenges in any way they could, including disparaging remarks that biostatisticians were not scientists are all, and that the MD degree yields more expertise in the emerging genome field than a Ph.D. At one point a Dean asked aloud who would believe a bunch of internet fools.
Recounting their conclusions, Baggerly and a joint team of 43 biostatisticians wrote in a joint letter to Nature recently:
The independent reanalysis of these signatures took so long because the information accompanying the associated publications was incomplete. Unfortunately, this is common: for example, a survey of 18 published microarray gene-expression analyses found that the results of only two were exactly reproducible (J. P. Ioannidis et al. Nature Genet. 41, 149–155; 2009). Inadequate information meant that 10 could not be reproduced.
To counter this problem, journals should demand that authors submit sufficient detail for the independent assessment of their paper’s conclusions. We recommend that all primary data are backed up with adequate documentation and sample annotation; all primary data sources, such as database accessions or URL links, are presented; and all scripts and software source codes are supplied, with instructions. Analytical (non-scriptable) protocols should be described step by step, and the research protocol, including any plans for research and analysis, should be provided. Files containing such information could be stored as supplements by the journal.
Context in climate science
These observations are particularly notable in light of Nature’s current editor-in-chief Philip Campbell’s recent comments in an email to consensus climate blogger Eli Rabett. In his comments, Campbell explains Nature magazine’s data access policy. What is interesting to the writer of this blog, is that Eli Rabett asked Campbell the good question of the history of Nature’s ‘evolving’ data availability policy. Hadn’t he already trundled down to the depths of the ‘dead-tree’ library long ago, to emerge with a version of this history himself?
In the face of a detailed response from Campbell however, Eli Rabett only managed to clutch at straws. He seems focused on one small bit forgetting about the rest, which implies to him, that authors need not submit software code to the journal during publication. The contrast between Eli Rabett’s response and Baggerly’s conclusions above is striking. Activist consensus climate bloggers still are far behind and retrogressive, in terms of their views on data access and research reproducibility.
A more open-eyed reading of Philip Campbell’s letter, however, makes for enlightening reading. Campbell appears to have taken his predecessor John Maddox’s beliefs about data availability and formalized them. Quoting:
During my time as Editor-in-Chief we have consistently promoted the maximal sharing of data and materials associated with papers in Nature and all other Nature journals (the journals generally have common policies).
My own first initiative was to invite Floyd Bloom, then Editor-in-Chief in Science, to undertake a common change of policy insisting that all reduced structure data be deposited for immediate access rather than with a 6-month delayed release.
Finally in a recent editorial on data availability, replication of results and fraud, Nature magazine reiterated its approach with telling comments. Noting that paper retractions were “painful’, it wrote:
The need for quality assurance and the difficulties of doing it are exacerbated when new techniques are rapidly taken up within what is often a highly competitive community. And past episodes have shown the risk that collaborating scientists — especially those who are geographically distant — may fail to check data from other labs for which, as co-authors, they are ultimately responsible.
If we at Nature are alerted to possibly false results by somebody who was not an author of the original paper, we will investigate. This is true even if the allegations are anonymous — some important retractions in the literature have arisen from anonymous whistle-blowing. However, we are well aware of the great damage that can be done to co-authors as a result of such allegations, especially when the claims turn out to be false.
Where do we go?
A well-known Achilles heel in medical molecular biology is that the advancing forefront is oftentimes plagued with experiment irreproducibility. Results of flawed experiments may seem to make perfect clinico-biologic sense and appear mechanistically valid. High-throughput experiments have an added layer of intricacy in requiring complex statistical methodologies to infer or detect significant change. Complex and untested statistical methods are not the preserve of high-throughput nucleic acid research alone.
In climate science, the concept of ‘unprecedented’ global change appears intuitive. But as Baggerly points out: “Our intuition about what “makes sense” is very poor in high dimensions”. It becomes difficult to troubleshoot complex but flawed scientific and statistical methods producing results synergistic to prior intuition. Full availability of data, including metadata, and good description of method therefore becomes crucial.
Bishop Hill and Climate Audit have reported preliminary details from the UK Commons Science and Technology Select Committee report on the University of East Anglia (UEA) Climategate emails, out due on the 25th of January. Certain conclusions can be drawn about the Jones 1990 UHI paper at this point. It is not possible to say that Phil Jones committed fraud in the publication of his 1990 Nature urban heat island paper. As Jones states, data acquired from geographically distant institutions formed an important part leading to the conclusions of the paper. However, Douglas Keenan points out that Jones’ continued reliance and citation of this paper even as he co-authored other publications that contradicted it, can only amount to fraud. The data that can verify’s Jones’ position is not available either.
In view of the above, retraction of Jones’ paper could remedy this defect.