Under the Influence of Confirmation Bias

Quantitative and computer-based methods and models can help - or hinder - the transformation of data into knowledge. Whether they help or hinder depends on how well their designers and users know the strengths and weaknesses not only of computers, but of their own minds. The "Climategate" files, the working documents and correspondence of the East Anglia climate modelers, show what happens when the perils of confirmation bias, and of other defects of intuition, are ignored by those whose job it is to build knowledge from data.

Confirmation bias should not be new to scientists. In the 1970s and 1980s, many reputable physical scientists processed experimental data in ways that suggested, and claimed to demonstrate, the reality of telepathy and telekinesis. When the measurements used to support paranormal claims were examined by human and social scientists familiar with the operation of cognitive bias, those claims turned out to be undemonstrable. The East Anglia climate files suggest that some of the physical scientists involved in climate research are still ignorant of the hazards of cognitive bias, and are mired in methodological errors that replicate the errors of historical parapsychologists.

Confirmation bias is likely to be involved when some data points in a dataset are suspected of mismeasurement and are "corrected." The correction procedure, when selected or written under the influence of confirmation bias, will tend to confirm the model favored by those doing the "correction." The only defense against confirmation bias in such cases is to compare "corrected" data with the original raw data, to check that the corrections are neutral with respect to departures from the favored model. Not only was this not done by the East Anglia climate team, but the original record may have been destroyed.

Confirmation bias may enter when numbers not available from records are extrapolated from other data. Such extrapolation is valid - when the extrapolated relationship is uniform across all contexts in which it has been measured. Many long-term climate studies (not only those of the East Anglia team) rely on extrapolations of temperature from tree rings. Those extrapolations depend on a relation between tree rings and temperatures that held between the mid-1700s to around 1960, but that relation has not been observed since about 1961. No one has given any physical reason for supposing that the relation between tree rings and temperature hundreds or thousands of years ago was more like the relation observed in the nineteenth century, than like that measured between 1961 and today. Experience with the extrapolations of parapsychologists in the 1970s and 1980s suggests that an extrapolation may be chosen because its result fits the model that the extrapolator wishes to confirm.

The final playground of confirmation bias is in the causes considered to explain the data. The spectacular rise in global temperatures near the end of the 1990s corresponded to, among other things, highs of the Atlantic multidecadal oscillation and of the Solar irradiance and Solar flux cycle. Models that test the anthropogenic warming hypothesis only against the null hypothesis are biased in their implicit assumption that nothing else could have contributed to the observed warming. When there may be several causes, the analyses should include a multivariate assessment of their relative contribution to the observed results.

The study of climate change has become as complicated, and as fraught with social implications, as any issue in the social and human sciences. Fortunately, the methods developed in the social and human sciences to produce valid knowledge about complicated issues, even in the presence of inevitable human bias, stand ready to be used in climate science as well. It may well be that the current conclusions of climate scientists will be confirmed when re-evaluated by methods that are equal to the task. We cannot know, unless all their data and all their methods are held open to inquiry.

(This was a draft for some OpEd submissions I sent to popular science media. It was not published - they had their hands full with submissions from scientists much more directly involved with the climate sciences.)