Revisiting John Cook's 97% climate consensus study. How bad was it?

My conclusions at the end of this post may surprise you

Jose Duarte is one of the few genuine scientists in social psychology.  I am in a position to know that.  Over a period of 20 years beginning in 1970 I encountered countless pieces of published social psychological research that were so flawed that they proved nothing. And that's not just my opinion.  I put my thoughts into writing and on many occasions, journal editors saw that I had a point and published my critiques.

And it is notable that they did so.  Journal editors HATE publishing critiques.  It exposes their own review processes as inadequate.  So I had to be making a very strong scientific case in what I wrote.  So the fact that more than half of my critiques did get into the journals is actually rather amazing.

What I found was that any thin excuse to come to a Leftist conclusion will get published.  Nothing else much seemed to matter as long as there was a veneer of scientific method to it.  It was actually common for authors to come to conclusions which were contradicted by their own data!  Their urgent priority was to prop up their fallacious Leftist wordview.  That trumped everything else.  More on that here.

So I gave it 20 years of doing social psychological research but in the end concluded that I was pissing into the wind.  Truth and the facts were just not of interest to most of my colleagues and my pointing the facts out was CERTAINLY not of interest.  The Leftist talent at denial -- ignoring uncomfortable facts -- was in full force when it came to my writings.

So over two decades ago I gave up doing survey research and concentrated on family and business interests instead, with results that please me.  I still occasionally read the literature, however and I am pleased that there is a small band of real scientists trying to make social psychology more scientific.  Duarte is one, as are Jonathan Haidt, Lee Jussim and that old hand, Philip Tetlock.

Those four obviously know one-another.  They were co-authors of a paper 18 months ago that pointed out what I have just said about ideological bias in psychology.  The paper is:

Duarte, J. L., Crawford, J. T., Stern, C., Haidt, J., Jussim, L., & Tetlock, P. E. "Political diversity will improve social psychological science". Behavioral and Brain Sciences / Volume 38 / January 2015, e130.

It appeared earlier this year but a preprint was available from around July, 2014.  There is a Readers Digest form of the paper here

Why am I noting this on a blog devoted to tracking environmentalism?  Because the situation in climate science is incomparably worse than in psychology.  The global warming enthusiasts make the psychologists look rigorous. And Jose Duarte has noticed that. Below are some of his astounded comments on the subject.  He is looking particularly at the risible "97% consensus" studies that have come out, particularly the study by John Cook, a psychologist from the University of Queensland in Australia.

If some third world political leader gets 97% of the vote, we know immediately that the vote was rigged and laugh at the naivety of the leader in thinking anyone would believe that figure.  Yet Warmists do exactly the same.  They are too fanatical to see how absurd they are. Their need to believe has overcome their reason and sense of caution.  Their claims are not even good propaganda.

But over to Duarte.  He details just how bad the Warmist studies are:


"Ignore climate consensus studies based on random people rating journal article abstracts. Ignore them completely – that's your safest bet right now. Most of these studies use political activists as the raters, activists who desired a specific outcome for the studies (to report the highest consensus figure possible), and who sometimes collaborated with each other in their rating decisions. All of this makes these studies completely invalid and untrustworthy (and by customary scientific standards, completely unpublishable.) I had no idea this was happening. This is a scam and a crisis. It needs to stop, and those papers need to be retracted immediately, especially Cook, et al (2013), given that we now have evidence of explicit bias and corruption on the part of the raters. (It's crazy that people think the consensus needs to be artificially inflated to absurd heights – do they think 84% or 90% isn't good enough?)

In social science, it's common to use trained human raters to subjectively rate or score some variable — it can be children's behavior on a playground, interviews of all kinds, and often written material, like participants' accounts of a past emotional experience. And we have a number of analytical and statistical tools that go with such rating studies. But we would never use human raters who have an obvious bias with respect to the subject of their ratings, who desire a specific outcome for the study, and who would be able to deliver that outcome via their ratings. That's completely nuts. It's so egregious that I don't think it even occurs to us as something to look out for. It never happens. At least I've never heard of it happening. There would be no point in running such a study, since it would be dismissed out of hand and lead to serious questions about your ethics.

But it's happening in climate science. Sort of. These junk studies are being published in climate science journals, which are probably not well-equipped to evaluate what are ultimately social science studies (in method). And I assume the journals weren't aware that these studies used political activists as raters.

Examples of the unbelievable bias and transparent motives of the raters' in Cook, et al (2013) below. These are excerpts from an online forum where the raters collaborated with each other in their ratings:

"BTW, this was the only time I "cheated" by looking at the whole paper. I was mystified by the ambiguity of the abstract, with the author wanting his skeptical cake and eating it too. I thought, "that smells like Lindzen" and had to peek."

"Man, I think you guys are being way too conservative. Papers that talk about other GHGs causing warming are saying that human GHG emissions cause global warming.  How is that not an implicit endorsement?  If CFC emissions cause warming because they're GHGs, then CO2 emissions cause global warming for the same reason.  That's an implicit endorsement."

Jesus. This is a joke. A sad, ridiculous, confusing joke. And it's exactly what you'd expect from raters who are political activists on the subject they're rating. Who in their right minds would use political climate activists as raters for a serious report on the consensus? This is so nuts that I still have a hard time believing it actually happened, that the famous 97% paper was just a bunch of activists rating abstracts. I've called on the journal – Environmental Research Letters – to retract this paper. I'm deeply, deeply confused how this happened. If this is what we're doing, we should just call it a day and go home – we can't trust journals and science organizations on this topic if they're going to pull stunts like this.

I don't care who you are – even if you're a staunch liberal, deeply concerned about the environment and the effects of future warming, this isn't something you should tolerate. If we're going to have a civilization, if we're going to have science, some things need to be non-political, some basic rules need to apply to everyone. I hope we can all agree that we can't seriously estimate the AGW consensus by having political activists rate climate paper abstracts. It doesn't matter whether the activists come from the Heritage Foundation or the Sierra Club – people with a vested interest in the outcome simply can't be raters.

We don't need random people to interpret climate science for us, to infer the meaning of abstracts, to tell us what scientists think. That's an awful method – extremely vulnerable to bias, noise, incompetence, and poor execution. The abstracts for many papers won't even have the information such studies are looking for, and are simply not written at the level of abstraction of "this study provides support for human-caused warming", or "this study rejects human-caused warming". Most climate science papers are written at a more granular and technical level, are appropriately scientifically modest, and are not meant to be political chess pieces.

There's a much better method for finding out what scientists think — ask them. Direct surveys of scientists is a much more valid method than having ragtag teams of unqualified political activists divine the meanings of thousands of abstracts. I don't mean ask about them their abstracts, as Cook, et al did – that inserts an unnecessary layer and potential selection bias. I mean ask them directly what they think about the principal questions. Interestingly, but not surprisingly, survey studies tend to report smaller consensus figures than the abstract rating studies (I'll have more on that later, see the Bray and von Storch series for now) The consensus will be strong regardless, so it's especially confusing why people feel the need to rig it.

(For subjective ratings of abstracts to be a valid and useful method, it would need to be a carefully selected pool of raters, without ideological agendas, implementing a very specific and innovative method, under strict procedures of independence. I can imagine deep philosophy of science questions that might be anwerable by such methods, things like the usage of certain kinds of words, the way hypotheses are framed and results reported, etc. – but much of that could be done by computers. The studies that have been published are nothing like this, and have no hope of being valid.)

NOTE: The Cook, et al data was leaked or hacked a few months ago – I'm confused by what's going on here. Cook wouldn't release some of his data, and ultimately a bunch of data was hacked or scraped off a server, and it included the raters' online discussion forum. Climate science features far too many stories of people refusing to release their data, and mysteriously hacked data. The person who posted this data, Brandon Shollenberger, is a complete unknown. It's amazing that if it weren't for him, we wouldn't know how rigged the study truly was. There's much more to report – the issues raised by the leaked dataset extend far beyond the quotes above.

The University of Queensland has apparently threatened to sue Schollenberger, on some sort of "intellectual property" grounds. Australia is one of my favorite countries, but we need to stand up for him. To the best of my knowledge, he hasn't done anything wrong – he hasn't posted any sort of sensitive information or anything that would violate our core principles of scientific ethics. The identities of the raters were not confidential to begin with, so there was no new disclosure there. He's exposed the cartoonish bias and corruption of the rating process that underlied this "study", and in so doing, he's served the interests of scientific ethics, not violated them".


Much more from Duarte HERE.  See also here for another searching critique of the Cook et al. paper.  The extraordinary actions of the University of Queensland in defence of Cook are detailed here.

The original Cook paper is here.  Cook is in fact usually misquoted by Warmists.  People allege that Cook showed that 97% OF ALL climate scientists support global warming.  Cook did not say that at all.  He said:  "Among abstracts expressing a position on AGW, 97.1% endorsed the consensus position that humans are causing global warming". And he adds that two thirds of the papers took no position on global warming (which was wise of them). So even in Cook's own words he showed that a MINORITY of papers supported global warming.  The 97% was not of all scientists nor even of climate scientists.  It was 97% of a minority of climate scientists.  Below again is that amusing video of a poor bewildered black man using like a crutch Cook's misquoted findings.  The habitual misquoting just shows how desperate Warmists are.



So although Cook's research methods were rubbish and a disgrace to psychology, I am inclined to think that he is more sinned against than sinning.  I would be quite happy to quote his figures in any debate with a Warmist.  The only difference would be that I would quote what he ACTUALLY said rather than what Warmists say he said!  LOL