A Place to Stand

Not my idea, I have lifted it bodily from The Air Vent (via the Global Warming Policy Foundation) but it is a brilliant one:

By Sean M O’Brien

Up until now, peer review has been held up as the gold standard in scientific discourse. Recent developments in the climate science arena, such as Climategate, have led many to conclude that peer review is not all that it is cracked up to be. Having said that, peer review may well be perfectly adequate as a scientific standard when the issues in debate are the mating habits of squirrels. However, if the issue in debate is whether or not trillions of dollars should be spent combating global warming, perhaps a new more rigorous standard should be applied.


I propose that henceforth, five levels of scientific rigour be defined. In brief, they are Level zero which is grey literature from advocacy organisations such as the WWF. Level one, which is the current peer review process. Level two, which I will call replicatable, is the current peer review process but with mandatory archiving of data and software code within six months of publication. Level three, which I will call audited, is where an authoritative body of some sort holds a competition on the internet to “find something wrong” with the calculations in the paper with a prize for any independent researcher who can find incorrect calculations. Level four is what I will call Cross Examined and is where the paper in question is deemed so important that, a full scale “internet trial” is conducted. You can think of it as a Scopes Monkey Trial of the researchers and their paper by competent legal personnel advised by scientists. It would mainly consist of oral testimony but with anyone on the internet free to comment and interject in any forum they wish. Naturally these comments can inform the questions put to the researchers.

The main effect of this idea would be to shift the public debate to a new level. Instead of the phrase “the science is peer reviewed” being used to silence sceptics, the phrase “the science is only at level 1” would be used to prompt debate.

What is needed to give this idea teeth is a law. Let us call it the Scientific Integrity Act. Let me focus the discussion from here forward on the United States as most people from around the world will be familiar with how the process of government operates there. The key thing about the Scientific Integrity act is that it should place a limit of the amount of government spending and / or related industry and consumer costs, that can be legislated for based on the level at which the science is at. To take an example. Level 1 science, would not be allowed to justify any government spending or imposed costs. Level two science might perhaps justify $10 million dollars of government spend and $50 million of imposed costs and so on. Needless to say, any proposal with trillion dollar price tags would need level four science to justify it.

Another facet of the law would be that researchers in receipt of a federal grant have to designate their research paper as being at a particular level. In order to encourage realistic assessments of papers by their authors, a penalty system would be included that would operate like this. If a researcher designates their paper as a level three, then should an independent researcher find something wrong with the maths, the funding agency who funded the study would pay a fee to the independent researcher. The fee would be the equivalent of the cost of producing the paper. Naturally a funding agency that got hit as a result of the sloppy mathematics of a researcher would be less likely to fund more bad science by the researcher.

So the big question is of course how would level four scientific papers be selected. I see it working like this. Any government agencies, or legislators, that wish to propose laws based on science, have to nominate ten scientific papers to be assessed at level 4. One would hope they would pick the ten best papers, but just in case, there would be a provision for a minority of legislators in the relevant legislature to substitute five of them. IE if the majority in a 100 seat legislature propose some legislation and nominate 10 papers, then a grouping of 20 legislators would have the right to substitute five of them for other papers.

I would not see anyone sitting in judgment on level four papers. Remember in order to get to Level four, they will already have been audited at Level three so the math will be correct or at least defensible. Rather I would see questioning both orally and via written submissions of researchers as to why they made particular decisions and interpretations. This information would then be available on the web to anyone and the preponderance of scientific opinion and comment would influence legislators to vote for or against the legislation.

Let us consider a hypothetical example. Supposing a researcher decided to use a non-standard mathematical technique that had the effect of producing a particular shape for plotted data even if random number were fed into it. Then, if the researcher tried to defend this notion on the stand, one would expect that the legal teams questioning him would expose this and that the legislators would spot this for what it was. (Then again, one would hope that the paper in question would not have made it past level three in any case.)

Some people, especially AGW believers, may view this proposal as being expensive, both in terms of cost and in that it would delay much needed action. I would maintain the opposite. Currently billions are being spent to persuade an increasingly sceptical public that the world is about to fry and despite the billions, legislative attempts to curb carbon emissions have stalled in most countries across the world. The cheapest and fastest way to get action would be to simply put the top ten research papers that prove the AGW hypothesis through a level four Scopes Monkey type trial.

If the science emerges unscathed from this process, then I for one can see myself persuaded. I suspect that many other sceptics, as opposed to contrarians, would similarly be persuaded. So let us issue the challenge to the AGW community. Name your top ten papers and let’s put them on trial.

With regard to how to progress this idea, I would suggest three things.

Firstly, someone or somebody should do an audit of all the scientific papers referenced in the IPCC reports to see if they are at level one or two. Level two remember is where the data and code is released within six months of the publication of the paper. Perhaps a project similar to Surfacestations.org might be the way to go. Given what I read on the various sceptic climate blogs, I suspect that little enough of the science would make it to level two. As an aside, Donna LaFramboise already did a similar audit looking for grey literature references across one of the IPCC reports and found that 35% of the references were level zero. Imagine the change in discourse if one could confidently say that most of the science in IPCC 4 is level zero or one.

Secondly shift the debate. When discussing the issue with friends or in the media, explain the levels and ask them would they think it reasonable that the science behind a measure that will cost billions should be subject to a certain amount of scrutiny.

And finally, all those Republican controlled state legislatures across America should put forward a Scientific Integrity act in their state. Given the power balance federally there would be no point in introducing the act federally at this time. However, a couple of state legislatures enacting a Scientific Integrity bill would put huge pressure on the AGW community to nominate the top ten papers and put them on trial
===============================
Afterword

I have been unstinting in my enthusiasm for X_Prizes as a means of encouraging technological breakthroughs, being at least 33-100 times more effective than conventional government funding. However I had not been able to see any way of using it on pure research because that does not produce a specific unimpeachable result you can forecast up front, without which prizes have to be awarded subjectively & we are back where we started.

Sean O'Brien has end runned this by making it not the initial research but the checking of it that is the subject of the prize. That is an elegant solution. Because such a prize is open to all there is no government gatekeeper to ensure only the politically correct can check it. Since I would suspect the prize for checking the maths of a theory need almost never exceed £10,000 it would be very inexpensive, by government standards. Indeed financially it could be run privately though this would reduce its credibility.

Stephen McIntyre investigated the Hockey Stick theory, which had passed "peer review". It would only have partly reached level 2 because, unintentionally, Michael Mann had left information on his calculations on a different obscure site & McIntyre found them. Stephen McIntyre's investigation was at level 3 & he proved beyond doubt, that it was fraudulent. This was long after the IPCC had, obviously without any serious checking, announced it as unimpeachable & spread it all over their report. Had there been level 3 prizes in place at the time I think it impossible that none of the thousands of people on the IPCC, whose specific duty it was to satisfy themselves on its accuracy, would have seen if a prize was available & if none of them had somebody else would.

Indeed if Mann had known that even politically correct theories would have been thoroughly & impartially checked I doubt he would have tried it in the first place.

Note Sean's point about it being [possible for individual states to adopt this (as pointed out it is inexpensive). If it were adopted by the UK it would certainly greatly add to the credibility of British science. It is also something the Scottish government alone could adopt with the same effect for Scottish science.

The Catastrophic Warming Hypothesis has passed level 1, "peer review", (to the shame of the chosen reviewers. On many occasions they have failed level 2, making the figures available. The Linear Non-Threshold Hypothesis on radiation has not, as far as I know, even officially passed level 1, it was simply politically adopted. Bernard Cohen's work on the opposing radiation theory, Hormesis, has passed levels 1 & 2 & being a real scientist I assume would be happy to have it tested on level 3.

And if we are going to have some appointed members of the House of Lords we could do much worse (indeed we already do much worse) than giving that as part of major prizes.


UPDATE. The BBC have officially replied here not disputing both lies (indeed making one worse). They have no intention of correcting & claim that censoring scepticism & reporting purely from the alarmist side is showing "due balance" for "minority" scepticism.

Labels: Fixing the economy, global warming, Science/technology, X-Prizes