Koehler's influence on my prior beliefs

Perhaps the most important paper on parapsychology in recent years did not appear in a parapsychology journal and it did not concern itself with communicating information via telepathy or influencing random number generators. Instead, it investigated the role of prior beliefs when evaluating new (especially contrary) evidence.

The paper confirmed a number of previous investigations into how people considered the result when deciding on the quality of a scientific paper, but it specifically focused part of its investigation on ESP, and the views of parapsychologists and skeptics.

Koehler made a mailing list of scientists from addresses taken from parapyschological and skeptical organisations and "each scientist was asked to evaluate a hypothetical (but representative) ESP study that either agreed or disagreed with his or her prior beliefs." The hypothetical study in question was a ganzfeld experiment.

The questionnaire and one set of study materials were sent to 195 parapsychologists and 131 skeptics, with a similar return-rate for both (38% and 30% respectively). The quality of the report ("good" or "bad") and the results ("positive" or "negative") varied across four types of reports and additionally some parapsychologist were sent two further types of reports: high or low quality, with no results.

Quality varied according to things like method of randomisation, means of separating the sender and the receiver, and use of blind judges. Respondents completed nine questions using a seven-point scale, and then some open questions deigned to encourage written discussion, and lastly a survey of demographics.

The study found that, as predicted, scientists gave favourable ratings to those experiments whose results reflected their beliefs. Skeptics had a greater tendency to do so, to a marginally significant degree (p less than 0.10)

In the open questions the comments were separated and graded as positive/ neutral/ negative so the percentage of each category given to the high or low quality studies could be measured.

It was found that most comments were negative and high quality studies did not receive significantly fewer negative comments than low quality one. Also, skeptics were less critical of the high quality study than the low quality one. Parapsychologists were equally critical of both in their written responses. This finding seems odd, considering how much skeptics' views were influenced by the results but it could be that, once they'd given a low mark on the 1-7 scale, they were not able to justify their grade in detail.

Koehler's work is one of three scientific papers (the other two are Ioannidis (2005) and the Ignobel-winning Kruger, Dunning (2009)) that made me stop and reconsider my own methods of approaching evidence, and also the methods of those sources that I trusted. It was these that convinced me to use sources as close to first-hand as possible, despite the inconvenience, and to reduce my reliance on commentaries written long after the event. Such commentaries can be useful (after all, I write some myself) but as a beginning of research, not an end.

References:
Ioannidis J.P.A., (2005) "Why Most Published Research Findings Are False." PLoS Med 2(8): e124
Koehler, J.J., (1993) "The influence of Prior Beliefs on Scientific Judgments of Evidence Quality" Organizational Behavior & Human Decision Processes, Vol. 56, p. 28
Kruger, Dunning, (2009) "Unskilled and Unaware of It: How Difficulties in Recognizing One's Own Incompetence Lead to Inflated Self-Assessments," Psychology, 2009, 1, 30-46