To the Editor:
   Re: "Testing Natasha"

      In Ray Hyman's piece "Testing Natasha" (SI,
   May/June 2005), a new milestone in the debunking of
   paranormal claims appears to have been crossed. Once,
   critics were content to wait for a parapsychologist to
   commit a methodological gaffe and then pounce on the
   experiment's shortcomings to refute the reality of the
   claimed effect. In a sense, the critics were content to act
   like a pack of hungry jackals (or noble lions, depending
   on one's political preference) circling a herd of antelopes
   (the parapsychological research community) waiting for
   a sick calf (methodologically incompetent researcher) to
   separate from the herd.

      Hyman and his coworkers (Richard Wiseman and
   Andrew Skolnick) were apparently not content to wait for
   a sick calf to emerge on its own. They manufactured one
   of their own by designing and conducting a highly flawed
   experiment in which Natasha Demkina (the "Girl with the
   X-ray Eyes") attempted to match seven human subjects to
   their medical diagnoses. Natasha was not screened from
   the subjects, but could observe them at close hand. This
   procedure allows many sensory cues that would enable
   Natasha to match diagnoses to subjects (e.g., the person who
   was missing a large portion of her left lung might for instance
   breathe with somewhat more difficulty than the other
   subjects). Few members of the serious parapsychological
   research community would run a study with such poor
   safeguards against sensory cues (although, alas, some
   would). Hyman then attacks his own study on the basis of
   the methodological flaws just outlined. Thus, with no sick
   antelope in sight, Hyman has taken upon himself both the
   role of sick antelope and devouring lion. As he notes, there
   is really no way the psi hypothesis could be upheld given the
   methodological deficiencies of the experiment that he has
   both designed and attacked. One therefore wonders what
   point there is in running such a study.

      Another point of interest is that Hyman, Weisman
   and Skolnick declared the experiment a failure in any
   event, insofar as Natasha successfully matched "only"
   four of the seven subjects' diagnoses to the appropriate
   subject, whereas Hyman et al. had prespecified five correct
   matches as the criterion for success (by chance, one would
   expect only one correct match). However, assuming
   the experiment had been conducted with appropriate
   safeguards against sensory cues, the probability of
   matching four or more diagnoses to the correct subject
   is 0.01825 under the permutation-matching distribution.
   (The probability of four or more correct matches under
   the binomial distribution is 0.01015; however, due to the
   violation of the assumption of independence of trials in
   Hyman et al.'s experiment, it is not really appropriate to
   use the binomial distribution in this case.) Thus, there is
   less than a 2% chance that Natasha could have done as well
   as she did by luck. Something more is involved (most likely,
   the amateurish design of Hyman, Weisman and Skolnick's
   experiment, which failed to isolate Natasha from sensory
   cues). It is, strictly speaking, true that the results of the
   experiment fell just short of statistical significance at the
   0.01 level (presumably the unstated basis for Hyman's
   et al.'s five-match criterion for a successful experiment).
   However, given such a small number of trials, it would
   surely be more appropriate to use a 2% significance level
   to increase the power of the analysis.

      In concluding, I would ask CSICOP not to manufacture
   any more sick antelopes; the parapsychological
   community already has all the staggering quadrupeds it
   can deal with.