The Story of The Hot Hand: Powerful Myth or Powerless Critique?

An important and influential stream of research in cognitive psychology has been the study of human reasoning under uncertainty, and especially the study of cognitive illusions of logical and statistical inference. Of particular importance for methodologists are Kahneman and Tversky's investigations of a fallacious belief in the "Law of Small Numbers" (Tversky and Kahneman, 1971), since a substantive implication is that much of the experimental work using orthodox statistical significance testing is of little interest, due to a common lack statistical power. In an interesting thread of this research, Thomas Gilovich and Amos Tversky famously claimed that the belief in the "Hot Hand" in basketball is a cognitive illusion, an instance of the common, but erroneous, belief in the Law of Small Numbers (Gilovich et al, 1985;Tversky and Gilovich, 1989). They supported this claim with a variety of statistical tests failing to show any clear evidence of a Hot Hand phenomenon. This idea that belief in the Hot Hand is illusory has since been very widely promulgated in the cognitive sciences and in popularizations, for example by Stephen Jay Gould in Bully for Brontosaurus (Gould, 1991) and by Gilovich himself in his How We Know What Isn't So (Gilovich, 1991). The curious fact is, however, that these cognitive psychologists themselves employed orthodox statistical significance tests, without performing any power analysis. Here we show that their belief in having demonstrated the illusory status of the Hot Hand is itself an illustration of the Law of Small Numbers, for their statistical tests were of such low power that they could not have been expected to find a Hot Hand even if it were present.