A British judge has thrown a use of Bayes’s rule out of his court. Not only that, his honor (Lordship?) ruled “against using similar statistical analysis in the courts in future.”
A ruling to which this dedicated Bayesian says, “Hear, hear!”
My opinion may be in the minority: the Guardian quotes Professor Norman Fenton, a mathematician at Queen Mary, University of London: “The impact will be quite shattering.”
“We hope the court of appeal will reconsider this ruling,” says Colin Aitken, professor of forensic statistics at the University of Edinburgh, and the chairman of the Royal Statistical Society’s working group on statistics and the law. It’s usual, he explains, for forensic experts to use Bayes’ theorem even when data is limited, by making assumptions and then drawing up reasonable estimates of what the numbers might be. Being unable to do this, he says, could risk miscarriages of justice.
“From being quite precise and being able to quantify your uncertainty, you’ve got to give a completely bland statement as an expert, which says ‘maybe’ or ‘maybe not’. No numbers,” explains Fenton.
Fenton in his objection has hit upon the key reason I support the judge: no numbers! Let me explain.
Bayes’s rule is so simple that it can be proved using the most elementary of arguments. Even frequentist theory admits Bayes’s rule. It is, given the axioms of probability, simply true. How can any judge ban what is both true and trivially so? Here are the details of the particular case:
In the shoeprint murder case, for example, it meant figuring out the chance that the print at the crime scene came from the same pair of Nike trainers as those found at the suspect’s house, given how common those kinds of shoes are, the size of the shoe, how the sole had been worn down and any damage to it. Between 1996 and 2006, for example, Nike distributed 786,000 pairs of trainers. This might suggest a match doesn’t mean very much. But if you take into account that there are 1,200 different sole patterns of Nike trainers and around 42 million pairs of sports shoes sold every year, a matching pair becomes more significant.
All probabilities, including of course those used in Bayes’s rule, are conditional on given evidence. For instance, we can calculate, using the rule, and assuming the suspect is guilty, the probability his shoe prints match that from a pair of “random” tennis shoes.
But to do this requires knowing how many shoes are “out there.” And just what does that mean? The evidence that Nike “distributed 786,000 pairs of trainers” in the years 1996 to 2006 was given. That’s fine, and using that information will give us, after inputing them into the formula, the probability we want. A deliciously precise number, too, to as many decimal points as we like.
But why use 1996 as the starting year? Why not 1995 or 1997? Why not start in June of 1998? Nike might have distributed that exact amount of shoes—and chances are this number is only an approximation—but how many were actually sold? What about other shoes not manufactured by Nike but which are similar? And how many were sold to residents living in just the area in which the suspect lived or murder took place?
And what it is “the area”? Ten blocks? A square mile? How many shoes sold elsewhere were bought on Ebay, say, and shipped to the area? Do all the shoes leave prints at the same rate? Some might have deeper treads and thus are more likely to leave a trail.
It doesn’t matter which assumptions you make. Any set of assumptions will give you, via the formula, a precise answer. That is, an answer which appears precise and which has the imprimatur of science behind it.
But each set of assumptions will give you a different precise answer. Which set of assumptions is just the right one? I have no idea, and neither do the lawyers. But the jury might know.
They form a combined common sense and can better judge what this kind of evidence might mean. The prosecution and defense can bring up the points which they consider salient, and Bayes’s rule can still be explained without the use of explicit formulae—the difference between the probability of guilt given the shoe prints match and the probability of shoe prints matching given guilt can still be highlighted.
But showing the jury some impressive mathematical apparatus which when invoked spits out exact numerical results isn’t, as the judge rightly ruled, fair. The math is not and should not be evidence because some might assume that the complexity of the math is itself proof of the results of using the math. And in this case, the assumptions are so varied and so vague that insisting on precise answers is silly.
Actually, the judge did not ban Bayes’s rule: he banned unwarranted precision. He “decided that Bayes’ theorem shouldn’t again be used unless the underlying statistics are ‘firm’.” To which I again say, Amen.
(Incidentally, I would ban p-values for the same reasons.)
Thanks to readers Andrew Kennett and Mr Anonymous who suggested this post.