Statistical inference issues

Statistics is an infamously difficult field of mathematics (or, as some pure mathematician will say, of "pretend mathematics") in that many of its conclusions seem very counter-intuitive at first. Human brains naturally operate on deduction: we hold a model of the world in our brain and apply this model to estimate something, to choose between two actions, et cetera. Statistics, on the other hand, is inductive and inferential: we look at the data and try to understand how it came about. Unfortunately, induction is much harder than deduction: deduction always leads to a definitive conclusion, while induction, in principle, may have us consider infinity of possibilities before we land on something that works.

Here is an example illustrating the difference.

Deduction: "What is 71 * 83?" The answer is 5893, and a strong middle school student should be able to do the math by hand and obtain the answer.

Induction: "What are the two prime factors that, when multiplied, give us 5893?" The answer is 71 and 83, and even professional number theorists will be stumped by this question.

In both cases we are talking about the same equation: 71 * 83 = 5893. However, in the first case we calculate the product of two known numbers, and in the second we look for two numbers the product of which equals the known number. In the first case we look for the answer; in the second case we look for the question giving the answer.

As a result, statistics is frequently misused by people leading them to obtain bogus conclusions. I highly recommend Darrell Huff's "How to Lie with Statistics" book illustrating just how common this is. Virtually all statistical claims you run into in newspapers or popular science articles are, at best, misleading, and at worst, flat-out wrong.

---

Here I would like to talk about the particular error people make: ignoring prior probabilities when choosing between multiple hypotheses. According to the Bayes Theorem (which is a hard mathematical result that can be rigorously derived), probabilities of different models being true have to be adjusted by the probabilities of their conditions being true. People often forget about this adjustment and, therefore, pick the wrong model.

Here is an illustration.

Suppose you have a bag of 1000 coins, and 1 of those coins is two-headed while 999 other coins are fair. You pull out 1 coin from the bag, flip it 6 times and get 6 heads. What is more likely, that you pulled out a two-headed coin, or a fair coin?

Wrong reasoning: "If this was a two-headed coin, the probability of getting 6 heads would be 100%. Otherwise, it would be less than 2%. I should bet on it being a two-headed coin."

Correct reasoning: "There are two possibilities: I pulled out a two-headed coin, then got 6 heads - and I pulled out a one-headed coin, then got 6 heads. The probability of the first outcome is 0.1%, and the probability of the second outcome is a little less than 2%. I should bet on the coin being fair."

What happened here? In the wrong reasoning one forgot to take into account the fact that the vast majority of the coins are fair, and that fact changes the calculation entirely. In the correct reasoning, one realized that, even though with a fair coin the probability of getting 6 heads in a row is low, the odds of pulling out a double-headed coin in the first place are far lower than that still.

---

This error is not just made in mathematical calculations, but it comes up all over the place: when choosing between different medical treatments, when deciding between different business decisions... and when choosing between multiple historical hypotheses.

Let us talk about "Jesus rising from the grave". According to our beloved Doctor William Lane Craig (from his first appearance on Alex O'Connor's podcast), between two explanations - naturalistic and religious - one should pick the one that best explains the evidence. He argues that the naturalistic explanation struggles with explaining the allegedly found Jesus' empty grave and a bunch of eye witness accounts records of which are spread across centuries - while religious explanation does not have this issue. Hence we should go with the religious one.

Believe it or not, but he makes the same error as in the coin example. He simply asks, "Given A is true, what is the probability of the observed outcome? And what is it given B is true? The latter is bigger than the former, so we should choose B over A". He does not take into account the fact that A and B might have the same probabilities of being true to begin with.

But what are those probabilities? What does it even mean to talk about prior probability that naturalism is true? Well, in Bayesian statistics we look at the evidence of similar things happening and ask ourselves, "How often historically did A happen and how often did B happen?" Now, even the most prominent Christian advocates such as Dr Craig admit that, aside from Jesus', there are no known cases of human resurrection. Furthermore, the claim that Jesus was resurrected is exactly the hypothesis we test, so we cannot use it as evidence of itself... Therefore, here our prior probability has to be incredibly low: since there are no other known cases of human resurrection despite tens of billions of humans having died throughout the recorded history, there are few reasons to believe that it is even possible.

On the other hand, history is littered with examples of erroneous claims, false witness testimonies, misunderstood writings, straight out fantasies and so on. The prior probability of this being just one more such story is not 100%, of course, but it is very high - since it has happened to be the case in every single instance ever recorded.

So now we have two hypotheses: naturalism, and religionism. The naturalistic hypothesis struggles a little with explaining a couple of historical facts, but given other observations, there is a very high chance that it still holds and the facts can be explained eventually. On the other hand, religious hypothesis explains everything perfectly well if it holds - but we have absolutely zero data supporting it even being possible in principle, let alone actually occurring here. In the first case we have an explanation that has always worked well and just struggles a little here; in the second case we have an explanation that has never worked at all, but if it worked, it would explain everything. Which one is better?

---

Instead of answering the question, I will suggest yet another analogy, a food for thought.

For the past 100 days my friend has been meditating every morning, until yesterday. Yesterday she skipped her meditation, and in the evening got a raise at her job. Consider two competing hypotheses:

1) Meditating exactly for 100 days and then stopping always causes one to get a raise.

2) It is a coincidence.

Which is a better explanation of what happened?