The problem with studying probability is that you end up devoting most of your life to finding new ways to say "I don't think we really know all that much." This is a particularly big problem if you'd like to study probability and get paid for it outside of a university. Business is about making decisions, not waxing poetic about the cloud of unknowing. For all of the things discussed in this blog, in practice everything boils down to the question "how sure are you?"

In every day language we often utter phrases such as "I'm 90% sure that she's the person I was talking to", "I'm 99% certain that we're out of eggs", "No way you can walk that far, 99.9% positive about that." Here 90%, 99% and 99.9% all mean "really sure", but how close are these values? When talking about certainty, they are as different as having $10, $100 and $1,000 in your wallet.

Measuring the Distance between Percents.

One of the most common, everyday errors I've witnessed in the analysis of percentages is to simply subtract the difference between two percents and claim that that is their difference. For example, many people will claim that 90% and 99% are only different by 9 percentage points, and they will say this is a "9% difference". Now 9% and 18%, 1% and 10% are all also only different by 9 percentage points but clearly there is a huge difference here! If your new customer acquisition rate goes from 9% to 18% you have clearly doubled your customers coming in. If you move from a state with 1% income tax to 10% income tax you the amount you pay for tax has increased 10 times (meaning you're paying what you were + an additional 9 times that amount)! Clearly 9 percentage points is not measuring the same thing in all these cases.

When we're talking about a change in percent we need to reflect that as:$$\frac{P_{\text{new}}-P_{\text{old}}}{P_{\text{old}}}$$

And with this new formula we get much better results:$$\frac{18-9}{9} = 1 \text
{ (i.e. 100%)},$$ $$\frac{10-1}{1} = 9 \text{ (i.e. 900%)},$$ $$ \frac{99-90}{90} = 0.1 \text{ (i.e. 10%).}$$ As percents these new distances are 100%, 900%, and 10%, instead of just 0.09. This is a much better answer. Part of our problem was thinking of 'percent' as a unit rather than simply 'part of a whole'. Most people experienced in probability or mathematics in general will not make this mistake, but it's worth realizing this error since it is extremely common in the general public.

Let's move on to seeing how we can apply this knowledge to our earlier question of certainty. Suppose we want to compare being 90%, 95% and 99% certain of something. This comes up frequently in A/B testing, and even in everyday conversation. We might be tempted to say that that the difference between 90% and 95% is $\frac{95-90}{90} \approx 5.6\% $, 95% and 99% would be $\frac{95-99}{95} \approx 4.2\% $, and 90% and 99% would be $\frac{99-90}{90} = 10\% $.

Clearly if you have a jar filled 90% of the way with water, and then you pour in enough water so that the jar is now filled 99% of the way you have 10% more in the jar. 10% isn't a whole lot, so if I'm running an A/B test and I am 90% sure that variant B is superior, what's the point of running the test much longer just so I can be 99% certain? Likewise running a test that is 95% certain to be correct for a meager ~4% increase seems not worth it.

Here we run into an equally dangerous error as the one we started with. This error arises because being certain is not at all similar to filling a jug with water.

Measuring Certainty

One of my absolute favorite images in the philosophy of the mind is Ned Block's Homunculi-headed Robot. It's importance to the philosophy of the mind has no relevance to the subject at hand, but we can use this image to help us understand certainty. For those unfamiliar, a homunculus is a tiny humanoid creature. The Homunculi-headed robot is a robot controlled by head filled with tiny people pushing buttons.

In our case we'll use these homunculi to represent the beliefs in our head. For example: If we believe there is a 90% chance of a variant in an A/B test being superior, we could imagine this as being 10 humonculi, where 1 of them thinks going with the variant is a bad idea, and 9 think the variant is a winner (i.e. 90% of our homunculi believe the hypothesis).

Being 95% certain can be represented by 20 homunculi with only a single homunculus standing in opposition to the idea that the variant is superior, and 19 (i.e. 95%) believing the variant is better.

Finally being 99% certain is the case of there being 100 homunculi, and only 1 of them believes that going with the variant is a bad idea.

In all cases we have only a single dissenter, but the total number of homunculi from 90% to 99% increases by an order of magnitude! Clearly we cannot measure distance in certainty the same way we do for most other percentages!

A more down to earth example would be money. Being 90% certain is like betting $1 when you have $10 in your wallet, and being 99% certain is still betting only a $1 but having $100 in your wallet!

Conclusion

The key thing about certainty is that it doesn't work like normal percents. The difference between 99.9% certain and 99.9999% may seem trivially small, but in reality the difference between the strength in these beliefs is the same as difference between having a rainy day fund and being a high net worth individual. The next time you run a test and think "why not just call this test at 95%, getting to 99% will just take too long", think about how much more you would prefer to have $100 in your wallet instead of just $20.

If you enjoyed this post please subscribe to keep up to date and follow @willkurt!