Observational Bayes > Parametric Bayes > Hypothesis Testing < Looking

This is a completion of the post I started two weeks ago which shows that “predictive” or “observational” Bayes is better than classical, parametric Bayes, which is far superior to frequentist hypothesis testing which may be worse than just looking at your data. Actually, in many circumstances, just looking at your data is all you need.

Here’s the example for the advertising.csv data found on this page.

Twenty weeks of sales data for two marketing Campaigns, A and B. Our interest is in weekly sales. Here’s a boxplot of the data.

It looks like we might be able to use normal distributions to quantify our uncertainty in weekly sales. But we must not say that “Sales are normally distributed.” Nothing in the world is “normally distributed.” Repeat that and make it part of you: nothing in the world is normally distributed. It is only our uncertainty that is given by a normal distribution.

Notice that Campaign B looks more squeezed than A. Like nearly all people that analyze data like this, we’ll ignore this non-ignorable twist—at first, until we get to observational Bayes.

Now let’s run our hypothesis test, here in the form of a linear regression (which is the same as a t-test, and is more easily made general).

Regression is this and nothing more: the modeling of the central parameter for the uncertainty in some observable, where the uncertainty is quantified by a normal distribution. Repeat that: the modeling of the central parameter for the uncertainty in some observable, where the uncertainty is quantified by a normal distribution.

There are two columns. The “(Intercept)” must (see the book for why) represent the central parameter for the normal distribution of weekly sales when in Campaign A. This is all this is, and is exactly what it is. The estimate for this central parameter, in frequentist theory, is 420. That is, given we knew we are in Campaign A, our uncertainty in weekly sales would be modeled by a normal distribution with best-guess central parameter 420 (and some spread parameter which, again like everybody else, we’ll ignore for now).

Nobody believes that the exact, precise value of this central parameter is 420. We could form the frequentist confidence interval in this parameter, which is 401 to 441. But then we remember that the only thing we can say about this interval is that either the true value of the parameter lies in this interval or it does not. We may not say that “There is a 95% chance the real value of the parameter lies in this interval.” The interval is, and is designed to be in frequentist theory, useless on its own. It only becomes meaningful if we can repeat our “experiment” an infinite number of times.

The test statistic we spoke of is here a version of the t-statistic (and here equals 42). The probability that if we were to repeat the experiment an infinite number of times, that in these repetitions we see a larger value of this statistic, given the premise that this central parameter equals 0, and given the data we saw and given our premise of using normal distributions is 2.7 x 10^-33. There is no way to say this simpler. Importantly, we are not allowed to interpret this probability if we do not imagine infinite repetitions.

Now, this p-value is less than the magic number so we, by force of will, say “This central parameter does not equal 0.” On to the next line!

The second line represents the change in the central parameter when switching from Campaign A to Campaign B. The “null” hypothesis here, like in the line above, is that this parameter equals 0 (there is also the implicit premise that the spread parameter of A equals B). The p-value is not publishable (it equals 0.19), so we must say, “I have failed utterly to reject the ‘null’.” Which in plain English says you must accept that this parameter equals 0.

This in effect says that our uncertainty in weekly sales is thus the same for either Campaign A or B. We are not allowed to say (though most would), “There is no difference in A and B.” Because of course there are differences. And that ends the frequentist hypothesis test, with the conclusion “A and B are the same.” Even though the boxplots look like they do.

We can do the classical Bayesian version of the same thing and look at the posterior distributions of the parameters, as in this picture:

The first picture says that the first parameter (the “(Intercept)”) can be any number from -infinity to +infinity, but it is most likely between 390 to 450. That is all this says. The second picture says that the second parameter can take any of an infinite number of values but that it most likely lives between -20 and 60. Indeed, the vertical line helps us quantify the probability this parameter is less than 0 is about 9%. And thus ends the classical or parametric Bayesian analysis.

We already know everything about the data we have, so we need not attach any uncertainty to it. Our real question will be something like “What is the probability that B will be better than A in new data.” We can calculate this easily by “integrating out” the uncertainty in the unobservable parameters; the result is in this picture:

This is it: assuming just normal distributions (still also assuming equal spread parameters for both Campaigns), these are the probability distributions for values of future sales. Campaign B has higher probability of higher sales, and vice versa. The probability that future sales of Campaign B will be larger than Campaign A is (from this figure) 62%. Or we could ask any other question of interest to us about sales. What is the probability that sales will be greater than 500 for A and B? Or that B will be twice as big as A? Or anything. Do not become fixated on this question and this probability.

This is the modern, so-called predictive Bayesian approach.

Of course, the model we have so far assumed stinks because it doesn’t take into account what we observed in the actual data. First thing to change is the equal variances; second is to truncate the data to ensure no sales are less than 0. That (via JAGS; not in the book) gives us this picture:

The open circles and dark diamonds are the means of the actual and predictive data. The horizontal lines shows the range of 80% of the actual data placed at the height where there is 80% of the predictive data below. Ignore these lines if they confused you. The predictive model is close to the real data for Campaign B but not so close for Campaign A, except at the mean. This is probably because our uncertainty in A is not best represented by a normal distribution and would work better with a distribution that isn’t so symmetric.

The probability that new B Sales are larger than new A Sales is 65% (from this figure). The beauty of the observational or predictive approach is that we can ask any question of the observable data we want. Like, what’s the chance new B sales are 1.5 times new A sales? Why that’s 4%. And so on.

In other words, we can ask plain English questions of the data and be answered with simple probabilities. There is no “magic cutoff” probability, either. The 65% may be important to one decision maker and ignorable to another. To stay with A or B depends not just on this probability and this question: you can ask your own question inputting the relevant information to you. For instance, A and B may cost differently, so that you have to be sure that B has 1.5 times as many sales as A. Any question you want can be asked and asked simply.

We’ll try and do some more complex examples soon.