There still exist defenders of p-values. The largest class, the superstitious, are those who remember nothing about p-values except that they must be wee. Let us in haste pass by these amateur magicians.
The hardcore cadre of p-value champions are our concern. These fine folks do not recognize that every use of a p-value except one results in a fallacy. P-values cannot discover cause, nor “links”, nor tell you the probability any hypothesis is true, nor judge the goodness or value of any model. They can do one thing alone: they can tell you the probability, given the model and an ad hoc test statistic and assuming the parameter or parameters of that model are set equal to some value, of seeing a larger value of the statistic also given you were able to repeat the identical “experiment” that gave rise to statistic an infinite number of times.
P-values rely on the ad hoc model choice. They rely on the ad hoc model error choice. They rely on the ad hoc statistic. Change any of these, change the p-value. There is no unique p-value for any set of observations.
P-values do not answer questions people ask. Most ask, “What is the probability of Y given X?” P-values say, “Don’t ask me.” Other query, “What is the probability the parameter lies in this interval?” P-values say, “It is forbidden for me to answer.” Still more want to know, “If I give this patient the new treatment, what is the chance he improves?” P-values say, “Let me be wee!”
Given all this, why are there still p-value champions? Because of their quite realistic fear of out-of-the-box Bayesian procedures.
For one, P-valuers disagree (or most do) that probability can be subjective, as most Bayesians say it is. Probability is not subjective. If your evidence is “There are 99 green states and 1 yellow state in this interocitor, which must take one of these two states”, then it is an unimpeachable statement of subjective probability to say, “Given the evidence, the probability of the yellow state is 82.7578687%”. If probability is subjective, and given there are no such things as interocitors, how can you prove this assessment wrong? Answer: you cannot.
Probability is not subjective, but is instead a deduction, not always quantitative, given evidence.
Some Bayesians, however, are “objective”, and reject subjectivism. P-valuers still dislike these Bayesians. Why? Because of “priors”.
Both P-valuers and Bayesians begin by proposing an ad hoc model, usually parameterized on the continuum, or on a segment of the continuum which is itself continuous. A regression, for instance, supposes one set of observables (the “Xs”) relate in a certain linear mathematical way to a parameter or parameters of the main observable (the “Y”).
The next step, also agreed to by P-values and Bayesians, is to specify the ad hoc model error. The regression supposes the central parameter of the observable can take any value from the limits of negative to positive infinity, and that every value in the continuum is a possibility.
This is convenient mathematical approximation, but it is always an approximation. Nothing infinite actually exists, and nothing can be measured to infinite precision. No process or sequence goes on to infinity, as the math of the continuum insists. (It remains to be seen whether space itself is continuous in this sense.)
These ad hoc models are not strictly needed. Finite, discrete choices aligned with measurement exact models exist, but they are not in wide use; actually, they are mostly unknown.
It is at this point the P-valuers and Bayesians split, the P-valuers to their hypothesis testing fallacies, and the Bayesian to their priors. These are the ad hoc assumptions of the uncertainty of the ad hoc parameters of the ad hoc model.
P-valuers complain that the posterior probabilities of the parameters given the choice of prior depends on that choice. But this is a feature, not a bug. It is a feature because all probability is conditional on the assumptions made. It cannot therefore be a surprise that if you change the assumptions, you change the probability, but P-valuers do express surprise.
They do so because frequentist theory says probability exists in the ontological sense. P-valuers know their models are ad hoc but they also believe that by imagining their data could go on forever, the ad hociness vanishes in some mysterious way. Which is false.
Now almost all Bayesians stop at talking about posteriors of parameters, as if these parameters where of interest. They too have forgotten the questions people ask.
That means, as should not be clear, there is a third choice between frequentist and Bayesian theory. And that is probability: plain, unadorned, matter-of-fact probability, not about parameters, but directly, about observables themselves. This is the so-called predictive approach.
Try, when possible, to use a finite-discrete model, deduced from measurement process. These is the least ad hoc approach of all.
How to do that is detailed in Uncertainty.