A strictly incorrect way of writing Bayes’s theorem.

Still with me? Hope so, because we’re only on the second page of Senn’s article (but don’t fret; we’ll be skipping most of it).

Review: in logical-probability Bayes (as in all Aristotelian logic), we begin with a list of premises (data, observations, evidence, or other synonymous term) and a proposition which is hoped to be related to the premises; from the premises we deduce the probability the proposition is true. Not all premises are sufficient to guarantee a numerical value, nor any value, nor any precise number: the probability could be stated merely in words, nonexistent, an interval, or a precise number.

Senn writes “we let stand for the so-called marginal probability of an ‘event’, ‘statement’ or ‘hypothesis’ A and we let stand for the conditional probability…of B given A.” (Note: I have edited the notation ever so slightly so that it will render well on a web page; I have not changed any meaning.)

As before, I already disagree. There just is no such thing as unconditional probability, or probability without respect to any evidence, thus it never makes sense to write “.” We can write (say) which is the probability of A (a proposition) given the evidence E (also a proposition, albeit possibly a complex one which includes data observations).

Example: A = “A ’6′ shows” and E = “We have a Martian breen, etc.” (see the previous part for an explanation). But it makes no sense to ask, “What is the probability a ’6′ appears” without reference to something—whether it be a breen, a die, or something else.

Failure to recognize this creates another stumbling block in understanding probability. Probability is usually introduced as unconditional, and the complexity of conditioning follows some time later: but this is a mistake. There just is no such thing as unconditional probability. Just as there is no unconditional truth (of a proposition, we always at least refer to our intuition or faith if not a list of premises.)

Of course, inside a given problem, once we have E in hand, and it’s agreed to by all and always understood to be there, and for the simple ease of notation, there is no harm in writing . But this should not be done until one is well used to logical probability, else it seems like probability is a thing and not a measure of knowledge.

Don’t think this is a big deal? Oh, boy, is it ever, as we’ll see.

Senn introduces for a hypothesis, i.e. some proposition, indexed by i, as there may be more than one proposition which is true in some situation. He adds a superscript T to indicate we believe is true, but I’ll skip this complication. Whenever we see shorthand like it means “The probability is true given E.”

Senn then writes (with my change in notation) “We suppose that we have some evidence E . If we are Bayesians we can assign a probability to any hypothesis and, indeed, to the conjunction of this truth, , with evidence E.”

Subjective Bayesians would agree: logical probability Bayesians do not. Subjective Bayesians are as free with numerical probabilities as politicians are with other people’s money. Subjectivists “feel” probability is a matter of emotion because they fail to write down the conditioning premises. They may arrive at a number (and bet using it: subjectivists are inveterate gamblers; at least in theory), but this does not mean the numbers they produce have any bearing on the probabilities unless it can be demonstrated their (unwritten) premises imply these (and no other) numerical values. There is much more to the errors subjectivists make, but given that they usually only make them in theory and not in many problems, where they usually agree with LPBs, we’ll let these go until another day.

LPBs are more Socratic and admit their ignorance. The original series proves the LBPs are right: we can not always discover a probability, and so we can not always discover a numerical value for a probability. We can still manipulate the symbols until we get the form of an answer, but that doesn’t make the answer right. I believe Senn would agree with that.

Thus we can always write Bayes’s theorem, which Senn does:

.

Spot the trouble? What could it possibly mean to say ? or even ? If we knew then we would know and thus we wouldn’t have to bother with any kind of experiment or other evidence or indeed anything. We’d just write the answer down!

We’re in just as much trouble with . How can we ask about the probability of evidence we just witnessed? It has to be a probability of 1, right?, or it wouldn’t have happened!

The problem does not lie in Bayes’s theorem, which nobody disputes (how could they?), but in the way it is written and what the symbols mean. Senn is right that Subjective Bayesians can (and do) say anything, but that doesn’t mean what they say has any bearing on reality (I’ll let you provide the politician comparison).

About that, more next time.

Follow @mattstat

Varying degrees of “fairness.”

“Is fair” can take one of several definitions. Our predicament arises from not being clear which, and by mixing versions at different stages of the problem.

Meaning 1: In any finite number of tosses, the proportion of observed tosses will match the probabilities deduced from the first example; i.e., the observed proportions will show 1/6 ’1′s, 1/6 ’2′s, and so on, or whatever is closest to these if the number of tosses is not divisible by six.

Assuming Meaning 1, and given our evidence, we deduce the probability the proposition is true is 0; it is false. If the proposition were true, we should have seen some combination of five numbers with one missing (e.g. ’6′, ’3′, ’5′, ’1′, ’4′); the missing could have been any number between ’1′ and ’6.’ (I keep the quotes around the outcomes to help us recall these are labels and not numbers.)

Meaning 2: In any finite number of tosses, the proportion of observed tosses will approximately equal the probabilities deduced from the first example; i.e., the proportions will approximately show 1/6 ’1′s, 1/6 ’2′s, and so on.

Assuming Meaning 2, and given our evidence, we deduce the probability the proposition is true is not calculable. The probability is unknown—because “approximately” is not defined. If “approximately” means (and I do not jest) “Leave me alone, I’m tired of playing dice” then the proposition is true, because the observed frequencies are more than close enough for somebody who doesn’t give a damn about dice. If you fail to appreciate this example, you are in for tough times ahead; so pause here and make sure this sinks in.

If “approximately” means “not varying more than 5% from” then the proposition is deduced to be false because, of course, the observed proportions have differed by more than 5%. But if “approximately” means “not varying more than 90% from” then the proposition is deduced to be true, because the observed variations are within this bound.

Who gets to decide what “approximately” means? Well, you do; as does Senn; as do I. Fights start over things like this. What is the one and only definition of “approximately”? There isn’t one! It depends on the situation. As we saw, for some it could mean “Leave me alone”, for others, say casinos, it would be much tighter.

Think this ambiguity bad? It’s even worse than this.

Meaning 3: In any finite number of tosses greater than or equal to 6, the proportion of observed tosses will equal the probabilities deduced from the first question; i.e., the proportions will be 1/6 ’1′s, 1/6 ’2′s, and so on, or whatever is closest to that if the number of tosses is not divisible by six.

Given this and our evidence, the proposition is not true or false (1 or 0) but somewhere in between because we haven’t yet reached the limit of 6 tosses. Kind of. If the die were tossed just one more time (for 6 in total), then there is no way the observed proportions could equal the deduced probabilities. The proposition would then be false. But the die hasn’t been tossed just one more time. It could be tossed 100 more times. Who knows? But we still have the feeling, based on the observations, that the future tosses won’t bring the final proportions in line with the deduced probabilities (I keep repeating deduced to remind us these are not subjective guesses nor are they estimates).

Our evidence and assumed definition isn’t proof the proposition is false, especially if we consider it with respect to Meaning 4, which is the same as 3 but with “approximately” put in usual place. Nor is the proposition true. But we also don’t seem in a strong position to quantify the probability. Nothing in the world wrong with that. Not all probability is quantifiable. See the original series for why.

If we insist on asking the original question, we’re left trying to understand what “is fair” could mean. We need to settle on a definite, unambiguous meaning before we can progress. And then even if we do we’re going to be left will all kinds of nagging questions about real dice.

Real dice have weight distributed unevenly. There’s no way to create perfect balance. We can prove this easily: displaying the numbers, which are of different shape, creates an imbalance, however minuscule. It might be possible to engineer a die down to the level of a quark, so that each side is precisely the same number of quarks across, and that the mass of the die is uniform at the Planck scale (except for the surface where the displays are). In practice, for macro-scale dice, this is impossible. But maybe some physicist will figure it out for some tiny thing. Even then, he won’t be sure that the strings which comprise the quarks are the “same length” everywhere and uniformly (if that even makes sense to say).

But even supposing we have this toy, we have the problem of tossing it. How? Onto what kind of surface? From what height? How much spin? With what downward force? In what gravitational field? After all, if we want to discuss tossing a “fair” die, all these things have to be considered. Tossing is part of “fairness.”

It is at this point it dawns on us that we’re on a fool’s errand. If the die were perfect, as we imagine (and as a logical die is in effect), and if the environmental conditions and forces were known precisely, then we’d know—before tossing—exactly what the outcome would be. Indeed, if the forces did not vary, the die would land the same way each time.

Point is, just by our knowledge of physics we know that any real die and its tossing environment isn’t “fair” in any complete physical sense. There’s no point to the original question. No real die (or its tosses) is “fair” in this sense. The proposition is contingent.

We’re asking the wrong question. What we really want to know is if the die is “fair enough”, and to answer that requires, as above, knowing what decisions we want to make regarding the die.

What we can do is to deduce the probability of seeing any arrangement of observations, either before seeing any observations whatsoever, or conditional on our initial knowledge of six-sided (logical) objects supplemented by a set of observations specified by the evidence. (We do this using Bayes’s theorem: see the next Parts.)

In other words, we can then make statements like this, “Given our evidence about six-sided objects and the old observations, the probability of seeing departures of future observed proportions at least as great as X% from the deduced probabilities is Y.” If Y exceeds a threshold, then we act as if the die is not “fair”, but if it is less than this threshold, we say it is. The threshold varies depending on the application. For the person sick to death of dice, X is unimportant and Y is quite low. Casinos want a small X and large Y for obvious reasons.

We’ll never have 100% certainty that any real die is “fair” in this (final) sense that Y = 0 (for vanishingly small X), because we knew before we started that question dealt with a contingent matter, and we are never 100% certain of contingent matters (though we can be 1 – ε certain).

And you’ll notice that nowhere did we confuse the observed proportions—i.e. the relative frequencies—as probabilities. We knew the probabilities and used them to discern whether the relative frequencies were in line with the them; this is what we meant by “fairness.”

We have proved what we set out to show. That we don’t, at least for the kinds of examples that Senn provided, need two kinds of probability. The one kind—probability as logic—was enough.

Yet there is still more to understand. Stick around!

Update We could also form statements like this: “Given our evidence and old observations, in the next n throws, there is probability Y of seeing X ’1′s” and so forth. In other words, this and the previous example are predictions, statements of uncertainty of the future (or of that which is as yet unseen).

What are the chances a green die will land on top?

Just after the introduction, Senn starts his argument by claiming an “important distinction between two types of probabilities: direct and inverse.”

The distinction is simply explained by an example. The probability that five rolls of a fair die will show five sixes is an example of a direct probability—it is a probability from model to data. The probability that a die is fair given that it has been rolled five times to show five sixes is an inverse probability: it is a probability from data to model.

If we accept this distinction and example as written, we are already lost; all the standard confusions are there.

If probability is all of one sort, then there is no distinction between “direct” and “inverse” kinds. Our candidate is logical probability, in which, as in just-plain-logic, there is only evidence (equivalently, premises), a proposition to be considered with respect to that evidence, and a probability this proposition is true deduced from the evidence.

Let’s begin by rewriting the examples. The evidence is what? Trouble starts with the words “fair die”. This is taken to mean that we have a real, physical, tangible object which must, when tossed, results in equal chance of any side face up. This is asserted and not proved. It is a dictate. It sets in the mind a view of an actual die, of the kind that cannot (or at least does not) exist. Once this die is imagined, objections immediately arise: what if it isn’t “fair”? Can real dice be “fair”? What about imperfections? The confusion between asserting a probability and wondering whether the asserted probability equals the “real” probability, i.e. the long-run frequency of tosses, is already ineradicable. It becomes impossible to keep in mind what the real question is.

Start over rewriting all as a logical argument. “We have a six-sided (logical) object, just one side of which is labeled ’1′, just one side of which is labeled ’2′, and so on up to ’6′, which when tossed must show just one of these sides.” No physical, real die is implied, though because of the ubiquity of dice-like examples, people usually think one is. So if you find yourself unable to imagine a logical, i.e. non-physical, six-sided object, change it to a six-state Martian bleen, a device which is activated by tentacle and displays each time it is activated on a screen one and only one of the figures (translated into English) ’1′, ’2′, etc. There is no hint—as in no hint—of the workings of this device. All—as in all—we know is that the device when activated can show one of ’1′ through ’6′; how it does so is a mystery.

I stress again (and again) that since there are no Martians, there are no bleens. Any imperfections we imagine in a bleen are our own creations and are not part of the evidence supplied. The key to LPB is that we must—as in must—use only the evidence supplied, and all of it, in our deductions of probability. What is not directly implied from the given evidence must—as in, well, you get the idea—be ignored.

Now using the statistical syllogism (which itself can be deduced from simpler principles), we deduce the probability a ’6′ shows on one activation of a bleen, just as we can deduce the probability of five ’6′ activations. Or we can deduce anything which can happen in any (for now stick to finite) number of activations.

We are done with the first example which ends with at a conditional probability; i.e. a probability deduced from given, fixed evidence. All probability is likewise conditional. If you think not, see the series linked above for examples, or see Part III tomorrow for more on this.

Notice that I do not use the word “model”. It isn’t needed. Not here, and in far fewer cases than usually thought.

Senn’s second (“inverse”) example is also confusing. This asks the probability the following proposition is true: “This die is ‘fair’.” The only written evidence is “This die has been rolled five times and has showed five ’6′s.” That we are dealing with a real, physical die is implied from the words, but it is never stated. But suppose this is wrong and Senn meant a logical die or a breen: then where would we be?

Right where we started. If this is the logical “die” or breen, then we start by knowing the chance each number is displayed is 1/6. We end there, too. We have deduced “fairness.”

So we must be talking of a physical, rea-life die. Our task is to interpret this proposition with regard to the given observations.

This evidence is easy and means just what it says: five rolls, five ’6′s of some real die. The proposition is less clear. The subject makes sense: “This die” means some real, actual physical die. The difficulty is with the verb: “is fair.”

Ah, fairness. From youth we are told that there is nothing finer! Indeed, fairness is so fine that we discuss it next time.

Follow @mattstat

Since I will be teaching at Cornell these two weeks, and the topics are the same, I will use part of this time to answer his paper in depth.

It would be best to start here Subjective

Stephen Senn

Probability

Senn went wrong before he even began, with his title: “You May Believe You Are a Bayesian But You Are Probably Wrong.” If you are only “probably wrong” about your belief then you also might be right. And if you were certainly wrong, then we would have a proof which says so. A proof is a string of deductions, i.e. a valid and sound argument, which begins with obviously true premises (agreed to by all) and ends at a proposition we must believe—even if we don’t want to.

Senn does not have, nor does he claim to have, a proof which shows being a Bayesian is certainly wrong. It is only his best guess that this philosophy is wrong. Probably wrong. So here we are, already at probability. What could Senn mean by his probabilistic statement “probably wrong”? (Besides the pun, I mean.) It can’t be any kind of frequentist statement, as in “I’ve collected a ‘random sample’ of Bayesian philosophies, itself embedded in an infinite sequence of such philosophies, and the mean of this sample (considering errors in theory equal to zero) tends towards zero.” That makes no kind of sense, as I’m sure Senn would agree, but it would have to if probability was frequentism.

Bayesian philosophy, at best, comes in a finite number of flavors. It could be that some of these are false (I agree subjectivism, as it is usually understood, is), but in no way can we imagine any individual theory as being embedded in an infinite sequence of theories, which is required for frequentist theory to hold. No: either we can prove each theory true or false, or our evidence is not (yet?) sufficient, and thus we are only probably sure each theory is true or false. This sounds like a Bayesian statement, no? (If so, do we fail because of self-reference? Well, no, because we can build this theory from simpler propositions.)

It could be that Senn took a subjective Bayesian tack when he formed his title, or perhaps he took a logical probability, or objective, Bayesian one. (Incidentally, I’ll call this latter theory LPB for short.) Or he could have meant some as yet unknown (or at least unidentified) theory. Whatever it was, it couldn’t have been frequentism, as shown.

His leading candidate is eclecticism (Senn is not frequentist), which is one of two things. One is no belief at all. It means “I’ll do whatever I want whenever it seems good to me.” There is no theory here to disprove, nor prove. To say “I’m an eclectic” this way means “I don’t want to argue for anything, just against things.” Since we go nowhere engaging with this “theory”, we pass on to number two. This is to say, “I’ll take a little of that, some of this, and some of the other.” Here we have several sub-theories. As such, this kind of eclecticism is actually a whole theory (the compilation of sub-theories) which might be true or false. Thus Senn might have used Bayes for his title and he might use frequentism for (say) dice tosses.

Senn recalls that Fisher himself was “skeptical” of attempts to unify probability. Hacking, another Big Cheese, in line with other well-aged curds, is of the same opinion. Why should we have a theory? Why not many? The obvious answer to this is that there is that which is true and that which is false and we should seek the truth. If it turns out a theory of probability works for all kinds of uncertainty, we’re stuck with it. If it must be that several theories are true, then we must accept them all. But it’s wrong to use desire or suspicion as proof there are many and not one theory.

Senn himself proved that frequentism is out (and forever) as a complete theory of probability because it cannot handle propositions like his “probably wrong.” But this isn’t proof that Bayes everywhere right; not yet. Senn’s later examples might be sufficient to show all versions of Bayes are wrong, in which case some other theory must be true.

But we’ll have to see next time, because we’re already out of space, and because next topic isn’t simple.

Follow @mattstat

The class works by me holding forth with dulcet but brief pontifications followed by intense questioning of the students, as a cop might grill a suspect. “What did I just say? What in the dark-mattered universe do you think I meant by that? Have you signed up for the wine tour yet?”

The wine tour—completely unofficial and off the books—ends Week One with a journey to several Finger Lakes wineries to sample their wares. To be cruelly honest, many of these are poor. If the wines aren’t sour and vinegary, they are so sweet you could stand a teaspoon up in them. One unbearable vineyard (the name of which is hidden in a riddle) produces nothing but pinkish paint thinner. But everywhere the wines are wet and contain (among other chemicals) ethanol, which is welcome after five full days of statistics statistics statistics and with another week of the same to come.

(But there are dangers, too. At one stop on the wine trail, I was once nearly abducted by a bachelorette party and had to be rescued by one of my students.)

The class contains almost no math and certainly no memorization of formulas. I figure the computer can do those things for you, and that time spent proving things mathematically removes time spent in understanding what probability is and learning the strengths and limitations of statistics. As regular readers know, the latter are many, nefarious, and ubiquitous.

I have only one or two canned examples. The rest have to be provided by the students themselves. This eliminates having to figure out a whole new field and its data and how to describe its uncertainty. Besides, textbook examples are far too neat, even coy. Better to see how messy, compromising, and ambiguous collecting data is. Gives a far better appreciation of the ease of making mistakes and the resultant over-confidence.

I teach R; successfully, too. Yes, it is a programming language, but that is its great advantage. I was able to teach R to a man who did not know what a spreadsheet was and could not type. He did not own a computer. This wasn’t because of my ability, but because learning the rudiments of any logical programming language is something almost anybody can do. (I do not include SAS in this list; it is an appalling language.)

Following my custom, for the next two weeks posts will reflect, broadly or in detail, what is going on in the class. I won’t have time to do anything more. Feel free to ask questions, but understand I might not be able to get to all of them.

Follow @mattstat

Update A good joke.

Father of the Martin Amis. (Original much superior to its remake.) #swapwordinafilmtitlewithMartinAmis

— William M. Briggs (@mattstat) June 14, 2013

A sure cure for cancer?

Yes: it is true that the next tobacco ad you see could lead to a cure for cancer. How?

I haven’t the slightest idea. I don’t have to know how, either. The headline is still true as long as somebody, sometime, somewhere could describe how, even if the description is only “in theory.”

My powers of imagination are weak, so I’ll rely on you to divine the path from tobacco ad to cancer cure. What I’ll do instead is distract you from thinking about this difficulty and talk about the glories of a cancer-free world.

Hey! No more cancer! Now that would be a fine thing. Right? No more pain, no suffering, no tears, grief, misery. No more mothers burying their blighted-by-disease daughters.

Like Sally Q. Evalston, 42, a Pinewood, Illinois elementary school teacher, beloved three-time winner of Teacher of the Year, who was carried away before her time with capital-C Cancer (which she “battled” with). Just you think about her. Look at her picture, feel for her mother, weep with her students.

This is the sort of tragedy that could be avoided thanks to our truthful headline. Admit it: you feel good thinking about this, don’t you? Isn’t it nice to be part of the cure for cancer, albeit in small proportion? Maybe you can email your Congressman (or woman!) and let him know you’re on his side, that you’d support him if he voted to increase funding for tobacco advertising. You could at least frown with severe disapproval at the next person you meet who suggests he’d rather not see more tobacco ads.

Assimilated all that? Then here is another true headline, “Tobacco Ads Could Lead to Daily Teen Smoking for Kids 14 and Under“.

Wait. Didn’t we just say that tobacco ads could cure cancer and now a rival claim says these same ads could cause kiddies to smoke? We did: both headlines are true. And so is is true that “Tobacco Ads Could Lead To More Cancer”. Just as it is true as true that “Tobacco Ads Could Lead To Mars Mission” or “Tobacco Ads Could Cause Nancy Pelosi To Stop Speaking Gibberish.”

The magic happens in could. Adding it—or might, may, possibly or the like—turns any proposition about the contingent into a truth. (Contingent = not logically necessary.) Anything contingent could or might be true; that is the nature of contingency. So adding a word like could in a contingent proposition merely makes the proposition tautological, and all tautologies are true.

Headlines like today’s are cheap journalist tricks; one of the most common, too. “Could Lead To” headlines and ledes betray the reporter’s prejudices and desires and make at best weak claims about reality. And the following articles usually fall prey to the standard human failing of searching only for supportive evidence, assuming that contradictory theories are the first refuge of scoundrels and “deniers.” No idea of the uncertainty in the claim of the headline ever appears.

Just for fun, I did a search on “Could Lead To” (surrounded by quotes; try this yourself). “Repetitive soccer ball ‘heading’ could lead to brain injury”, “10 nail deformities that could lead to bigger health problems”, “Heavy rain could lead to explosion in mosquito population”, “NYCHA Budget Cuts Could Lead To 500 Jobs Lost”, “Crowdfunding help could lead to a sandwich named after you”, “NHS changes could lead to hospital being sponsored by junk food firms.” An endless, ever-increasing stream.

And isn’t it curious that all of these, tacitly or directly, argue for government intervention?

Follow @mattstat

“My p-value was this big.”

Well, just a little (all emphasis mine and joyfully placed):

Statistical significance is junk science, and its big piles of nonsense are spoiling the research of more than particle physicists…

But here is something you can believe, and will want to: Statistical significance stinks…

The null hypothesis test procedure is not the only test of significance but it is the most commonly used and abused of all the tests. From the get go, the test of statistical significance asks the wrong question…

In framing the quantitative question the way they do, the significance-testing scientists have unknowingly reversed the fundamental equation of statistics. Believe it or not, they have transposed their hypothesis and data, forcing them to grossly distort the magnitudes of probable events…

They have fallen for a mistaken logic called in statistics the “fallacy of the transposed conditional.”

And that’s just the first part. I couldn’t finish the second because my eyes were overflowing with happy tears.

Ziliak and pal Deirdre McCloskey, incidentally, co-authored the must-read The Cult of Statistical Significance.

Cult, they say. Cult because there is an initiation at high price. Cult because statistical “significance” is invoked by occult incantations, the meaning of which has been lost in the mists of time. Cult because these things can not be questioned!

The p-value is a mysterious, magical threshold, an entity which lives, breathes, and gazes sternly over spreadsheets; a number gifted to us by the great, mysterious god Stochastikos¹. It was he who decreed that great saying, “Oh-point-oh-five and thrive; Oh-point-oh-six and nix.”

Adepts know the meaning of this shorthand. So 0.050000001 is sufficient to cast a result outside the gates where there is weeping and gnashing of teeth. Yet 0.04999999 produces bliss of the kind had when the IRS decides not to audit.

Members cannot be identified by dress but by their manner of speaking. Clues are evasiveness and glib over-confidence. They say, “The probability my hypothesis is true is Amen” when what they mean is “Given my hypothesis is false, here is the value of an obscure function—one of many I could have picked—applied to the data assuming the model which quantifies its uncertainty is certainly true and that one of its parameters is set to zero and assuming I could regather my data in the same manner but randomly different ad infinitum.”

In the hands of a master, more significant p-values can be squeezed out of a set of data than donations Al Sharpton can secure by marching into an all-white corporation’s board room.

“Statistically significant” does not imply true nor useful nor even interesting. “Significance” is a fog which emanates from a computerized thurible, thick and pungent. It obscures and conceals. It woos and insinuates. It distracts. It is a mathematical sleight-of-hand, a trick. It takes the eye from the direct evidence at hand and refocuses it on the pyrotechnics of p-values. So delighted is the audience at seeing wee p-values that all memory of the point of a study vanishes.

Statistical significance is so powerful that it can prove both a hypothesis and its contrary simultaneously. One day it pronounces broccoli as the awful cause of splentic fever and tomorrow it is asserts unequivocally that broccoli is the only sane cure for the disease.

Both results will be accepted and believed, especially by those manning (and womanning!) bureaucracies and press rooms. Journalists won’t tell you about the deadly effect of either until 10 p.m. Government minions will latch gratefully on to anything “significant” as proof their budget (and therefore power) should be increased.

Time for statistical significance to be slain, its bones cremated, and its ashes scattered in secret. No trace should remain lest the infection re-spread. The only word of it should appear in Latin in tomes guarded by monks charged with collecting man’s (and woman’s!) intellectual follies.

Follow @mattstat

Update Wuhahaha!

———————————————————————————

Thanks to Steve E for finding Ziliak’s piece.

¹I didn’t think of this; I recall the name from the old usenet days.

The reason is obvious: academics must publish whether they want to or not, whether or not they have anything useful to say, and whether or not anybody reads what they write.

The glut appears across all areas of knowledge, but the effects are different in the humanities and sciences. In the former, the world would be a far better place had many of its practitioners obeyed the ancient truism that silence is golden. Over-supply in the sciences is less troublesome because poor and inconsequential works are ignored. The presence of this chaff only makes it difficult to discover the wheat.

In the humanities (which I take to incorporate the gooier sciences, like education) one can say anything, the more outré the better. Not so in the hard sciences where at least some passing resemblance to the truth is expected.

Too much resemblance, as a matter of fact. Editors, reviews, and authors follow a rigid positivistic philosophy: only good news shall find its way into print! Papers with “statistically significant” effects are vastly likelier to be published than are works which admit there’s nothing to see. Failures with billets are as rare as Republicans in English departments.

Then because traditional statistical methods used are fertile in labeling results positive, even when they are not, there exists a tremendous publication bias. Many false things are believed true.

All this is known and of concern to the seventy-plus signatories to the article Trust in science would be improved by study pre-registration in The Guardian. This open letter proclaims “We must encourage scientific journals to accept studies before the results are in.”

The eminences lament publish and perish and say the “publishing culture is toxic to science.”

Recent studies have shown how intense career pressures encourage life scientists to engage in a range of questionable practices to generate publications — behaviours such as cherry-picking data or analyses that allow clear narratives to be presented, reinventing the aims of a study after it has finished to “predict” unexpected findings, and failing to ensure adequate statistical power. These are not the actions of a small minority; they are common, and result from the environment and incentive structures that most scientists work within.

It’s worse than just that. “[J]ournals incentivise bad practice by favouring the publication of results that are considered to be positive, novel, neat and eye-catching.” Although there is no conceivable universe where the string of letters which comprise “incentivise” should be used when ladies are present, we cannot help but agree that the situation is grim.

The “file-drawer” problem adds to the misery. This is when a study which is not a success or isn’t sexy or part of the consensus rests in a lonely file in the forgotten reaches of a scientist’s computer. Lack of negative results in print gives an over-optimistic picture of scientific progress.

The solution the Guardian writers have is to publish “pre-registration” papers, outlines of the studies which are not yet conducted. Journal which air these outlines must agree to publish the eventual results whatever they may be. Thus “questionable practices to increase ‘publishability’” will be “greatly reduced.”

I doubt it. Authors will still aim for high “impact factor” journals for their “pre-registrations.” The “impact factor”, incidentally, is an “arguably meaningless as an indicator of scientific quality”, though always a matter of bragging rights.

There will be a minor flood of papers pre-registering sketchy theories, and these will be all that is remembered. Some authors will publish their negative results, but many will forget them and move on to more fertile grounds. The bulk of these maybe-so works will be taken as positive evidence even if positive effects are never found or if negative effects are published.

Journalists, by nature not very inquisitive, will tout “If these promised results hold…”, and again these reports will be all that is remembered. Retractions will never appear. What’s to retract?

And worst of all will be the huge increase in papers that must be navigated to get to the good stuff. Pre-registration papers will only be “read”—i.e. their abstracts will be glanced at on PubMed—by other authors looking to pad their bibliographies.

No. The real solution is to judge a fellow by the quality and promise of his work, not by its quantity, and not by even a hint of a numerical rating.

Follow @mattstat

——————————————————————————–

Thanks to Bryan Davies for pointing us to this.

We’re spying on you for your own good.

He compared NSA’s sucking up data to the police carrying guns and using other tools, and that these guns and tools can be used maliciously by those sworn to protect us, and sometimes are, but not to the extent we would disarm cops.

He didn’t use the word “spying.”

The analogy is poor and if considered at length supports the opposite conclusion: NSA ought not to spy on citizens without probable cause. Which is to say, NSA, and every other government agency, ought to follow the Fourth Amendment in its letter and spirit.

If the police suspect a man of crime they must provide sufficient and lengthy evidence to a judge before receiving a search warrant to implement a wire tap (and equivalents). After the warrant is issued, the tap is put in place. Contrariwise, the NSA wire taps everybody first, stores the information, and then swears not to look at it until it receives a warrant from a court with the sole purpose of handing out warrants.

This violates the spirit and letter of the Constitution. Whether our dear leaders come to this same conclusion and voluntarily renounce the power they have awarded themselves is a separate question. Smart money says no: smarter money says these programs will intensify.

Add to this knowledge that the chief law enforcement officer of the country was discovered lying to both Congress and a judge for political reasons, accusing one of his enemies falsely of treason and spying, a double irony (but unfortunately no record).

Police carry guns because crooks do. Except in a very few enlightened locales, such as blood-stained Chicago, any citizen may arm and protect himself in the absence of police. The armament on both sides is in rough parity, with the police holding a slight edge.

There is no equivalent in this case. No citizen can compete with or evade a Dark Star or PRISM, the 007-enemy-like names our beneficent government chose for its programs (this was also Krauthammer’s observation). All a citizen can do is avoid anything electronic: no cell phone, no computer, no medical records (which the IRS will soon have), no credit card, no car, and no heat signature—drones will track these. Remember drones? Only proles and animals are free.

The best comparison is between Stop & Frisk and NSA surveillance. Under Stop & Frisk, cops patrol high-crime areas and detain and search individuals who meet minimal suspicion standards. Minimal is not none. The visible presence of the police is often a sufficient deterrent. Records of Stop & Frisk cannot be used for personal identification.

The NSA can scan records which are minimally suspicious, too. But to do that, they must first collect information on everybody, including people who meet no standard of suspicion. A person frisked on the street knows what has happened to him. The citizen whose records are pored over remains ignorant. The former can complain of abuse, the latter cannot. This is why there are no complaints of harassment.

Revealingly, the government does not claim electronic snooping does deters, only that it discovers. Perhaps people would feel better were they to learn the spying programs have uncovered innumerable foreign spies and stopped countless attacks. The government is understandably reticent to admit these successes, assuming they exist, fearing its enemies will backward engineer its methods and thus countervail them.

However, there is good reason to think the number of these successes is low. For one, our current administration loves to brag, and it has not about uncovering spies. And the failures have been many. The Boston bombers, the shoe and underwear bombers, Mumbai, Benghazi, England’s bombings; many others including Snowden himself. Contradictorily, these will encourage governments to increase its powers rather than admit its programs don’t work.

I had, in the 1980s, a Top Secret SCI etc. etc. clearance, natural for a USAF Sergeant in cryptography. The clearance came with endless briefings about traitorous spies and the damage they caused. These were mostly useless as training to spot spies, because all we learned was that spies excel at hiding. But because spies existed and the government feared them, it thought it should screen for them using lie detectors.

These were not useless: they were able to produce many accusations, all (or nearly all) false. Real spies pass polygraphs. (Incidentally, the attack is psychological: “There seems to be a problem with question seven, Sergeant Briggs. Can you help me with that?” “Nope.” “Are you sure?” “Yep.” “You are dismissed.” Always the same.)

It’s a good guess the same will and is happening with NSA’s terrorist screening. The failures suggest screening is too blunt a tool to catch bad guys, especially well organized and intelligent adversaries.

Follow @mattstat

Update Recommended reading. Liberty in the Tentacular State.

We’re spying on you for your own good.

We hear this primarily from the young, who are said to be used to living openly on-line, and from the old who share this in common with the young: ignorance of history and of human nature. Today, and for the benefit of these mis-, or rather uninformed, people a brief lesson.

No man is innocent in the eyes of his enemy. A zealous prosecutor can take the most harmless of circumstance and through innuendo, contrivance, and brazen lie turn it into at least dark suspicion if not “proof” of heinous crime.

You bought a book on the Middle East because of a noble interest in history? So do terrorists read those books. You made a phone call from nearby a mosque? So do terrorists make these calls. You made a crude and in poor taste joke after an incident? So do terrorists make these quips.

You decided to join a local Tea Party-like organization because of your sincere belief in limited government? Or have you renewed your membership in the NRA? So do “domestic”, “home-grown”, “self-radicalized” terrorists join these groups.

Have you not heard the term railroaded? How about framed? How about falsely accused, hounded, harassed? Is it is merely paranoia and lapsing into extremism to suggest that the government, sated on your secrets, could act in these ways as it has and far too often?

Could the IRS target groups which it perceives as its enemies? Could sealed divorce records be publicly aired? Could a zealous prosecutor who cares only of her public image and is a stranger to truth convict the innocent? Could a government libel and slander a man in order to bamboozle a judge into issuing a warrant against this man?

The powers of my imagination pale, but a story of your culpability can always be weaved by a determined enemy. Anything can be turned against you, and the more information government has on you, the easier it becomes to manufacture “evidence” of your misdeed.

Information is power: it is the lifeblood of politics. Giving bureaucrats and politicians this much power is to tempt them beyond human ability to resist. (Giving power to computer and statistical algorithms used to data mine records is no solution. These cannot be perfect, and it is people who run them.)

It’s probably far too late to remind anybody of these words, taken from the document which at one time dictated the law under which even politicians had to live:

The right of the people to be secure in their persons, houses, papers, and effects, against unreasonable searches and seizures, shall not be violated, and no Warrants shall issue, but upon probable cause, supported by Oath or affirmation, and particularly describing the place to be searched, and the persons or things to be seized.

It has now become a “reasonable” search to track your every movement, your every call, perhaps even your every email and on-line transaction. It is now “probable cause” that you are guilty of a crime just because you exist.

“But it’s only meta-data.” Meta-data forsooth! Have you any idea what this is? It tells the time of your call and where you were when you made it. It tells who you made it to, and tells of the people you contacted who they in turn contacted. Do you text? Then they have that information too.

It’s rumored the government even has your emails. Perhaps not the text of these, but the meta-data. Again, this tells much. It tells where you were and on what you wrote them. It tells the time and length. It tells who it went to, and it tells this of everybody.

Even without your exact words spoken or written, this is a dense and nearly complete picture of your behavior. If a bureaucrat cannot find something in this trove that at least casts you in a bad light, then he isn’t trying.

“But what about the children! We demand safety!” I have yet to hear any politician respond to these words of a man of a far superior mind:

Those who would give up Essential Liberty to purchase a little Temporary Safety, deserve neither Liberty nor Safety. Ben Franklin. #NSA

— William M. Briggs (@mattstat) June 6, 2013

If the government can’t catch terrorists without spying on its own citizens, then tough luck. Let if find some other way. The price we have to pay for this program of extremely limited success is just too high.

If NSA algorithms so good at detecting spies, how did Edward Snowden go undetected?

— William M. Briggs (@mattstat) June 10, 2013

Update Until my server’s DNS problems (not “issues”) are resolved, you might not be able to see the tweets linked. They are, in order:

“Those who would give up Essential Liberty to purchase a little Temporary Safety, deserve neither Liberty nor Safety. Ben Franklin.”

“If NSA algorithms so good at detecting spies, how did Edward Snowden go undetected?”

Update See this from brother statistician John Cook: A statistical problem with “nothing to hide.”