William M. Briggs

Statistician to the Stars!

Category: Statistics (page 1 of 361)

The general theory, methods, and philosophy of the Science of Guessing What Is.

A Simple (But Not Short) Stats Question

From reader JH:

Thank you again for taking the time to write your blog. It is exceptionally thought-provoking.

My job involves much of the six sigma “capability” studies and such. I have lots of tools available to “quantify” our measurement data. But I’m wondering now if much of this approach isn’t baloney.

Given my questioning of our corporate orthodoxy, I decided to try a different approach to testing one aspect of a part. I’m curious what embedded fallacies I may have indulged in doing so?

Let’s say I want to demonstrate that the torque required to damage something is > 80. I have a capable measuring system and a representative sample and my uncertainties are (to our knowledge) randomly distributed (or presumed to be, given they are unknown).

The first measurement comes in at 280. Statistical value? Worthless. It’s a single data point.

The second data point comes in at 270. Ok, perhaps slightly less worthless. I could theoretically calculate my mean (275) and standard deviation (5) from a whopping two data points. Indulging corporate orthodoxy, I could say that my goal of > 80 is a whopping 39 standard deviations below my mean, and generate some impressively high probability that all future parts are going to be > 80. From two little points.

Has anything actually been demonstrated? If I shift the “mean” down to the lower-95% T-value (which is ~230.077 for these two points), can I claim ‘there is at least 95% probability that the population mean is > ~230.077’? If so, that still doesn’t let me calculate P{x>80} unless I assume some kind (Gaussian kind?) of distribution.

It feels wrong, but I can’t clearly articulate the source of error beyond assuming a “normal” distribution just because. Who knows what the actual distribution is?

Terrific set of questions. First, there is no actual distribution. Statistical, or rather probability, distributions are purely epistemological, i.e. measures of information, and are not properties of actual things.

Now this torque you are measuring. It actually was, once, 280, and another time 270. You have thus conclusively demonstrated the torque is, or can be, greater than 80. That is, given the measurements and assuming the measurements are without error, the probability the torque can be greater than 80 is 1. You are certain.

Questions like this next one are entirely different: M = “The next measured torque will be greater than 80.” The probability of that given just your two measurements—and nothing else—is, as is obvious, greater than 0. How much greater than 0 cannot be quantified unless you are to make (what are probably) ad hoc assumptions. Or the probability doesn’t have to be quantified, but can be made sharper if you were to add implicit information about these kinds of torques. Something like, “Given my experience of these kinds of machines, 280 and 270 are way above 80”, and then the probability of M with that implicit premise and given the two measurements is “high”.

Again, to say how high is “high” is requires ad hoc assumptions. Saying a normal distribution represents uncertainty in torque measurements is one such assumption. Then you can say the probability of M given this ad hoc assumptions, and given the two measurements, but leaving out the implicit expert knowledge about your experience.

This is all fine because, as is proved in this one-day best seller, all probability is conditional. Probability is not a property of any system, which is why there are no correct distributions to use—unless the probability can be derived from information you know or assume to be true about the process at hand. That kind of information appears to be missing here.

So, yes, Pr(M > 80 | x = 280, 270, and assuming a normal distribution with certain known central and spread parameters 275 and 7.07) = 1 (or, rather, .999 with about 160 or so 9’s). That probability will be less if you assume you don’t know the parameters and they are instead estimated from the data (something like .999999).

These are the correct numbers given these assumptions—and no other assumptions.

Instead of a normal, you could have used your ad hoc freedom to use any of hundreds of other standard distributions, and none of these would be any more correct. That is, they all would have been correct. Conditionally correct. Since the distribution is not derived, or deduced, from known causal principles about the process, that’s the best you can do.

Unless you bring in outside, expert knowledge. We saw above how that works: and it works well. Problem is, the hunger for quantification. Management wants a hard number, not an opinion. It rarely matters were the hard number comes from; that it is hard is what counts. This is why Six Sigma is so beloved. It gives precision where precision is desired. Not that it is giving useful precision, or numbers from which excellent decisions will be made.

The final answer is—drum roll, please—there is no answer. Unless you’re willing to live with expert knowledge and non-quantified probabilities, there is no way to come to numerical probabilities without making ad hoc assumptions.

You can use history, too, as expert knowledge. For instance, you’ve found normal distributions to work well in the past, hence you use them again. This is weakly ad hoc.

To Undergo Chemotherapy Or Not

Headline: “How I used math to conquer my cancer” by Michael Kaplan.

Gist: fellow named Reitzen, 45, discovered he had kidney cancer.

“The doctor came up to my house and had fantastic bedside manner,” says Reitzen, now 57. “He told me that the tumor was larger than the kidney itself, which would necessitate removing my kidney and the lymph nodes around it.”

But Reitzen didn’t want a body part removed — unless data showed it to be absolutely necessary — so he sought a second opinion.

So Reitzen used “data mining”, i.e. he conducted a search of the Internet, to find the kind of surgeon he wanted. He found two. One “thought that it would be better for me not to lose [the kidney] because I had high blood pressure…He had no bedside manner, but I liked his opinion of there being a 70 percent chance that I would not be left with just one kidney.”

The cancer was duly hacked out. “But the oncologist/hematologist who gave him the original diagnosis suggested chemotherapy to avoid a recurrence.”

Using decision-theory math, Reitzen took into account the likelihood of surviving with and without chemo. With the chemo, there was an average life expectancy of 8.1 years; without it, he would be looking at 7 years.

But taking chemo’s side effects into account, his quality of life would be only 70 percent, which he based on information from health-related quality-of-life studies on chemotherapy patients. Without the chemo, he would have no side effects and 100 percent quality of life. After doing the math, quality-adjusted life years came to 5.7 with chemo and 7 without.

“Would you want to exchange a 15 percent life-expectancy increase for a 30 percent drop in quality of life?” he asks.

Quality-adjusted life years? Yet another attempt to quantify the unquantifiable. Wikiwik’s dry statement puts it best: “To be dead is associated with 0 QALYs, and in some circumstances it is possible to accrue negative QALYs to reflect health states deemed ‘worse than dead’.”

Reitzen’s calculation is simple: 8.1 years of “expected” life left under chemo times 70% (0.70) of “worthy living” equals 5.7 QALY. And 7 years “expected” life without chemo times 100% (1) of full life equals 7 QALY. Since 7 is greater than 5.7, the calculation says to pick no chemo.

Which was his choice: no chemo. “Ten years later, Reitzen is cancer-free, with two functioning kidneys, and did not have to endure the misery of chemotherapy. The treatment that he was initially offered has been deemed ineffective for kidney cancer.”

To which we say, God bless him. But do note, and do pause, at the statement The treatment that he was initially offered has been deemed ineffective for kidney cancer. And then recall that the “expected” 8 years he was to live with that now-deemed-ineffective treatment was calculated on the belief (or assumption) that the treatment was effective. All probability is conditional.

While Reitzen routinely makes critical investment decisions with the guidance of math, and found comfort in his cancer-by-the-numbers strategy, doctors don’t think his approach to treatment should be relegated only to statistics-loving professionals.

To which we say capital-A-men. A doctor’s “loss” and “gain” are different, and sometimes far different, than a patient’s. The internal “calculations” the doctor uses to recommend a specific treatment might be best in his mind and for him, but not for the patient. Only the patient can decide what a “worthy” life is.

Now the real problem is only partially the attempt to quantify the unquantifiable. A life’s worth cannot be put to a number, and doing so can only be the crudest of approximations. Collapsing a life’s worth to a single number necessarily strips away vast amounts of information. And that means bad decisions can be made.

The second difficulty is, even is all can be quantified, all should be quantified in its fullness. When it was believed (by at least that one doc) that chemo would work, this doc’s calculation said patients would live an “expected” 8 years. That number is found by multiplying the probability (conditional, as all these probabilities are, on the doc’s belief in the treatment) a patient would live one year by 1, the probability the patient would live 2 years times 2, and so on, the result being a weighted average.

Collapsing these probabilities to one “expected” number again strips away information. The whole swath should be presented to the patient—not necessarily at the finest levels. Do we really need information per year 10, 20 years out?

Well, you get the idea. What is most crucial to grasp is not all this numeric mumbo jumbo, but that not accepting harsh treatment is an solid option. Though it cannot be quantified, a shorter life without the brutal suffering caused by a treatment can be much better than one with it. Especially when it is, as it always should be, recalled that All men are mortal, etc.

Thoughts On One-Time Pads For Cell Phones

Best short description of one-time pads (OTPs) is from Jason Matthews, who is describing their use in the heyday of the cold war in the book Strangers on a Bridge by James B Donovan.

Before the advent of automatic enciphering technology, secure radio communications between an intelligence headquarters and its agents in the field were abetted by use of one-time pages (OTPs, sometimes referred to as “cut numbers”). These cipher pads were individual sheets of printed rows and columns of five-digit numerical groups. The pads were bound with rubberized adhesive on all fours sides, and normally printed small for concealment purposes.

A field agent would receive a shortwave radio broadcast from headquarters via one-way-voice link (OWVL.) These OWVL broadcasts consisted of a monotone female voice reading a series of recited numbers—an enciphered message. The agent would record the recited numbers in five-digit groups and subtract them on the correct OTP page. The resultant values would correspond to the 26 letters of the alphabet and reveal the message. Because each page of the OTP is randomly different and used only once, looking for patterns in cryptanalysis is futile. It is an unbreakable cipher…

Indeed it is unbreakable. Eat your heart of quantum cryptography! Because, in essence, every character in a OTP is separately encrypted, and each pad used only once, the code is impossible to break. I use impossible in its literal sense. No computer no matter how powerful running for any amount of time can decipher the message. That is to say, unless the “key” which generates the OTP can be discovered.

Since random means unknown, the “secret” to key generation is an unknown process. Here, of course, “quantum” events can be used, say, in the form of static of radios tuned to unused stations—as long as that static is atmospheric, or preferably extra-gallactic, in origin and thus unpredictable. Using any kind of “random number algorithm” produces, as all experts know, perfectly predictable, deterministic keys. (This, incidentally, is why in Uncertainty, I recommend against simulation methods.) Also, the device used to capture static must itself be as “noise-free” as possible, since known circuitry could generate predictable signals.

OTPs were used well after the advent of “automatic enciphering technology”. I recall in the early 80s listening on shortwave to “numbers broadcasts”, almost always in Spanish and male voices, in San Antonio. (Not only did I get my start in the Air Force in a cryptographic specialty, I was and am a “ham”; back then I was KA5YHN and am now K2JM.)

Shortwave broadcasts have the added benefit of disguising the intended receiver, which could be anybody with a radio and a length of wire. This is important to discourage “SIGINT“, or signals intelligence, which is the study of where, when, and how signals are sent. A surprising amount of information can be gathered about an encrypted message, even if the cipher is never broken, simply by paying attention to the transmission. SIGINT is called “meta data” with respect to your cell phone and computer messages, and that “secrets” about you can be discovered using it alone and ignoring the actual content of your phone calls and emails is why we don’t want the government, or other sources, evasedropping on our conversations.

Real OTPs must be destroyed immediately after use, or the cipher can be broken. They must be used only one time, or patterns will stick out like a Republican in an Anthropology department.

Now, with our hand-distractions, it is easy to store very large electronic OTPs (which can be used in encrypting text or digitized voice); it is even easy to generate keys, assuming the cautions about unpredictable generation are minded. The problem comes in swapping keys with recipients. You have a cell phone on which is the OTP App. How do you communicate this key to your friend? The key has to migrate from your device to his. It could do this via Bluetooth, but doing so exposes the key to the world. The device itself, unless it is well shielded against electronic emanations, can leak the key (this is called Tempest security). The key may be shifted to something like a thumb drive or SD chip, and then the chip inserted into your friend’s phone. The chip must then be destroyed, as in utterly, or otherwise rendered unreadable (perhaps by rewriting on it new unused keys many times).

This meeting between you and friend must take place. You can’t use an old key to transmit a new one, because with OTPs it’s digit-for-digit: compression of keys is impossible. Transmission of the key over the air or, say, internet exposes it. Anything short of a hand-to-hand swap exposes it. Since a meeting must take place, the usefulness of OTPs is limited. But very useful is absolute, unbreakable security is desired.

There are more problems, besides Tempest leakage. Suppose you are receiving the encrypted message from your friend, and decrypting on your device (ignoring electronic leakage, which is no small consideration). The device will still have the key and the plain-text message! Of course, this is no different a situation than the spy who sits in his room and has on hand the OTP and decrypted message. But a small piece of paper, or two, is easier to destroy and conceal than a cell phone.

This means the key must be self-destroying. As it is used, the places on the storage device must be re-written dynamically, and in such a way that no fine probing will ever reveal what was originally written. No easy task. And the same must happen to the message itself, after it is made use of. For voice communications, this is easy, because they’re (forgive me) in one ear and out the device. But texts (or emails, etc.) must be guarded more zealously.

OTPs are in use still on the internet, with otherwise innocuous web pages and images containing updated version of the five-number groups. Decrypting short messages can, and surely are, still processed by hand using paper OTPs. But long messages or other formats is not different than the two cell-phone case. Key swapping must still take place—as it did with paper OTPs, of course.

SIGINT for cell phones, and even web sites, is still a problem. Even thought the OTP App works as desired, your enemy will still know when you sent the message, where you were when it was sent, where your friend was when he received it, and how long that message was. That last item is perhaps the most revealing. So lucrative, if I can use this word, is this that stations have taken to swapping continuous messages so that outsiders never know when the real one starts and ends and how large the message was.

One last point about spoofing. A concern is that an enemy agent can inject numbers into the “code stream” which might mistakenly be taken to be real by the recipient. But unless the spoofer knows the key, and therefore hasn’t much need of spoofing, injection is immediately detectable. Which is also a boast of quantum-key cryptography. In that, incidentally, key swapping must still take place, though it is of a different form.

Conclusion? For cell phones, anyway, the whole thing is possible, and not even so difficult. The problems are signal leakage, lost phones, SIGINT and of course the key swapping. Just as with paper OTPs, we aren’t limited to only two phones, but an indefinite number in a network.

I always wanted to try this, but I am not a coder (though I code). The ideas are so obvious they must already be in use somewhere, but I’m too lazy to look them up.

Guardian’s ‘How Statistics Lost Their Power’

Been a wealth of material lately, which explains why we’re just now coming to the Guardian’s, “How statistics lost their power — and why we should fear what comes next“. Now by statistics the author means the old-fashioned, and really to be preferred, definition of “measurements taken on observables”, and not models and math, or not models and math per se.

In theory, statistics should help settle arguments. They ought to provide stable reference points that everyone — no matter what their politics — can agree on. Yet in recent years, divergent levels of trust in statistics has become one of the key schisms that have opened up in western liberal democracies…

Rather than diffusing controversy and polarisation, it seems as if statistics are actually stoking them. Antipathy to statistics has become one of the hallmarks of the populist right, with statisticians and economists chief among the various “experts” that were ostensibly rejected by voters in 2016. Not only are statistics viewed by many as untrustworthy, there appears to be something almost insulting or arrogant about them. Reducing social and economic issues to numerical aggregates and averages seems to violate some people’s sense of political decency.

Not trust the ever-burgeoning bureaucracies official numbers? Heaven forfend! Why, bureaucrats, NGO flacks, and politicians are entirely disinterested actors, who only want what is best for one and all. Yes? They would never consider cooking books so that things came out in favor of requiring more of their services, would they? No, sir!

After all, liars figure but figures don’t lie. Yes? To gauge unemployment, all we have to do is count those without jobs, right? And the number of folks needing a government service? Easy too, with no chance of bias. Yes?

Well, that’s counting. There is also modeling.

The declining authority of statistics — and the experts who analyse them — is at the heart of the crisis that has become known as “post-truth” politics. And in this uncertain new world, attitudes towards quantitative expertise have become increasingly divided. From one perspective, grounding politics in statistics is elitist, undemocratic and oblivious to people’s emotional investments in their community and nation.

The Guardian, of course, of the Left, and of the old-guard Left, a group well used to victory, having had them with only rare interruptions for the last century. Until recently. One of the explanations the Left has given to themselves about why they are losing is that the “other side” has abandoned “truth”. Which it has, if you define “truth” as that which accords with progressive ideology.

In Germany, for example (from where we get the term Statistik) the challenge was to map disparate customs, institutions and laws across an empire of hundreds of micro-states. What characterised this knowledge as statistical was its holistic nature: it aimed to produce a picture of the nation as a whole. Statistics would do for populations what cartography did for territory.

This is still, and still should be, the goal of official statistics. Dry facts, which almost are accompanied by their uncertainties. God bless the statisticians provide this wealth! Yet…

The emergence in the late 17th century of government advisers claiming scientific authority, rather than political or military acumen, represents the origins of the “expert” culture now so reviled by populists.

A concern of the author is preserving democracy, which, as I often say, is populism by definition. It’s the losing side that throws the term “populist” out as one of derision. Bad statistics have nothing to do with populism. The reason the Guardian’s enemies dislike traditional experts is because (a) they are far too often wrong, and (b) they confuse measurement with ought. For example, government, activist, and bureaucrats have promised us we would have plunged into another ice age by now, or something like it, with bodies stacked by the side of the road like cord wood. It didn’t happen. Not only that, all the experts’ “solutions” to fix this non-problem were nuts.

And then came global warming…but that is a story for another day. Our author then admits:

Not every aspect of a given population can be captured by statistics. There is always an implicit choice in what is included and what is excluded, and this choice can become a political issue in its own right…In France, it has been illegal to collect census data on ethnicity since 1978, on the basis that such data could be used for racist political purposes.

Somebody ought to suggest the latter move here, at top levels and at campuses and work places. But, nah. Proscribe asking about race and euphemistic statistics would quickly take their place.

The potential of statistics to reveal the state of the nation was seized in post-revolutionary France. The Jacobin state set about imposing a whole new framework of national measurement and national data collection.

The potential of statistics wasn’t the only thing seized by the Jacobins.

During the 1920s, statisticians developed methods for identifying a representative sample of survey respondents, so as to glean the attitudes of the public as a whole. This breakthrough, which was first seized upon by market researchers, soon led to the birth of the opinion polling.

How well did that turn out? Hate to mention it, but doesn’t this smack, just a little, of populism? I’m just asking.

We can agree with this:

Yet in recent decades, the world has changed dramatically, thanks to the cultural politics that emerged in the 1960s and the reshaping of the global economy that began soon after. It is not clear that the statisticians have always kept pace with these changes. Traditional forms of statistical classification and definition are coming under strain from more fluid identities, attitudes and economic pathways. Efforts to represent demographic, social and economic changes in terms of simple, well-recognised indicators are losing legitimacy.

Is this not admitting experts are falling more often into error? I’m just asking.

The article goes on—and on—and on—and on—even coming to the expected criticism of Steven Bannon and Donald Trump. But it’s long on wind and short on solutions.

Which are? The more open source the better, and the more numbers are given with predictive uncertainties, the better, too.

Older posts

© 2017 William M. Briggs

Theme by Anders NorenUp ↑