William M. Briggs, Statistician

The same experts, in Congress and out, who did not foresee and who promulgated the current banking/credit crisis are the same ones assuring us their plan for salvation is just the thing.

Is it rational to believe that these creatures have finally figured out what is best for us? Or is better to say: Stop! Just let things fall out where they may. Let the people who caused this pay the price for their own mistakes.

Analysis so far suggests that the entire mess was brought on by Congressional prompting, in the form of laws which would penalize banks for not making risky loans, and by unscrupulous financiers who figured how to game the system. Also to blame are the people who bought absurdly constructed loans, the kind which they knew they would not be able to eventually afford. It is ridiculous to claim that these people were duped by forces more powerful than themselves. Nobody coerced anybody into buying a house.

And who twisted arms of bank executives to pay their failing CEOs millions? What rewards were given to financial engineers who packaged and sold the most creative sub-prime mortgage-backed securities?

There are many guilty parties here, not the least of which are our own elected representatives who reflexively believe that throwing money—as quickly as possible—at any problem is always the solution.

Thus, the belly-aching speech about partisanship by the appalling Nancy Pelosi after the failed vote was particularly galling. Here are two statistics of interest about yesterday’s “bailout” vote:

40% of House Democrats voted no.

33% of House Republicans voted yes

Which is to say, a fairly uniform rejection by both sides of the aisle.

Thank God for that. I in no way want to give any of my money to either over-confident corporate executives or to the people who will lose their homes. I am utterly unconvinced that I should feel any sense of responsibility for any of this mess.

In this current “bailout”, I feel the same way as when asked to contribute money to people who built their houses on a coastal area well known to be in a location of frequent hurricanes. How is their stupidity my problem?

Whatever the solution is, it is not more government.

Incidentally, it’s been little reported so far, but the Fed has already started printing more money to “infuse cash into the system.” One figure I read is $600 million. Next stop: inflation. Some bailout!

Here is a question I added to my chapter on logic today.

New York City “Health Czar” Thomas Frieden (D), who successfully banned smoking and trans fat in restaurants and who now wants to add salt to the list, said in an issue of Circulation: Cardiovascular Quality and Outcomes that “cardiovascular disease is the leading cause of death in the United States.” Describe why no government or no person, no matter the purity of their hearts, can ever eliminate the leading cause of death.

I’ll answer that in a moment. First, Frieden is engaged in yet another attempt by the government to increase control over your life. Their reasoning goes “You are not smart enough to avoid foods which we claim—without error—are bad for you. Therefore, we shall regulate or ban such foods and save you from making decisions for yourself. There are some choices you should not be allowed to make.”

The New York Sun reports on this in today’s paper (better click on that link fast, because today could be the last day of that paper).

“We’ve done some health education on salt, but the fact is that it’s in food and it’s almost impossible for someone to get it out,” Dr. Frieden said. “Really, this is something that requires an industry-wide response and preferably a national response.”…”Processed and restaurant foods account for 77% of salt consumption, so it is nearly impossible for consumers to greatly reduce their own salt intake,” they wrote. Similarly, regarding sugar, they wrote: “Reversing the increasing intake of sugar is central to limiting calories, but governments have not done enough to address this threat.”

Get that? It’s nearly impossible for “consumers” (they mean people) to regulate their own salt intake. “Consumers” are being duped and controlled by powers greater than themselves, they are being forced to eat more salt than they want. But, lo! There is salvation in building a larger government! If that isn’t a fair interpretation of the authors’ views, then I’ll (again) eat my hat.

The impetus for Frieden’s latest passion is noticing that salt (sodium) is correlated—but not perfectly predictive of, it should be emphasized—with cardiovascular disease, namely high blood pressure (HBP). This correlation makes physical sense, at least. However, because sodium is only correlated with HBP, it means that for some people average salt intake is harmless or even helpful (Samuel Mann, a physician at Cornell, even states this).

What is strange is that, even by Frieden’s own estimate (from the Circulation paper), the rate of hypertension in NYC is four percentage points lower than the rest of the nation! NYC is about 26%, the rest of you are at about 30% If these estimates are accurate, it means New York City residents are doing better than non residents. This would argue that we should mandate non-city companies should emulate the practices of restaurants and food processors that serve the city. It in no way follows that we should burden city businesses with more regulation.

Sanity check:

[E]xecutive vice president of the New York State Restaurant Association, Charles Hunt…said any efforts to limit salt consumption should take place at home, as only about 25% of meals are consumed outside the home.

“I’m concerned in that they have a tendency to try to blame all these health problems on restaurants…This nanny state that has been hinted about, or even partially created, where the government agencies start telling people what they should and shouldn’t eat, when they start telling restaurants they need to take on that role, we think its beyond the purview of government,” Mr. Hunt said.

Amen, Mr Hunt. It just goes to show you why creators and users of statistics have such a bad reputation. Even when the results are dead against you, it is still possible to claim what you want to claim. It’s even worse here, because it isn’t even clear what the results are. By that I mean, the statements made by Frieden and other physicians are much more certain than they should be given the results of his paper. Readers of this blog will not find that unusual.

What follows is a brief but technical description of the Circulation paper (and homework answer). Interested readers can click on. Continue Reading »

Hope in academia?

Thanks again to Dennis Dutton’s Arts & Letters Daily for the link to Graphs on the death of Marxism, postmodernism, and other stupid academic fads.

The author, named “agnostic”, did a text search on articles from the journals indexed at JSTOR (you have to be at a university to use it, or you can pay yourself). He searched for the number of times certain faddish words like Marxism, deconstruction, post colonialism, hegemony, and the like were used in academic prose. He found that they all peaked—some in the late 1990s, others in the early 2000s—and are on the decline.

There are a number of caveats to his analysis, which he acknowledges. The biggest is that the counts are normalized by the number of articles published nor the number of authors publishing. Plus, he didn’t check for the growth of any new fad words,

Still, it is hard not to be hopeful that some academics in the humanities are regaining their minds.

Too many kids in school?

I had never been to that blog before, so I was delighted to find a discussion of Charles Murray’s contention that there are too many kids going to college in the post College is Still the Best Payoff. The analysis presented by the blogger isn’t fully convincing and I don’t think he countered Murray’s suggestion that the best in trades earn more than the average or below average with college degrees.

Murray is well known for arguing that too many kids with “low IQs” are going to college when they should not.

On this topic was a link to the blog The Inductivist, by a gentleman who might also be a statistician. Look for the post from Wednesday, August 27, 2008, wherein he states

Every semester I get a stack of confidential letters describing all sorts of diagnosed learning disorders, along with requests to make accommodations for these students. They need extra time on exams, permission to record lectures, etc.

Educators seem to be more comfortable recognizing limits if they are understood as disorders. We are told that these students are not dumb; they are smart, but just face extra obstacles.

Maybe people don’t like “dumb” because it sounds like forever, and labeling it as a disability enhances our compassion for the person, and it gives hope that eventually we’ll discover a cure. The medicalization of IQ might be the only palatable way to confront the reality.

I left this comment:

I was a visiting professor at Central Michigan U last fall, teaching statistics courses.

I got one of these letters in each of three classes of about 30 students. My impression, after talking with colleagues, was that this was a usual number.

If these letters truly represent learning disabilities among introductory statistics students, then that is an enormous rate.

So what is more likely: (1) The rate of learning disabilities in colleges students really is about 1 in 30, or (2) Learning disabilities are being over diagnosed so that kids don’t drop out of school (and that school loses tuition dollars).

JH, what do you say?

Henry Ford Anniversary

Today was an important anniversary date for Henry Ford Hospital in Detroit. Strangely, they didn’t post any information about it on their site.

New Prime

Again from A&LD, a link to a story about the discovery of a new prime number. Edson Smith at UCLA lead a group to find a new Mersenne prime, which is a prime of the form 2^P-1 where P itself is a prime. Smith’s new prime has P = 43,112,609, and the new prime itself has 13 million digits!

You have to be a real geek to get excited about news like this. But if you own at least one copy of The Book of Prime Number Records by Paulo Ribenboim, then today is a happy day.

Stuff Scientists Like

A new blog that is a take off on the popular Stuff White People Like. Not too many posts yet, and what’s there is telegraphic, but my favorite is At the Movies.

You know those scientific inaccuracies that the directors miss or ignore? Explosions in space and whatnot?

Well, scientists love to gripe about them. Loudly and repeatedly. Both while in the theater and thereafter.

Later they enjoy arguing about whether said inaccuracies fundamentally undermine the quality of the film. Eventually someone gets frustrated and storms off.

Anybody who has ever watched a movie/television with me will know just what he is talking about.

“Why can’t they just pay some guy like me fifty bucks to tell them that that’s impossible! Hell, there are hundreds of geeks out there that would edit scripts for free! This makes no sense! Let me tell you exactly why what there trying can’t be done…”

A reader recently disputed my condensation of the tenets of Bayesian subjective probability. (I promised a thread on which we could discuss the matter more fully, so here it is.) Here is what I said:

To [subjective Bayesians], all probabilities are experiences, feelings that give rise to numbers which are the results of bets you make with yourself or against Mother Nature (nobody makes bets with God anymore). To get the probability of A you poll your inner self, first wondering how you’d feel if A were true, then how you’d feel if A were false. The sort of ratio, or cut point, where you would feel equally good or bad becomes the probability. Subjective Bayesianism, then, was a perfect philosophy of probability for the twentieth century. It spread like mad starting in the late 1970s and still holds sway today; it is even gaining ground on frequentism. In its favor, it should be noted that, after we get past the bare axioms, the math of subjective Bayesianism and logical probability is the same.

The formal name subjectivists have given to guessing probabilities is elicitation, i.e. the process whereby you “poll your inner self.” I invite all to search this term, where you will find many sources. A good summary is given by this paper from the Aviation Human Factors Division Institute of Aviation at the University of Illinois at Urbana-Champaign:

Gamble Methods [of probability elicitation]. Probabilities can also be determined using two gamble-like methods.
In the certain-equivalent method, the expert chooses either a certain payoff or a lottery where the payoff depends on the probability in question, and the elicitor adjusts the amount of the certain payoff until the expert is indifferent between the two choices. In the lottery-equivalent method, the expert chooses either a lottery where the outcome depends on a probability set by the elicitor or a lottery where the outcome depends on the probability in question.

Now, if that doesn’t sound like “To get the probability of A you poll your inner self, first wondering how you’d feel if A were true, then how you’d feel if A were false. The sort of ratio, or cut point, where you would feel equally good or bad becomes the probability” then I’ll eat my hat (the old straw one, the Montecristi, that no longer fits).

There are whole groups of people whose job is to investigate ways of guessing probabilities. One group is, I kid you not, BEEP (Bayesian Elicitation of Experts’ Probabilities) at the University of Sheffield. They have a wide array of, semi-psychological, semi-statistical papers on the pitfalls and joys of probability guessing. An excellent summary is this paper by some pretty big wigs in statistics.

Back of the envelope

As rough estimates, and in many cases, I have absolutely no problem with guessing. I’d even go so far as to say that when any decision has to be made, then unless the situation can be completely deduced, we nearly always fall back on guessing. “Guessing” is usually called “decision making” or “expert opinion.”

But this does not imply, and it is not true, that probability is subjective. There is also the, potentially very large, problem that the elicitation makes you too certain because of the quest for quantification (the point of the original post).

This following is an excerpt from my forthcoming introductory book; this comes after discussions of frequentism and logical probability.

Why probability can’t be subjective

If 3 out of 4 dentists agree that using Dr Johnston’s Whitening Powder makes for shiny teeth, what is the probability that your dentist thinks so? Given only the evidence (premises) that 3 out of 4 etc., then we know the probability is 0.75 that your dentist likes Dr Johnston’s Whitening Powder.

But what if you learned your dentist had just attended an “informational seminar” (with free lunch) sponsored by Galaxy Pharmaceuticals, the manufacturer of Dr Johnston’s Whitening Powder? This introduces new evidence, and will therefore modify the probability that your doctor would recommend Dr Johnston’s.

It may suddenly seem that probability is a matter of belief, of subjective feeling, because different people will have different opinions on how the free lunch will effect the doctor’s endorsement. Probability cannot be a matter of free choice, however. For example, knowing only that a die has 6 sides, and knowing nothing else except that the outcome of the die toss is contingent, then the probability of seeing a 6 is 1 in 6, or about 0.17, regardless of what you or I or anybody thinks. [This is from a discussion of logical probability where the evidence is “A die which will be tossed once has six sides, just one of which is labeled ‘6′” and we want the probability of “We see a 6″, which, given the explicit evidence, is 1/6.]

After you learn of your doc’s cozying up to the pharmaceutical representative, you would be inclined to increase your probability that he would tout Dr Johnston’s to, say, the extent of 0.95. I may come to a different conclusion, say, 0.76 (just slightly higher). Why? Because we are using different sets or collections of information, different evidence or premises, which naturally change our probability assessments. You might know more about pharmaceutical companies than I do, for example, and this causes you to be more cynical, whereas I know more about the purity and selflessness of doctors, and this causes me to be trusting.

But, if I agreed with you about the new evidence, and I felt it was as relevant as you did, then we would share the same probability that the conclusion was true. This, of course, is very unlikely to happen. Rarely will two people agree on a list of premises when the argument involves human affairs, and so it is natural that for most complex things, people will come to different probabilities that the conclusions are true. Does this remind you of politics?

Because people never agree on the set of premises—and they cannot loosely agree on them, they have to agree on them exactly—probabilities will differ. In this sense, probabilities are subjective—rather, it is the choice of premises that is subjective. The probabilities assigned to a conclusion given a set of premises is not. The probability of a conclusion always follows logically from the given premises.

From Jerry Pournelle (What? You haven’t read Lucifer’s Hammer yet?) on how just about everybody making bets in the financial markets were wrong. This “everybody” includes very highly educated, extraordinarily well paid, respected, etc. etc., people.

One of my favorite lines, “Given incorrect models to work with, the computers continued to forecast profits right up to the crash.”

Another “As to what can be done, it may not matter. That is, it’s important what we do, but the chance that it will be done sanely and rationally is very small.” Of course, what we do will be pronounced as “the” thing to do. After all, the eventual plan, whatever it might be, will be made by experts.

Pournelle’s worry, as should be ours, is that the only thing that will happen is the creation of yet another big-government bureaucracy.

“Pournelle’s Iron Law of Bureaucracy states that in any bureaucratic organization there will be two kinds of people: those who work to further the actual goals of the organization, and those who work for the organization itself. Examples in education would be teachers who work and sacrifice to teach children, vs. union representative who work to protect any teacher including the most incompetent. The Iron Law states that in all cases, the second type of person will always gain control of the organization, and will always write the rules under which the organization functions.”

Ah, government bureaucracy. Is there anything experts at the government can’t fix? I know I can’t wait for the EPA to start regulating the “pollutant” CO₂. They ought to figure a way to tie mortgages to global warming. Then things will really get better.

Yes, a disconnected rant today. All I know is that I have been prudent and actually have saved to buy a house, did not try to purchase anything I couldn’t afford, and now I will be asked to pay for the mistakes of all the experts and fools who brought this on.

In any government bailout, the first thing I would require is that any executive of the firms that are being helped would lose all of their personal assets. Every penny. Then I’d sue the traders and stockholders to recover more. I’d do all that before I started taking money from innocent civilians.

As it is, the executives from Fannie Mae, Lehman Brothers, etc., will all walk away very rich men. They will be rewarded.

And the government will continue to bloat.

(This essay will form, when re-written more intelligently, part of Chapter 15, the final Chapter, of my book. Which is coming….soon? The material below is not easy nor brief, folks. But it is very important.)

To most of you, what I’m about to say will not be in the least controversial. But to some others, the idea that not all risk and uncertainty can be quantified is somewhat heretical.

However, the first part of my thesis is easily proved; I’ll prove the second part below.

Let some evidence we have collected—never mind how—be E = “Most people enjoy Butterfingers”. We are interested in answering the truth of this statement: A = “Joe enjoys Butterfingers.” We do not know whether A is true or false, and so we will quantify our uncertainty in A using probability, that is written like this

#1    Pr( A | E )

and which reads “The probability that A is true given the evidence E”. (The vertical bar “|” means “given.”)

In English, the word most at least means more than half; it could even mean a lot more than a half, or even nearly all—there is certainly ambiguity in its definition. But since most at least means more than half, we can partially answer our question, which is written like this

#2    0.5 < Pr( A | E ) < 1

and which reads "The probability that A is true is greater than a half but not certain given the evidence E.” This answer is the best we can do with the given evidence.

This answer is a quantification of sorts, but it is not a direct quantification like, say, the answer “The probability that A is true is 0.673.”

It is because there is ambiguity in the evidence that we cannot completely quantify the uncertainty in A. That is, the inability to articulate the precise definition of “most people” is the reason we cannot exactly quantify the probability of A.

The first person to recognize this, to my knowledge, was John Maynard Keynes is his gorgeous, but now little read, A Treatise on Probability, a book which argued that all probability statements were statements of logic To Keynes—and to us—all probability is conditional; you cannot have a probability of A, but you can have a probability of A with respect to certain evidence. Change the evidence and change the probability of A. Stating a probability of A unconditional on any evidence disconnects that statement from reality, so to speak.

Other Theories of Probability

For many reasons, Keynes’s eminently sensible idea never caught on and instead, around the same time his book was published, probability theory bifurcated into two antithetical paths. The first was called frequentism: probability was defined to be that number which is the ratio of experiments in which A will be true divided by the total numbers of experiments as that number of experiments goes to infinity¹. This definition makes it difficult (an academic word meaning impossible) to answer what is the probability that Joe, our Joe, likes Butterfingers. It also makes it difficult to define the probability for any event or events that are constrained to occur less than an infinite number of times (so far, this is all events that I know of).

The second branch was subjective Bayesianism. To this group, all probabilities are experiences, feelings that give rise to numbers which are the results of bets you make with yourself or against Mother Nature (nobody makes bets with God anymore). To get the probability of A you poll your inner self, first wondering how you’d feel if A were true, then how you’d feel if A were false. The sort of ratio, or cut point, where you would feel equally good or bad becomes the probability. Subjective Bayesianism, then, was a perfect philosophy of probability for the twentieth century. It spread like mad starting in the late 1970s and still holds sway today; it is even gaining ground on frequentism.

What both of these views have in common is the belief that any statement can be given a precise, quantifiable probability. Frequentism does so by assuming that there always exists a class of events—which is to say, hard data—to which you can compare the A before you. Subjective Bayesianism, as we have seen, can always pull probabilities for any A out of thin air. In every conceivable field, journal articles using these techniques multiply. It doesn’t help that the many times probability estimates are offered in learned publications, they are written in dense mathematical script. Anything that looks so complicated must be right!

Mathematics

The problem is not that the mathematical theories are wrong; they almost never are. But because the math is right does not imply that it is applicable to any real-world problems.

The math often is applicable, of course; usually for simple problems and in small cases the results of which would not be in much dispute even without the use of probability and statistics. Take, for example, a medical trial with two drugs, D and P, given to equal numbers of patients for an explicitly definable disease that is either absent or present. As long as no cheating took place and the two groups of patients balanced, then if more patients got better using drug D, that drug is probably better. In fact, just knowing that drug D performed better (and no cheating and balance) is evidence enough for a rational person to prefer D over P.

All that probability can do for you in cases like this is to clean up the estimates of how much better D might be than P in new groups of patients. As long as no cheating took place and the patients were balanced, the textbook methods will give you reasonable answers. But suppose the disease the drugs treat is not as simply defined. Let’s write what we just said in mathematical notation so that certain elements become obvious.

#3    Pr ( D > P | Trial Results & No Cheating & Patients Like Before) > 0.5.

This reads, the probability that somebody gets better using drug D rather than P given the raw numbers we had from the old trial (including the old patient characteristics) and that no cheating took place in that trial and the new patients who will use the drugs “look like” the patients from the previous trial, is greater than 50% (and less than certain).

Now you can see why I repeatedly emphasized that part of the evidence that usually gets no emphasis: no cheating and patients “like” before. Incidentally, it might appear that I am discussing only medical trials and have lost sight of the original thread. I have not, which will become obvious in a moment.

Suppose the outcome of applying a sophisticated probability algorithm gave us the estimate of 0.72 for equation #3. Does writing this number more precisely help if you suppose you are the doctor who has to prescribe either D or P? Assume that no cheating took place in the old trial, then drug D is better if the patient in front of you is “like” the patients from the old trial. What is the probability she is so (given the information from the old trial)?

The word like is positively loaded with ambiguity. Not to be redundant, but write out the last question mathematically.

#4    Pr ( My patient like the others | Patients characteristics from previous trial)

The reason to be verbose in writing out the probability conditions is that it puts the matter starkly. It forces you, unlike the old ways of frequentisim and subjective Bayesianism, to specify as completely as possible the circumstances that form your estimate. Since all probability is conditional, it should always be written as such so that it is always seen as such. This is necessary because it is not just the probability from equation #3 that is important, equation #4 is, too. If you are the doctor, you do not—you should not—focus solely on probability #3 because what you really want is this:

#5    Pr ( D > P & My patient like before | Trial Results & No Cheating & Patients Character)

which is just #3 x #4. I am in no way arguing that we should abandon formal statistics which produces quantifications like equation #3. But I am saying that since, as we already know, exactly quantifying #4 is nearly impossible, we will be too confident of any decisions we make if we, as is common, substitute probability #3 for #5 because, not matter what, the probability of #3 and #4 both is always less than the probability of #3.

Appropriate caveats and exceptions are usually delineated in journal articles when using the old methods, but the results are buried in the text, which causes them to be weighed more or less importantly, and which give the reader a false sense of security. Because, in the end, we are left with the suitably highlighted number from equation #3, that comforting exact quantification reached by implementing impressive mathematical methods. That final number, which we can now see is not final at all, is tangible, and is held on to doggedly. All the evidence to the right of the bar is forgotten or downplayed because it is difficult to keep in mind.

The result to equation #3 is produced, too, only from the “hard data” of the trial, the actual physical measurements from the patients. These numbers have the happy property that they can be put into spreadsheets and databases. They are real. So real that their importance is magnified far beyond their capacity to provide all the answers. They fool people into thinking that equation #3 is the final answer, which it never is. It is always equation #5 that is important to making new decisions. Sometimes, in simple physical cases, probabilities #3 and #5 are so close as to be practically equal; but when the situation is complex, as it always is when involving humans, these two probabilities are not close.

Everything That Can Happen

The situation is actually even worse than what we have discussed so far. Probability models, the kind that spit out equation #3, are fit to the “hard data” at hand. The models that are chosen are usually picked because of habit and familiarity, but responsible practitioners also choose the models so that they fit the old data well. This is certainly a rational thing to do. The problem is that, since probability models are only designed to say something about future data, the old data does not always encompass everything that can happen and so we are limited in what we can say about the future. All we can say for certain is what has happened before might happen again. But it’s anybody’s guess whether what hasn’t happened before might happen in the future.

The probability models fit the old data well, but nobody can ever know how well they will fit future data. The result is that over reliance on “hard data” means that probabilities of extreme events are underestimated and mundane events overestimated. The simple way to state this is the system is built to engender overconfidence.²

Decision Analysis

You’re still the doctor and you still have to prescribe D or P (or nothing). No matter what you prescribe something will happen to the patient. What? And when? Perhaps the malady clears up, but how soon? Perhaps the illness is merely mitigated, but by how much? You not only have to figure out what treatment is better, but what will happen if you apply that treatment. This is a very tricky business, and is why, incidentally, there is such a variance in the ability of doctors.³ Part of the problem is explicitly defining what is meant be “the patient improves.” There is ambiguity in that word improve, in what will happen with either of the drugs is administered.

There are two separate questions here: (1) defining events and estimating their probability of occurring and (2) estimating what will happen given those events occur. Going through both of the steps is called computing a risk or decision analysis. This is an enormously broad subject which we won’t do more than touch on, only to show where more uncertainty comes in.

We have already seen that there is ambiguity in computing the probability of events. The more complex these events the more imprecise the estimate. It is also often the case that part (2) of the risk analysis is the most difficult. The events themselves cannot be articulated, either completely or unambiguously. In simple physical systems they often can be, of course, but in complex ones like the climate or ecosystems they are not. Anything involving humans is automatically complex.

Take the current (!) financial crisis as an example. Many of the banks and brokerages failed to both define the events that are now happening, and they extent of the cost of those events. How much will it cost to clean it up? Nobody knows. This is the proper answer. We might be able to bound it—more than half a billion, say—and that might be the best anybody can say (except that I have been asked to pay for it).

Too Much Certainty

What the older statistical methods and the strict reliance on hard data and fancy mathematics have done is to create a system where there is too much certainty when making conclusions about complex events. We should all, always, take any result and realize that they are conditional on everything being just so. We should realize those just so conditions that obtained in the past might not in the future.

Well, you get the idea. There is already far too much information to assimilate in one reading (I’m probably just as tired of going on and on as you are of reading all this!). As always, discussion is welcome.

—————————
¹Another, common, way to say infinity is the euphemism “in te long run”. Keynes has famously said that “In the long run we shall all be dead.” It’s always been surprising to me that the same people who giggle at this quip ignore its force.

²There is obviously a lot more to say on this subject, but we’ll leave it for another time.

³A whole new field of medicine has emerged to deal with this topic. It is called evidence based medicine. Sounds good, no? What could be wrong with evidence? And it’s not entirely a bad idea, but there is an over reliance on the “hard data” and a belief that only this hard data can answer questions. We have already seen that this cannot be the case.

I’ve been taking the past few days and building an Index for the my “101″ book. It is painstaking, meticulous…well, excruciatingly dull work. But it’s nearly done.

This is slowing me down from posting new entries here.

One that I especially want to get to was suggested by reader Mike D and a May issue of the Economist. Can all risks be quantified? Can all probabilities be quantified?

The answer, perhaps surprisingly, is no. It’s surprising if you hang out in statistics and economics departments. Places with far too much math and far too little philosophy.

Anyway, book should be done “soon.” End of the month?

Stay tuned so you can be the first one on your block to own a copy.

In today’s New York Post (p. 31) runs a full page add by CNN. The ad itself looks like a PowerPoint presentation, that is, a dull layout driven by bullet points. Here are the first three:

1. #1 Most Watched Cable News Network Across Both DNC and RNC Conventions for P25-54, P18-49, & P18-34
2. #1 Most Watched Cable News Network at 10PM Across Both Conventions
3. #1 News and Information Site During Both Conventions-CNN.com

In the first bullet, what is that odd “P25-54″? What in the world could that be? The ad, even in the fine print, nowhere says. But we can guess is means “people aged 25 to 54″. OK, so people aged 25 to 54—a good slice of people—according to “sources”, liked CNN. Sounds impressive, but there are two misleading elements.

The first is that it was the most watched “cable news network”, which means that the “non-cable news networks” might have been watched by more people. We can guess that this is true else the ad would have touted that CNN was the most watched network period.

The second problem is the bizarre way they sliced the age groups. The 18-49 groups entirely contains the 18-34 group, does it not? So why mention the smaller-sized group? Does it mean that the 35-49 years olds did not prefer CNN? But the 35-39 year olds are certainly among the 25-54 years olds, which was the first group mentioned.

There is no making sense of any of this except by supposing that CNN scrounged through the data to find any hint of subgroups that supported their “#1″ contention. Experience shows that you can do this for any statistical analysis, which is why so many rightly suspect whatever statisticians have to say.

The second bullet point is just as screwy. How many different networks did they compare anyway? The barely readable small print says “CNN, FNC, and MSNBC.” So they only had two competitors, the last of which, MSNBC, has always struggled for viewership. Coming in number 1 in a few categories with only one real competitors is not a laudable achievement.

But it also means that CNN must have lost, probably to Fox, in the 7p-8p slot, the 8p-9p slot, the 9p-10p slot, and the 11p-12a slot, where are 4 out of the 5 slots the small print says the “sources” checked. Losing 80% of the time hardly makes you number 1.

The third bullet is more tepid. The “source” says “Information Site” means “Current events and Global News Category.” We have no idea how many other sites were compared against CNN, nor how many other categories—say Analysis, Opinion, Politics, and so on—were checked.

Still, for cheating, the third bullet is best, because it’s a rare person will be pause much over the claim, nor will most browse the small print.

In any case, CNN should just have presented their fourth bullet, which was

• #1 Most Trusted & Credible Name in News

This bullet is so vague it can mean anything. It’s crafted so that readers can take any meaning they like from it. Most people will be left with a dull sense of the importance of CNN.

This ad, while fairly misleading, only earns a 4 (out of 10) on the Briggs Statistical Deception Scale.

For the teachers out there, these ads often make good homework problems for students. Chopping up an ad into component parts and reading it critically is always great fun for the students. Especially when you find deceitful ads.

The author of Fooled by Randomness and The Black Swan, Nassim Nicholas Taleb, has penned the essay THE FOURTH QUADRANT: A MAP OF THE LIMITS OF STATISTICS over at Edge.org (which I discovered via the indispensable Arts & Letters Daily).

Taleb’s central thesis and mine are nearly the same: “Statistics can fool you.” Or “People underestimate the probability of extreme events”, which is another way of saying that people are too sure of themselves. He blames the current crisis on Wall Street on people misusing and misunderstanding probability and statistics:

This masquerade does not seem to come from statisticians—but from the commoditized, “me-too” users of the products. Professional statisticians can be remarkably introspective and self-critical. Recently, the American Statistical Association had a special panel session on the “black swan” concept at the annual Joint Statistical Meeting in Denver last August. They insistently made a distinction between the “statisticians” (those who deal with the subject itself and design the tools and methods) and those in other fields who pick up statistical tools from textbooks without really understanding them. For them it is a problem with statistical education and half-baked expertise. Alas, this category of blind users includes regulators and risk managers, whom I accuse of creating more risk than they reduce.

I wouldn’t go so far as Taleb: the masquerade also often comes from classical statistics and statisticians, too. Much of the statistical methods that are taught to non-statisticians had their origin in the early and middle part of the 20th century before there was access to computers. In those days, it was rational to make gross approximations, assume uncertainty could always be quantified by normal distributions, guess that everything was linear. These simplifications allowed people to solve problems by hand. And, really, there was no other way to get an answer without them.

But everything is now different. The math is new, our understanding of what probability is has evolved, and everybody knows what computers can do. So, naturally, what we teach has changed to keep pace, right?

Not even close to right. Except for the modest introduction of computers to read in canned data sets, classes haven’t change one bit. The old gross approximations still hold absolute sway. The programs on those computers are nothing more than implementations of the old routines that people did by hand—many professors still require their students to compute statistics by hand! Just to make sure the results match what the computer spits out.

It’s rare to find an ex-student of a statistics course who didn’t hate it (”You’re a statican [sic]? I always hated statistics!” they say brightly). But it’s just as rare to find a person who had, in the distant past, one of two courses who doesn’t fancy himself an expert (I can’t even list the number of medical journal editors who have told me my new methods were wrong). People get the idea that if they can figure out how to run the software, then they know all they need to.

Taleb makes the point that these users of packages necessarily take a too limited view of uncertainty. They seek out data that confirms their beliefs (this obviously is not confined to probability problems), fit standard distributions to them, and make pronouncements that dramatically underestimate the probability of rare events.

Many times rare events cause little trouble (the probability that you walk on a particular blade of grass is very low, but when that happens, nothing happens), but sometimes they wreak havoc of the kind happening now with Lehman Brothers, AIG, WAMU, and on and on. Here, Taleb starts to mix up estimating probabilities (the “inverse problem”) with risk in his “Four Quadrants” metaphor. The two areas are separate: estimating the probability of an event is independent of what will happen if that event obtains. There are ways to marry the two areas in what is called Decision Analysis.

That is a minor criticism, though. I appreciate Taleb’s empirical attempt at creating a list of easy to, hard to, and difficult to estimate events along with their monetary consequences should the events happen (I have been trying to build such a list myself). Easy to estimate/small consequence events (to Taleb) are simple bets, medical decisions, and so on. Hard to estimate/medium consequence events are climatological upsets, insurance, and economics. Difficult to estimate/extreme consequence events are societal upsets due to pandemics, leveraged portfolios, and other complex financial instruments. Taleb’s bias towards market events is obvious (he used to be a trader).

A difficulty with Taleb is that he writes poorly. His ideas are jumbled together, and it often appears that he was in such a hurry to gets the words on the page that he left half of them in his head. This is true for his books, too. His ideas are worth reading, however, though you have to put in some effort to understand him.

I don’t agree with some of his notions. He is overly swayed by “fractal power laws”. My experience is that people often see power laws where they are not. Power laws, and other fractal math, give appealing, pretty pictures that are too psychologically persuasive. That is a minor quibble. My major problem is philosophical.

Taleb often states that “black swans”, i.e. extremely rare events of great consequence, are impossible to predict. Then he faults people, like Ben Bernanke, for failing to predict them. Well, you can’t predict what is impossible to predict, no? Taleb must understand this, because he often comes back to the theme that people underestimate uncertainty of complex events. Knowing this, people should “expect the unexpected”, a phrase which is not meant glibly, but is a warning to “increase the area in the tails” of the probability distributions that are used to quantify uncertainty in events.

He claims to have invented ways of doing this using his fractal magic. Well, maybe he has. At the least, he’ll surely get rich by charging good money to learn how his system works.

Here is a question I often give on exams:

What is the probability that the next child to be born will be a genius? Give me a number and fully explain your answer.

There is not, of course, a single correct answer. What I just said is an important point, so let’s not skip lightly over it: there is no correct answer; at least, there is no way anybody can know the correct answer.

That nobody can know with certainty answers to questions of this type is under appreciated. I want people to learn this because we are, as I often say, too sure of ourselves.

What I want to see in the answer is acknowledgment of the ambiguities. First, what is a genius? Surely that word is overused to a remarkable extent. For example, this list says, with a straight face, authoress JK Rowling and movie maker Stephen Spielberg are geniuses. I often have the idea that to not call some eminence a genius is nowadays taken as a slight. However, a moments’ thought suffices to show that people exaggerate—if you are willing to take that moment.

The next step is to think of some geniuses for the sake of comparison. It’s best to think of dead ones so that you are not overly influenced by current events. After all, only history can truly judge genius. If you agree with even part of this, you will have made the next most important step: admitting that you can be biased.

How about some dead geniuses? Einstein pops into nearly everybody’s head first. Then, for me, Mozart, Beethoven, Shakespeare, Newton, and the guy who invented beer. No, I’m not joking about that last name. The point is my historical knowledge is modest, and most of the names I pick are men from the last 500 years, and most are from Western culture. Humanity is older than 500, of course, and there are other cultures besides our own, so I know that my knowledge of who is a genius is limited. That’s what got me to thinking about the brilliant soul who invented beer. He did so, probably in Sumer, before people wrote down incredible deeds of this sort.

This line of thought eventually leads to other cultures (Confucius, maybe Lao Tzu) and other times where writing was non-existent (was there just one person responsible for the wheel and agriculture?). There must be a lot of geniuses I don’t know, and some that nobody can ever know.

Next step is to count, and to acknowledge that exact counting is an impossibility. Still, we can count to the nearest order of magnitude. This means “power of 10″, and it represents an enormously popular method of approximation. If you can get your answer to within “an order of magnitude” (a power of 10), you are doing good. The first power of 10, or 10¹, is just 10. The second power is 10²=100, and so on.

So how many geniuses? Certainly more than 10, definitely less than 10,000, or the 5th order of magnitude. Could there have been a 100 geniuses? Given my above list, I say yes. 1000? I’m less likely to believe this number, but since I have said that there were lots of geniuses who went unsung, I can’t exclude it. Still, an order of magnitude more than this seems too large.

We have done a lot so far, but we still haven’t answered the question “What is the probability that the next child to be born will be a genius?” The answer will look something like # of geniuses who have ever lived / # of people who ever lived. Coming to this equation is crucial. This is because the question implies—I emphasize, it does not explicitly state—we are asking a question about all humanity. And all humanity certainly means all humans who have ever lived.

Thus far, we have nailed down the numerator in this equation to the nearest order of magnitude or so (10² to 10³). How about the denominator?

What evidence do we have? Well, there about about, to an order of magnitude, 10¹⁰ or 10 billion people alive today. 100 years from now, nearly of these people will be dead and a new set, probably the same order of magnitude will take its place. Anyway, 100 billion people alive 100 years from now feels way too large to me, and 1 billion way too small, especially given recent population trends.

100 years ago, there were about an order of magnitude less people alive (nearly all of them different from the set we have today), or about 10⁹ or 1 billion. How many 100s of years can we go back? About 2000, since the best guess is humanity arose about 200,000 years ago. That’s close enough; it’s within an order of magnitude. Without doing any math—just going by gut—we can guess that adding today’s 10¹⁰ to last century’s 10⁹ (11 billion so far), and to the previous 199 centurys’ diminishing contributions (each previous century had fewer people), we arrive at about 10¹¹, or 100 billion.

Was that larger than you had first guessed? This number usually surprises most people. But having a guess gives us our denominator as that we can finally solve our equation, which is

10²
—— = 10^-9
10¹¹

of, if there were 10³ geniuses, 10^-8. In words, it’s anywhere from 1 in a billion to 1 in 100 million.

Not very good odds, right?

This was a lot of thinking for such a simple question, wasn’t it? If you would have written down, as student’s often do, an answer “1 in a 100″ or “1 in 1000″ you would have got the answer wrong. Both answers imply that we should be flooded with geniuses, an answer which no observation supports.

Of oft-heard complaint among professors is that students don’t think about the answers they give. I agree with this, but I think it’s more than just students. It holds for professors and ordinary civilians, too.

“1 in a 100″ is absurd, and far too certain. Just a few moment’s thought shows this. How many answers that we give in life are just as absurd?

Some kids will write, “I don’t know.” I usually give them 1 point for this because, after all, it is the strictly correct answer. But that answer is too certain itself. We do know something about the answer and we can answer it partially. We should always quantify uncertainty in any question and not seek the easy way out by given answers that are too certain.

Here, for fun, is another question I give:

How many umbrellas are there in New York City?