Skip to content

Category: Book review

Please email me at matt@wmbriggs.com before sending books to be reviewed.

September 16, 2008 | 20 Comments

The limits of statistics: black swans and randomness

The author of Fooled by Randomness and The Black Swan, Nassim Nicholas Taleb, has penned the essay THE FOURTH QUADRANT: A MAP OF THE LIMITS OF STATISTICS over at Edge.org (which I discovered via the indispensable Arts & Letters Daily).

Taleb’s central thesis and mine are nearly the same: “Statistics can fool you.” Or “People underestimate the probability of extreme events”, which is another way of saying that people are too sure of themselves. He blames the current crisis on Wall Street on people misusing and misunderstanding probability and statistics:

This masquerade does not seem to come from statisticians—but from the commoditized, “me-too” users of the products. Professional statisticians can be remarkably introspective and self-critical. Recently, the American Statistical Association had a special panel session on the “black swan” concept at the annual Joint Statistical Meeting in Denver last August. They insistently made a distinction between the “statisticians” (those who deal with the subject itself and design the tools and methods) and those in other fields who pick up statistical tools from textbooks without really understanding them. For them it is a problem with statistical education and half-baked expertise. Alas, this category of blind users includes regulators and risk managers, whom I accuse of creating more risk than they reduce.

I wouldn’t go so far as Taleb: the masquerade also often comes from classical statistics and statisticians, too. Much of the statistical methods that are taught to non-statisticians had their origin in the early and middle part of the 20th century before there was access to computers. In those days, it was rational to make gross approximations, assume uncertainty could always be quantified by normal distributions, guess that everything was linear. These simplifications allowed people to solve problems by hand. And, really, there was no other way to get an answer without them.

But everything is now different. The math is new, our understanding of what probability is has evolved, and everybody knows what computers can do. So, naturally, what we teach has changed to keep pace, right?

Not even close to right. Except for the modest introduction of computers to read in canned data sets, classes haven’t change one bit. The old gross approximations still hold absolute sway. The programs on those computers are nothing more than implementations of the old routines that people did by hand—many professors still require their students to compute statistics by hand! Just to make sure the results match what the computer spits out.

It’s rare to find an ex-student of a statistics course who didn’t hate it (“You’re a statican [sic]? I always hated statistics!” they say brightly). But it’s just as rare to find a person who had, in the distant past, one of two courses who doesn’t fancy himself an expert (I can’t even list the number of medical journal editors who have told me my new methods were wrong). People get the idea that if they can figure out how to run the software, then they know all they need to.

Taleb makes the point that these users of packages necessarily take a too limited view of uncertainty. They seek out data that confirms their beliefs (this obviously is not confined to probability problems), fit standard distributions to them, and make pronouncements that dramatically underestimate the probability of rare events.

Many times rare events cause little trouble (the probability that you walk on a particular blade of grass is very low, but when that happens, nothing happens), but sometimes they wreak havoc of the kind happening now with Lehman Brothers, AIG, WAMU, and on and on. Here, Taleb starts to mix up estimating probabilities (the “inverse problem”) with risk in his “Four Quadrants” metaphor. The two areas are separate: estimating the probability of an event is independent of what will happen if that event obtains. There are ways to marry the two areas in what is called Decision Analysis.

That is a minor criticism, though. I appreciate Taleb’s empirical attempt at creating a list of easy to, hard to, and difficult to estimate events along with their monetary consequences should the events happen (I have been trying to build such a list myself). Easy to estimate/small consequence events (to Taleb) are simple bets, medical decisions, and so on. Hard to estimate/medium consequence events are climatological upsets, insurance, and economics. Difficult to estimate/extreme consequence events are societal upsets due to pandemics, leveraged portfolios, and other complex financial instruments. Taleb’s bias towards market events is obvious (he used to be a trader).

A difficulty with Taleb is that he writes poorly. His ideas are jumbled together, and it often appears that he was in such a hurry to gets the words on the page that he left half of them in his head. This is true for his books, too. His ideas are worth reading, however, though you have to put in some effort to understand him.

I don’t agree with some of his notions. He is overly swayed by “fractal power laws”. My experience is that people often see power laws where they are not. Power laws, and other fractal math, give appealing, pretty pictures that are too psychologically persuasive. That is a minor quibble. My major problem is philosophical.

Taleb often states that “black swans”, i.e. extremely rare events of great consequence, are impossible to predict. Then he faults people, like Ben Bernanke, for failing to predict them. Well, you can’t predict what is impossible to predict, no? Taleb must understand this, because he often comes back to the theme that people underestimate uncertainty of complex events. Knowing this, people should “expect the unexpected”, a phrase which is not meant glibly, but is a warning to “increase the area in the tails” of the probability distributions that are used to quantify uncertainty in events.

He claims to have invented ways of doing this using his fractal magic. Well, maybe he has. At the least, he’ll surely get rich by charging good money to learn how his system works.

July 12, 2008 | 1 Comment

Transforming American Military Policy

Finding the Target: The Transformation of American Military Policy by Frederick W. Kagan, 2006. Encounter Books. Recommendation: read

This book is an excellent accounting of the theories that have gripped and influenced American military thinking and planning since the Vietnam war.

Theories?

If all your information on the military has come from Hollywood (and there is a new series about Iraq out on HBO, surely this time written by writers who actually served and are thus knowledgeable), then it might come as a surprise that when planning a war you actually have to decide what to hit, what resources are needed, when those resources should be in place, what will happen in theater and out of it, the political consequences, and on and on. These decisions are made with reference to a guiding doctrine, a.k.a. a theory.

Since Clausewitz, a leading theory has been to attack an enemy’s “centers of gravity”. Destroy those, the theory goes, and the enemy collapses in confusion. Maybe so. But what is a center of gravity? Does this mean you try to kill as many troops in the field as possible? Or instead commit your resources to disrupting enemy supply lines, or perhaps the lines of communication and control? Or do you, as happened at the very beginning of Iraqi Freedom, attempt to take out the leadership (an effort, you will recall, which failed)? All good questions, the answers to which should depend on the situation. The danger is that people can pay more attention to the guiding theory—to what the theory says reality should be like—than to actual reality itself. This common human failing is found in war just as it is in other areas.

There is also the danger of rushing in, say after an unexpected attack of your country, and not having any plan:

[T]hey find it difficult—albeit no less important—to identify clear, achievable strategic aims. There is an emotional temptation to want to ‘do something’ without first clearly understanding what political purpose that ‘something’ is supposed to accomplish.

Kagan repeatedly emphasizes that military actions are subservient to, or an extension of, a country’s political aims. Just killing the enemy is not enough. The way that enemy is killed or defeated must be done in such a way to further the political aims. The lack of these thoughts harmed the Iraqi war. As is well known by now, the hostilities themselves were over very quickly. The war plan was to “topple the regime” as fast as possible. This was “mission accomplished.” But in toppling the regime, nothing took its place, and chaos prevailed. The problem was the enemy was not captured, they was instead allowed to disperse, taking their weapons with them, the result of which was the insurgence.

The situation in Iraq was not turned around until more boots were on the ground, handling things in the old fashioned way, opposite to dictates of the “revolution in military affairs” and “transformative” theories then touted by the leadership.

Kagan also takes to task the latest theories that holds some in thrall: Network Centric Warfare, or NCW. This is the idea that the miracles of the “Information age” will “revolutionize” and “transform” forever our view of the “battlespace” (the old term “battlefield” deemed musty). Generals, using these things called computers, will soon be able to see what every platoon-leading lieutenant sees, and so will be able to direct the battlespace more effectively. Information overload? Don’t bother me with details. Kagan sums up his objections to NCW:

First, it is a solution in search of a problem. Second, the technical requirements needed to produce the capabilities sought and promised are unattainable in the real world. Third, it proceeds from a misunderstanding of the nature of war…The NCW visionaries imagine a world in which the eternal race between offense and defense ands in our favor—we will be able to see everything and the enemy will be about to nothing about it. This notion is preposterous.

Instead, Kagan advocates the obvious strategy: plan for the situations you are most likely to face. You might still be wrong, but you, by definition, have the best chance of being right. Do not ask for “revolutionary” technologies, but build better weapons from known technologies.

Other topics are discussed. For example: “The Army still maintain garrisons as though it were preparing to subdue the Sioux and Apache once again.” These historical dispositions “impose significant delays” on deployment and offer the enemy “numerous bottlenecks to strike.” But to try and change base and post locations is a mighty political task. Try suggesting to your congressperson—Democrat or Republican—that the base in their state is aptly located and see what happens. Politicians, as ever, will usually opt for what is best for themselves and not the country.

The book is an intelligent, readable overview of military policy planning and I highly recommend it.

January 1, 2008 | 1 Comment

Calculated Risks: How to know when numbers deceive you: Gerd Gigerenzer

Gerd Gigerenzer, Simon and Schuster, New York, 310 pp., ISBN 0-7432-0556-1, $25.00

Should healthy women get regular mammograms to screen for breast cancer?

The surprising answer, according to this wonderful new book by psychology professor Gerd Gigerenzer, is, at least for most women, probably not.

Deciding whether to have a mammogram or other medical screening (the book examines several) requires people to calculate the risk that is inherent is taking these tests.? This risk is usually poorly known or communicated and, because of this, people can make the wrong decisions and suffer unnecessarily.

What risk, you might ask, is there for an asymptommatic woman in having a mammogram? To answer that, look at what could happen.

The mammogram could correctly indicate no cancer, in which case the woman goes away happy.? It could also correctly indicate true cancer, in which case the woman goes away sad and must consider treatment.

Are these all the possibilities?? Not quite.? The test could also indicate that no cancer is present when it is really there—the test could miss the cancer.? This gives false hope and causes a delay in treatment.

But also scary and far more likely is that the test could indicate that cancer is present when it is not.? This outcome is called a false positive, and it is Gigerenzer’s contention that the presence of these false positives are ignored or minimized by both the medical profession and by interest groups whose existence is predicated on advocating frequent mammograms (or other disease screenings, such as for prostate cancer or AIDS).

Doctors like to provide an “illusion of certainty” when, in fact, there is always uncertainty in any test.? Doctors and test advocates seem to be unaware of this uncertainty, they have different goals than do the patients who will receive the tests, and they ignore the costs of false positives.

How is the uncertainty of a test calculated?? Here is the standard example, given in every introductory statistics book, that does the job. This example, using numbers from Gigerenzer, might look confusing, but read through it because its complexity is central to understanding the his thesis.

If the base rate probability of breast cancer is 0.8% (the rate of cancer in women in the entire country), and the sensitivity (ability to diagnose the cancer when it is truly there) and specificity (ability to diagnose no cancer when it is truly not there) of the examination for cancer is 90% and 93%, then given that someone tests positive for cancer, what is the true probability that this person actually has cancer?

To answer the question requires a tool called Bayes Rule.? Gigerenzer has shown here, and in other research, that this tool is unnatural and difficult to use and that people consistently poorly estimate the answer. Can you guess what the answer is?

Most people incorrectly guess 90% or higher, but the correct answer is only 9%, that is, only 1 woman out of every 11 who tests positive for breast cancer actually has the disease, while the remaining 10 do not.

If people instead get the same question with the background information in the form of frequencies instead of probabilities they do much better.? The same example with frequencies is this: If out of every 1000 women 77 have breast cancer, and that 7 of these 77 who test positive actually have the disease, then given that someone tests positive for cancer what is the true probability that this person actually has cancer?

The answer now jumps out—7 out of 77—and is even obvious, which is Gigerenzer’s point.? Providing diagnostic information in the form of frequencies benefits both patient and doctor because both will have a better understanding of the true risk.

What are the costs of false positives?? For breast cancer, there are several.? Emotional turmoil is the most obvious: testing positive for a dread disease can be debilitating and the increased stress can influence the health of the patient negatively.? There is also the pain of undergoing unnecessary treatment, such as mastectomies and lumpectomies.? Obviously, there is also a monetary cost.

Mammograms can show a noninvasive cancer called ductal carcinoma in situ, which is predominately nonfatal and needs no treatment, but is initially seen as a guess of cancer. There is also evidence that the radiation from the mammogram increases the risk of true breast cancer!

These costs are typically ignored and doctors and advocates usually do not acknowledge the fact the false positives are possible.? Doctors suggest many tests to be on the safe side—but what is the safe side for them is not necessarily the safe side for you. Better for the doctor to have asked for a test and found nothing than to have not asked for the test and miss a tumor, thus risking malpractice.

This asymmetry shows that the goals of patients and doctors are not the same.? The same is true for advocacy groups.? Gigerenzer studies brochures from these (breast cancer awareness) groups in Germany and the U.S. and found that most do not mention the possibility of a false positive, nor the costs associated with one.

Ignoring the negative costs of testing makes it easier to frighten women into having mammograms, and he stresses that, “exaggerated fears of breast cancer may serve certain interest groups, but not the interests of women.”

Mammograms are only one topic explored in this book.? Others include prostate screenings “where there is no evidence that screening reduces mortality”, AIDS counseling, wife battering, and DNA fingerprinting.

Studies of AIDS advocacy group’s brochures revealed the same as in the breast cancer case: the possibility of false positives for screenings and the costs associated with these mistakes were ignored or minimized.

Gigerenzer even shows how attorney Alan Dershowitz made fundamental mistakes calculating the probable guilt of O.J. Simpson, mistakes that would have been obvious had Dershowitz used frequencies instead of probabilities.

The book closes with tongue-in-cheek examples of how to cheat people by exploiting their probabilistic innumeracy, and includes several fun problems.

Gigerenzer stresses that students have a high motivation to learn statistics but that it is typically poorly taught.? He shows that people’s difficulties with numbers can be overcome and that it is in our best interest to become numerate.

December 17, 2007 | 1 Comment

“The Future of Everything” by David Orrell

The Future of Everything by David Orrell. Thunder’s Mouth Press, New York.

I wanted to like this book, which was supposed to be an examination of how well scientists made predictions—my special area of interest—but I couldn’t. It wasn’t just Orrell’s occasional use of juvenile and gratuitous political witticisms: for example, at one point in his historical review of ancient Greek-prediction making, Orrell sarcastically assures us that the “White House” would not, as dumb as its occupants are, stoop so low as to rely on the advice gained from examining animal entrails. It also wasn’t that the book lacked detailed explanations of the three fields he criticizes—weather and climate forecasts, economic forecasts, and health predictions. Nor was it that Orrell was sloppy in some of his historical research: for example, he repeats the standard, but false, view that Malthus predicted mankind would overpopulate the world (more on this below).

No. What is ultimately dissatisfying about this book is that Orrell wants it two ways. He uses the first half of the book warning us that we are, and have been over our entire history, too confident in our forecasts, that we are unaware of the amount of error in our models, and that we should expect the unexpected. Then he uses the second half of the book to warn us that, based on these same forecasts and models, we are heading toward a crisis, and that if we are not careful, the end is near. He softens the doom and gloom by adding an unsatisfactory “maybe” to it all. He cannot make up his mind and make a clear statement.

Now, it might be that the most dire predictions of climate models, economic forecasts, and emergent disease predictions are true and should be believed. But it cannot also be true that the models that produced these guesses are bad and untrustworthy, as he assures us they are. So, which is it? Are scientists too confident in their predictions, given their less-than-stellar history at predicting the future? Almost certainly. For example, we recall Lev Landau, saying of cosmologists, “They are often wrong, but never in doubt.” Could this also apply to climatologists and economists? If so, how is it we should believe Orrell when he says we should prepare for the worst?

To solve that conundrum, Orrell approvingly quotes Warren Buffet who, using an analogy of Pascal’s wager, says it’s safer to bet global warming is real. Pascal argued that if God exists you’d better believe in him because the consequences of not believing are too grim to contemplate; but if He does not exist, you do not sacrifice much by believing anyway. This argument is generally acknowledged as unconvincing—almost certainly Orrell himself does not hold with it, as he shows no sign of devoutness. Orrell does, sometimes, allow himself to say that people are too sure of themselves and their predictions. To which I say, Amen.

You now need to understand that weather and climate models both require a set of observations of the present weather or climate before they can run. These are called initial conditions, and the better we can observe them, the better the forecasts can be. Ideally, we would be able to measure the state of the atmosphere at every single point, see every molecule, from the earth’s surface, way up to where the solar wind impacts on the magnetosphere. Obviously, this is impossible, so there is tremendous uncertainty in the forecasts just because we cannot perfectly measure the initial conditions. There is a second source of uncertainty in forecasts, and that is model error. No climate model accurately models the real atmosphere. Moreover, it is impossible that they can do so. Approximations, many of them crude and no better than educated guesses, are made for many physical phenomena: for example, the way clouds behave. So some of the error in forecasts is due to model error and some due to uncertainty in the initial conditions.

Orrell makes the claim that most of the error in weather forecasts is due to model error. Maybe so—though this is far from agreed upon—but he goes further to say that these weather models do not have much, or any skill. (Skill means that the model’s forecast is better than just guessing that the future will be like the past.) This is certainly false. Orrell is vague about this: at times it looks like he is saying something uncontroversial, like long-range (on the order of a week) weather forecasts do not have skill. Who disagrees with that? Perhaps some private forecasting companies providing these predictions—but that is another matter. But often, Orrell appears to lump all, short- and long-term, weather forecasts in the same category and hints they are all error filled. This is simply not true. Meteorologists do a very good job forecasting weather out to about three or four days ahead. Climatologists, of course, do a very poor job of even forecasting “past” weather; i.e., most climate models can not even reproduce past known states of the atmosphere with any degree of skill.

Lovelock’s Gaia hypothesis is lovingly detailed in Orrell’s warning that we had better treat Mother Nature nicely. This curious—OK, ridiculous—idea treats the earth itself as a finely tuned, self-regulating organism.? Orrell warmly quotes some “environmentalists” as saying that Gaia treats humans as a “cancer”, and that it sometimes purposely causes epidemics, which are its way of keeping humans in check and curing the cancer. Good grief.

Of course, the Gaia idea is invoked only after humans come on the scene. The earth is only in its ideal state right before humans industrialized. But where was Gaia when those poor, mindless and apolitical, anaerobic bacteria swam in the oceans so many eons ago? The finely tuned earth-organism must have decided these bacteria were a cancer too, as the oxygen dumped as their waste product poisoned these poor creatures and killed them off. So too have other species come and gone before humans came down out of the trees. Belief in Gaia in this sense is no better than those who also believe that the climate we now have is the one, the one that is perfect and would always exist (and didn’t it always exist?) if only it weren’t for us people, and in the particular the Bush “Administration.”

But again, Orrell is wishy-washy. He assures us that Gaia is “just another story” (though by his tone, he indicates it’s a good one). His big-splash conclusion is that models should not be used as forecasts per se, that they should only be guides to give us “insight”. Well, a guide is just another word for a forecast, particularly if the guide is used to make a decision. Making a decision is nothing but making a guess and a bet on the future. So, once again, he tries to have it both ways.

A note on Malthus. What he argued was that humans, and indeed any species, reproduced to the limit imposed upon them by the availability of food. If the food supply increased, the population would increase. Both would also fall together. What Malthus said was that humans are in *equilibrium* with their environment. He never said that people would overpopulate and destroy the earth. He was, though, in a sense, an early eugenicist and did worry that a March of the Morons could happen if somebody didn’t do something about the poor; but that is a story for another day.