Skip to content

Category: Philosophy

The philosophy of science, empiricism, a priori reasoning, epistemology, and so on.

November 7, 2008 | 32 Comments

Michael Crichton and SETI

Michael Crichton, as you will have heard by now, is dead. Unfortunately.

The Wall Street Journal today reprinted an excerpt of a speech Crichton gave called “Aliens Cause Global Warming.” Regular readers of this blog will know Crichton’s opinion on the certainty of man-made catastrophic climate change. Just a reminder (from his speech):

No longer are [climate] models judged by how well they reproduce data from the real world — increasingly, models provide the data. As if they were themselves a reality. And indeed they are, when we are projecting forward. There can be no observational data about the year 2100. There are only model runs.

This fascination with computer models is something I understand very well. Richard Feynman called it a disease. I fear he is right. Because only if you spend a lot of time looking at a computer screen can you arrive at the complex point where the global warming debate now stands.

Nobody believes a weather prediction twelve hours ahead. Now we’re asked to believe a prediction that goes out 100 years into the future? And make financial investments based on that prediction? Has everybody lost their minds?

To explain why he was flummoxed, Crichton first made a point about SETI, the Search for Extraterrestrial Intelligence. A lot of people in that field make reference to the Drake Equation, originated by SETI big cheese Frank Drake. That equation is

  • N = R * x fp x ne x fl x fi x f X L
  • .

We want to solve for N, which is the number of civilizations in our galaxy with which intelligent communication is possible. N depends on the rate of star formation R *, the fraction fp of those stars that have planets, and all those other things you can look up.

Crichton says:

This serious-looking equation gave SETI a serious footing as a legitimate intellectual inquiry. The problem, of course, is that none of the terms can be known, and most cannot even be estimated. The only way to work the equation is to fill in with guesses. And guesses — just so we’re clear — are merely expressions of prejudice. Nor can there be “informed guesses.” If you need to state how many planets with life choose to communicate, there is simply no way to make an informed guess. It’s simply prejudice.

The Drake equation can have any value from “billions and billions” to zero. An expression that can mean anything means nothing. Speaking precisely, the Drake equation is literally meaningless, and has nothing to do with science. I take the hard view that science involves the creation of testable hypotheses. The Drake equation cannot be tested and therefore SETI is not science. SETI is unquestionably a religion.

The fact that the Drake equation was not greeted with screams of outrage — similar to the screams of outrage that greet each Creationist new claim, for example — meant that now there was a crack in the door, a loosening of the definition of what constituted legitimate scientific procedure. And soon enough, pernicious garbage began to squeeze through the cracks.

I agree with him that none of these terms can be known exactly, or even sufficiently precisely to calculate a quantitative answer for N. I also agree that the pursuit of N can take on religious qualities.

But I can’t agree that SETI itself is worthless, nor can I agree that interest in it loosens the definition of “legitimate scientific procedure.” SETI is not just the Drake equation.

Now, I will not attempt to defend even one procedure that SETI workers use, nor will I comment on any statement made by any of its proponents. I cannot say, for example, that searching nearby stars for signals in the hydrogen line makes any sense. But I will say SETI is not the same as religion

I am interested in saying something about the probability of this proposition:

    S = “Intelligent/sentient life besides that on planet Earth exists”

Because we must calculate the probability of S is conditional on some evidence, I offer this blog. Yes, because this blog—because you and I—exist, it means that the universe is set up to allow at least one species of sentient life. Therefore, it is rational to believe that the probability of S given this evidence is greater than 0. I have no idea how much larger than 0 it is. If you are a fan of the reasoning behind the Fermi Paradox, you might say that the probability, while non-zero, is trivially small.

The Fermi Paradox basically says that, since the universe is about 10-13 billion years old, and the one sentient-life example we know of only took about 4-5 billion years to evolve, and since there are plenty of stars and galaxies, there should be sentient life all over the place. That is, SETI should be easy, and since it isn’t, since we haven’t made contact yet, this implies that we are the first or only sentient species. There are obvious subtleties to each stage of that argument that I glossed over, but that’s the gist.

The Fermi Paradox is also conditional on information not articulated. One obvious item is the proposition that all sufficiently advanced civilizations would want to make contact with us. Not just with other species, but with us. That’s a mighty big supposiion. Another hidden assumption is that we ourselves are sufficiently advanced enough to detect messages aimed at us, or have the ability to intercept messages meant for other beings. Pretty big guess, especially with the knowledge that the more efficient a message gets, the more it looks to an outside like noise (basic information theory; deep ties with probability and statistics there), and so civilizations more advanced than us might have communications which are impenetrable to us.

That argument cuts both ways, of course. If the messages are too complex, any search for them is fruitless. And, well, you get the idea. It’s complicated, so much so that it is not an open and shut judgment that SETI is valueless.

Though we have to be careful. Wishcasting is always a danger here, as everywhere. A lot of people—me included—want S to be true and this naturally clouds our judgment.

October 31, 2008 | 15 Comments

Breaking the Law of Averages: Real-Life Probability and Statistics in Plain English

It is finally done!

Breaking the Law of Averages

You may order directly from the publisher here1. The book will also be available on Amazon, Barnes and Noble, etc. in about a month. I’ll update this post with the links when the book is in the distribution channels. Order as many copies as humanely possible.

Signed copies
I have had several requests for signed copies. I’ll be happy to do this for people. If you want a signed book, please email me at Please use “SIGNED COPY” as a subject line, and include your address in the body of the email. I’ll buy a few books from the publisher and then re-ship them out to people who want a copy. The charge will be the same as the publisher’s plus the same as they charge for Media Mail shipping and handling ($5.90), plus $1.15 cents (to cover tax). This makes the cost an even US$32.00. Payment will be arranged through PayPal (apparently, you don’t have to have an account to pay this way). I’ll send those who email me a PayPal “Request for Payment”; after that is received, I’ll ship the book (anywhere in the world).

Because I first have to order copies, sign them and then mail them out, it will of course take longer for you to get your book. I will wait a couple of days to see how many people email so I have a rough idea of how many books I should order.

I have two permanent places for news of the book:

  1. My books tab (see upper right of screen): general news and information
  2. Code page: free R code examples, erratum, links to papers, data, etc.


Why is this book different?

Statistics has traditionally been taught decade after decade in a fashion that is long outdated. This book presents a brand new way of understanding probability and statistics at the introductory level. The approach taken does not require mindless memorization. There is very little math, and what there is requires nothing beyond multiplication and division. This book takes busy work out of standard statistics and puts insight back in.

Preface excerpt:

The regular readers of my blog, where parts of this book previewed piece by piece, provided razor sharp editing and keen questioning and kept me from making major blunders. So thanks to (screen names) Mike D, JH, Harvey, Joy, Noahpoah, Harry G, Bernie, Lucia, Luis Dias, Noblesse Oblige, Charlie (Colorado), Dan Hughes, Mr C Physics, Jinnah Mohammed, Ari, Steve Hempell, Wade Michaels, Raphael, TCO, Sylvain, Schnoerkelmanon, and many others (sorry if I left you out!). Any mistakes left are obviously their fault.

What’s next?
I use the book in my own classes, of course, and a few other professors have been either using a draft or have expressed interest in the book for their classes. If, by some miracle, the book becomes popular, I’ll start working on a “Answers to Selected Exercises” or, given that I get substantial comments from actual class use, a Second Edition. But that is all in the far, far future.

If you are a professor of a statistics class and want to chat about the book, send me an email at and we can set up a time to talk. I have had great success with this approach for beginning students and can let you know how I run the class. Some guidelines are also given in the Preface.

1The cover art looks terrible on the publisher’s page. They have scaled it down from an enormous PDF to a small JPEG and it is pixelated. It looks great when printed, however.

October 21, 2008 | 11 Comments

Science is decided by committee

Scientists still do not appear to understand sufficiently that all earth sciences must contribute evidence toward unveiling the state of our planet in earlier times, and that the truth of the matter can only be reached by combing all this evidence. . . It is only by combing the information furnished by all the earth sciences that we can hope to determine ‘truth’ here, that is to say, to find the picture that sets out all the known facts in the best arrangement and that therefore has the highest degree of probability. Further, we have to be prepared always for the possibility that each new discovery, no matter what science furnishes it, may modify the conclusions we draw.

—Alfred Wegener.

We have all heard Wegener’s sad story. How all of “science” aligned against him and his bizarre, false, ridiculous, obviously false theory of continental drift. What happened, more or less, and certainly not formally, was about 100 years ago all geologists got together and voted that Wegener had lost his mind. But, of course, and in fact, they had, and from Wegener arose the fascinating study of plate tectonics.

Then there is the Rene Blondlot saga. All of “science” aligned against his, too, and his weird, silly, sad, and pathetic theory of n-rays. What happened was that about 100 years ago all physicists got together and came to the consensus that poor Blondlot had lost his mind. And, of course, he had. From Blondlot came the cautionary tale of how easy it is to fool yourself, even if you happen to be a very smart man. There are no n-rays.

I don’t want to dwell on the point here, but there is no such thing as science. There are things we know and things we don’t. There are more things we think are true, and many more we think are false. And that’s it. But the purposes of this essay, I’ll, like everybody else, use the word but leave it vague and undefined.

Now, for every Wegener, there is at least one Blondlot and certainly hordes of nameless others, each touting their own personalized, probably false theories-of-everything. What this means is that because some person touts a theory which “science” denies, it is more likely that that theory is false than it is true. Thus, it is usually rational, for example, to seek Dr Smith’s of State U.’s opinion on Joe Jones’s new theory of zero-point energy. That is, an appeal to the consensus is rational.

The opinion of a great many learned persons concentrated in one place is a good filter of nonsense and falsity. But this filter is too often applied indiscriminately and too assiduously and it often blocks truth, particularly if the truth is new and different, or it is against a vogue that has taken tight, but temporary, grip on the academic masses.

Max Planck: “A new scientific truth does not triumph by convincing its opponents and making them see the light, but rather because its opponents eventually die, and a new generation grows up that is familiar with it.”

Thus, in with the new but only after the those with the old are out. It’s often not as bleak as Planck painted it, of course. Some fields, especially when they’re young and unossified, grow by leaps and bounds, and each new idea, no matter how trivial or valid, is celebrated. It’s only after a field has had time to metastasize, that is, be formally recognized by its own separate department—complete with chairs (endowed, naturally), meetings, and new journals—that the filter becomes fully functional.

Academic freedom and its opposite

It might, then, not surprise you to learn that as a professor at a university, the locus of emanations and endless chanting about academic freedom, you cannot teach what you want. You cannot even study what you want. You can still think what you like, but, as we all know by now, you cannot always speak or write it.

What I mean by this is that peer review is not confined to accepting or rejecting journal articles. The sword is also wielded inside departments. Courses, for example, are decided upon by a committee. Many committees, actually. There is a departmental one (or more than one), then usually one at the “school”, or group-department level. There is sometimes another beyond that at the university-wide level.

Each level has to vet and approve any new course so that, among other things, it fits in among other courses, that the material is aligned with the consensus, and so on. These are reasonable goals, but the constrictions lead to tremendous inertia.

The amount of innovation (in teaching method or material) allowed in a course is inversely proportional to its difficulty. Thus, very advanced courses—seminars,1 usually taught just to graduate students and other professors—are wide open. You can teach exactly what you like, require what you like, depart on any tangent. There is little consensus about what to teach or how.

But in 101-type classes, your behavior is strictly proscribed. The book2 is decided for you, the lesson plan is decided for you, and in some places even the quizzes, homeworks, and exams are decided for you. Again, usually this is not entirely bad. The closer the field is to being driven by logic or is empirically verifiable, the more likely that the basics in that field are known, and that the optimal order and method to teach the introductory concepts have been hammered out. So, for example, all physics students should learn that F = ma, “pre-calculus” students must know that ln(exp(x)) = x, and all chemistry students should know in what way a proton, neutron, and electron are different. Which is to say, there is a consensus about what is known and what is best about the fundamentals. This obviously works against you when the fundamentals have recently changed,3 or the fundamentals are in dispute.

A within-department consensus always exist to also ensure that the work professors do is limited. Most do not feel that it is a burden to toe the line; after all, most professors are hired to work on a specific sub-sub-area within a field, and it is this area in which they enjoy working. Academics attend meetings in their specialties and sub-specialties and the areas of work which are popular are discovered. This group-think can lead to success of the kind we have certainly seen in many fields, but it also tends to narrow the scope of new work. We have all heard somebody tell us “You’re working on that? Nobody is interested in that.” And we have taken its meaning: work on something popular or your tenure or promotion will be more difficult.

The trend towards specialization has a built-in positive feedback. The more people work in a narrow field, the narrower that area becomes, or the more likely an area splits into two or more areas which also constrict. Again, not always bad, as this can lead to rapid progress, but it clearly not the best model for all people or all fields. If you are hired to do radiative cloud modeling, for example, you are not encouraged to dabble in your neighbor’s boundary layer fluid flow problem. You certainly would receive odd looks if you were to suddenly discover an interest in, say, difference equations or philosophy. You might furtively work in these areas that are “not yours”, and you might even publish in them, but you will not receive any credit for doing so, and as I said above, papers published in other areas might even work against you: “She’s not focused” is a commonly heard phrase. Which is to say, broad curiosity is not rewarded; potentially stultifying specialization is.

On being wrong

The closer a field of study is itself to politics or any area which involves human behavior, the more the consensus acts to keep people in line than it does to promote innovation. Non-consensus ideas are not welcome. Professors holding verboten thoughts are not hired, or if they are found out, they are let go, or they even leave voluntarily, tired of the process.

Naturally, the more a field agrees on what is actually true, then the stronger the consensus is to be sought. Problem is—as you might have guessed—is that people in these human-centered fields often feel, as people in more physical fields do not, in the grip of enlightenment and so always advocate the consensus stridently. The reasons for this are obvious and well known. The solution seems to be, because people in areas which involve humans are prone to ill-informed zealousness, that they should all be taught and consistently reminded that they might be wrong. This is the reason, after all, that, on average, people involved in physical areas are humbler: they have seen and verified their failures, and they have seen and acknowledged that their predictions sometimes are a bust.

Not all who work in physical areas are so lucky as to face correction. Today, there are at least two fields in which predictions are being made that either cannot be verified or cannot be verified until quite a lot of time has passed: string theory and climatology. The best these two fields can say is “Observations we have seen are consistent with our theory.” A true, or mostly true, statement. But, and I need hardly point this out, the observations can be equally, or even more, consistent with different theories, even theories which make opposite predictions. This is why making predictions is more important than explaining what we have already seen.

In fields where making predictions is more difficult, again, the human-centered or influenced ones, the local consensus is stronger, and people in those fields look more to the past to find observations which support their views. Evidence is picked over, and the best—in the sense of most agreeable—is kept, the rest discarded or explained away. The more a field is in the grip of explanation, the stronger the consensus will be, and of course the greater the chance that there will be splinter consensuses.

This is contrasted with fields in which (verifiable) prediction is king. There may be—there certainly are—splinter groups, but people can and do swear allegiance to more than one group. The consensus in these groups is more fluid and more likely to change on short notice. If there are many factions—explanations for a phenomenon—the first from which arises a correct prediction is the one that gains the most support. If that explanation can continue to make verifiable predictions, then eventually the explanation is accepted and becomes part of the consensus.

Everybody who agrees with me, raise their hands

So far we have seen that the consensus can work both for and against what is true. This should not be surprising. Research is done by people, and people have foibles. The process, on the whole, and especially in areas which do not involve human behavior, appears to be working. It is a clunky system, but it has shown results and still has promise.

The system breaks, as it always has, when people fall in love with an idea because that idea fits in with other deeply held beliefs, or when people simply want the idea to be true. When these like-minded people form a group and then a consensus, progress is halted, or even set back. These people need more experience with failure—that is, with acknowledging failure. I have no clear idea how to do this.

Naturally everything in this essay is subject to dozens of caveats and exceptions to the rule. The general theme sticks, however: people are generally too sure of themselves.


1Incidentally, these seminar courses are often taught “off the books” by the professor. Meaning they do not always count towards their official teaching load. Credit for students taking seminars is usually limited, too.

2The difference between the 101-books used in these courses is driven more by economics and fad than by fact or material.

3This happened in physics about 60-70 years ago, and is happening in statistics now.

October 16, 2008 | 45 Comments

Bad news for Bonobo

It turns out—shockingly, to some correct-thinking academics—that the bonobo ape is just as bloodthirsty as the rest of the higher primates. Yes, it’s true.

Bonobos, a sex- and peace-loving species of ape often held up as an exemplar for human emulatation, like to hunt, kill, and eat other primates. Researchers first learned this by looking at bonobo poop, which contained more than just the expected half-digested berry seeds.

After the spoor had given up it secrets, researchers put a tail on some bonobos and they discovered the truth: the apes hunt in packs, which is obviously more efficacious than hunting singly. Their prey, after all, is fast and wary.

Now, this wouldn’t be the least interesting (or even surprising) except for a curious development in the Enlightened world (Europe, of course). Spain will grant human rights to apes.

Some of you will hail this special instance of Progressive thinking; but before you cheer let me remind you of a fact. Logically, you cannot have a right without entailing a responsibility. What this at least means is that if you grant “human” rights to apes, you must also ensure they own up to their “human” responsibilities.

Thus, if a certain ape were to, say, steal a banana from a fellow ape, he would be guilty of theft, and so must be held accountable and punished. If a gorilla were to be so bold as to take more than one mate, he must be prosecuted for polygamy. If a monkey, as monkeys sometimes do, kills a conspecific, then that monkey must pay the price (not the death penalty—that would be inhuman—but perhaps life in prison).

The immediate consequence is obvious. Spain will require an enormous number of translators so that, when a primate is brought to court, he can be made to understand the charges against him.

Meanwhile, in another Spanish-speaking country, Ecuador, people have just voted in a new constitution which—wait for it—grants human rights to Mother Nature. Some of the language from that constitution:

The State will apply precaution and restriction measures in all the activities that can lead to the extinction of species, the destruction of the ecosystems or the permanent alteration of the natural cycles.

The introduction of organisms and organic and inorganic material that can alter in a definitive way the national genetic patrimony is prohibited.

This can be read to mean, for example, that no more will farmers be allowed to breed their stock in an intelligent manner, nor will they apparently be allowed to use fertilizer. Tough luck for the farmers—and for the people who have to eat their food.

But to concentrate on the negatives that will befall some people misses the main point: this language snippet delineates certain rights for Mother Nature. And just like with the apes, you cannot have rights without responsibilities.

This obviously means that the next time a flood washes away some property, Mother Nature must be held accountable. When lightening kills a cat, a penalty must follow. If a person is killed in a storm….something must be done!

But I can’t bear to think of it. Because, as everybody knows, “You don’t fool with Mother Nature.”

(Thanks to SooperDave for finding the Mother Nature Clip!)