Skip to content
July 4, 2009 | 35 Comments

Do NOT smooth time series before computing forecast skill

Somebody at Steve McIntyre’s Climate Audit kindly linked to an old article of mine entitled “Do not smooth series, you hockey puck!”, a warning that Don Rickles might give.

Smoothing creates artificially high correlations between any two smoothed series. Take two randomly generated sets of numbers, pretend they are time series, and then calculate the correlation between the two. Should be close to 0 because, obviously, there is no relation between the two sets. After all, we made them up.

But start smoothing those series and then calculate the correlation between the two smoothed series. You will always find that the correlation between the two smoothed series is larger than between the non-smoothed series. Further, the more smoothing, the higher the correlation.

The same warning applies to series that will be used for forecast verification, like in the picture below (which happens to be the RSS satellite temperature data for the Antarctic, but the following works for any set of data).

Temperature data

The actual, observed, real data is the jagged line. The red line is an imagined forecast. Looks poor, no? The R^2 correlation between the forecast and observation is 0.03, and the mean squared error (MSE) is 51.4.

That jagged line hurts the eyes, doesn’t it? I mean, that can’t be the “real” temperature, can it? The “true” temperature must be hidden in the jagged line somewhere. How can we “find” it? By smoothing the jagginess away, of course! Smoothing will remove the peaks and valleys and leave us with a pleasing, soft line, which is not so upsetting to the aesthetic sense.

All, nonsense. The real temperature is the real temperature is the real temperature, so to smooth is to create something that is not the real temperature but a departure from the real temperature. There is no earthly reason to smooth actual observations. But let’s suppose we do and then look at the correlation and the MSE verification statistics and see what happens.

We’ll use a loess smoother (it’s easy to implement), which takes a “span” parameter: larger values of the span indicate more smoothing. The following picture demonstrates four different spans of increasing smoothiness. You can see that the black, smoothed line becomes easier on the eye, and gets “closer” in style and shape to the red forecast line.

Smoothed Temperature data

How about the verification statistics? That’s in the next picture: on the left is the R^2 correlation, and on the right is the MSE verification measure. Each is shown as the span, or smoothing, increases.

R^2 grows from near 0 to almost 1! If you were trying to be clever and say your forecast was “highly correlated” with the observations, you need only say “We statistically smoothed the observations using the default loess smoothing parameter and found the correlation between the observations and our forecast was nearly perfect!” Of course, what you should have said is the correlation between your smoothed series and your forecast is high. But so what? The trivial difference in wording is harmless, right? No, smoothing always increases correlation—even for obviously poor forecasts, such as our example.

Verification

The effect on MSE is more complicated, because that measure is more sensitive to the wiggles in the observation/smoothed series. But in general, MSE “improves” as smoothing increases. Again, if you want to be clever, you can pick a smoothing method that gives you the minimum MSE. After all, who would question you? You’re only applying standard statistical methods that everybody else uses. If everybody else is using them, they can’t be wrong, right? Wrong.

Our moral: always listen to Don Rickles. And Happy Fourth of July!

Update: July 5th

The results here do not depend strongly on the verification measure used. I picked MSE and R^2 only because of their popularity and familiarity. Anything will work. I invite anybody to give it a try.

The correlation-squared, or R^2, between any two straight lines (of non-zero) slope is 1 or -1. The more smoothing you apply to a series, the closer to a straight line it gets. The “forecast” I made was already a straight line. The observations become closer and closer to a straight line the more smoothing there is. Thus, the more smoothing, the higher the correlation.

Also, to about frequency spectra, or measuring signals with noise, miss the point. Techniques to deal with those subjects can be valuable, but the uncertainty inherent in them must be carried through to the eventual verification measure, something which is almost never done. We’ll talk about this more later.

July 3, 2009 | 2 Comments

Internet Advertising Company to Fund Itself With Advertising

FOR IMMEDIATE RELEASE:

Internet Advertising Company to Fund Itself With Advertising

New York – July 4, 2009 – Word leader in being a new company engaged in ways of bringing advertising to the internet, Hoovem.com announced today that it will fund its operations by selling ad space on its advertisements.

“Making money on the internet is hard,” said CEO Malcom Impsnead. “Most sites don’t think about how they are going to pay off their investors. Look at Twitter, Facebook and the rest. Tens of millions out the door and not one penny coming in. We have figured out a strategy to change all that.”

Hoovem.com serves banner, rich media, flash, and other “lifestyle experience” ads to a number of leading web sites, such as Yahoo.com, ESPN.com, and Foxnews.com. When surfers go to one of these sites, chances are the ads they see will come from Hoovem.

“All those ads we serve take up a lot of bandwidth, and bandwidth costs money,” explained Impsnead. “To pay for that, we’re going to sell advertisements on the advertisements we serve.” Impsnead went on to explain that any site on the internet that is making money is doing so by selling advertising. “We’re sticking with what has been proven to work.”

Internet strategist and writer Bill McTwoon said, “Nobody has attempted what Hoovem’s doing. If it works—and there’s no reason it shouldn’t since advertising is all what drives the internet—we’ll see this implemented at sites everywhere. The time to get in on this is now.” McTwoon said that he thinks the ads Hoovem is selling on the ads is serves, might also serve as platforms for newer, tinier ads.

Contact:
Malcom Impsnead
mimpsnead@hoovem.com
New York, NY

###

July 1, 2009 | 9 Comments

Public Enemies — by Bryan Burrough

I am re-posting this book review because the movie came out today, and it might be of interest.

Public Enemies
Public Enemies: America’s Greatest Crime Wave and the Birth of the FBI, 1933-1934

Bryan Burrough

Recommendation: read

It was one of those rare confluences in history where the notable and notorious reach critical mass and all manage to bump into one another. It happened when Dorothy Parker, Robert Robert Benchley, George S. Kaufman and others coalesced at the Algonquin Round Table; times were right for Werner Heisenberg, Albert Einstein, Niels Bohr and other great brains in Europe to meet and create modern physics; and so with Machine Gun Kelly, Ma Barker, Bonnie & Clyde, Pretty Boy Floyd, Baby Face Nelson, Dillinger. These Midwest kidnappers and bank robbers—or yeggs—flourished during the same brief time.

These underworld entities kept running into each other. Verne Miller, hit man responsible for the notorious Kansas City Massacre—a botched rescue of yegg Frank Nash, who was killed along with several agents—was being chased by cops in Chicago and ran into an apartment building. “In a bizarre coincidence, this was the very building where John Dillinger was then living.” Miller and Dillinger didn’t know each other.

If you were raised in the States in my generation or earlier, you will have heard of most of the names (probably from Bugs Bunny cartoons). These criminals were celebrities, with all the disgusting whitewash that status brings. For example, if you’ve only seen the movie, and haven’t read Mike Royko’s classic column reviewing it (near bottom of link), then you don’t know Bonnie & Clyde. Human scum. Murderous dreck. In the book, I cheered when Bonnie & Clyde’s car crashed and caught on fire, burning her down to the bone. She was in well-deserved agony for weeks. Both ran like terrified animals, scared and mentally defeated most of the time—but still managing to get in a murder or two—until they were finally hunted down by a Texas Ranger and some Louisiana cops and given the death they so richly deserved.

The book also details the creation of the FBI and the rise of Hoover, a man unknown to the world until he was able to ride the crime wave to shore up the struggling Bureau. Burrough argues that Hoover would have disappeared had he not had the all those lucky kidnappings and bank robbings that needed policing. The FBI was no certain thing.

There aren’t as many details about the FBI as there are of the crooks. We do learn that Hoover took his job seriously and that he sought publicity for the sake of the Bureau and not just for himself, but he wasn’t against exaggerating. He painted Ma Barker, in life a lonely solver of jigsaw puzzles, as the brains behind the Barker outfit after she was accidentally killed in a shootout by a stray bullet. His concocted story was better than explaining how an innocent old lady got shot. My guess is that Burrough will follow this book up with another focusing on Hoover.

Public Enemies

Crime buffs will love the detail, sometimes given minute by minute. But it’s a lot to go through. Names come at you fast, actions are sometimes hard to follow. This isn’t Burrough’s fault: his stated purpose is to be complete, and he had access to newly opened FBI files, but there’s work involved on the reader’s part to keep up. But the writing is good and pleasurable.

We meet everybody who was anybody in criminal and law enforcement circles—except for those in the Chicago syndicate and New York mob, organizations which had few ties to the outlaws—but it’s Dillinger that’s star of the show. Charismatic bank robber, one time murder of a cop who tried to arrest him, and by all accounts a very charming man. He knew he was a media darling and played up to his image of a tommy-gun toting robber of the rich, using it at one point, with assistance from his mafia-connected lawyer Louis Piquett, to make an escape from jail easier.

Hoover assigned initially Melvin Purvis to the job of hunting crooks, but he turned out to be a walking CF (a precise, and deeply meaningful term I learned in the military: the first letter stands for cluster; if you don’t know the whole phrase, it means screwup). Purvis was lazy, negligent, and couldn’t even find his own hat, let alone any crooks. But guess who the media gave the glory to? The man Hoover sent to supervise Purvis—Samuel Cowley—never cared for credit, and never got it, even after he was murdered by Baby Faced Nelson (at one point, Dillinger was a member of Nelson’s gang). Once reporters find an angle, they stick to it, truth be damned (some things never change).

Hollywood writers are the same. There is a moved “based on”—which almost always means “some of the names are right, all other details are changed”—the book about to appear. The trailer makes it appear that the flick’s title is “Dillinger versus Purvis: Battle to the Death”. Johnny Depp looks OK as Dillinger, but intensity-ham Christian Bale comes across goofy as always. Anyway, there is no way the movie can capture the rich details of the book.

Dillinger, it must be said, is highly filmable and had excellent taste in dress, a “clothes horse.” Discussion of dress salts the book. One passage sees Dillinger buying “sassy cravats” to go with his sharp boater. Another incident describes him donning a pair of size 24 Hanes briefs, “a pair of lightweight gray slacks, black socks, red Paris garters, and white buckskin Nunn Bush shoes. He buttoned his white kenilworth broadcloth shirt and twisted on a red-print tie.”

Hoover was no sartorial slouch, either. One picture shows that he knew that when wearing an overcoat, one must also wear a scarf for balance; and he didn’t neglect the pocket square. This was the time when male elegance was at its peak, and kids dressed like adults instead of the other way around.

June 30, 2009 | 19 Comments

The MIT Dahn Yoga Brain Respiration Experiment: Part I

This series originally begin 3 January 2009. This is only Part I; the remaining parts are linked below.

INTRODUCTION

On 20 September 2004, in the Physics Department at MIT, sponsored locally by the Body and Brain Student Club, I proctored a test I had designed to probe the psychic powers of three Korean children.

The kids were under the tutelage of Ilchi Lee and the Korean Institute of Brain Science (KIBS). That group claimed that kids who had training in their methods of Brain Respiration and Dahn Yoga would be able to “read” the colors of cards inside opaque envelopes while blindfolded.

KIBS called this ability “non-visual, heightened sensory perception”, or HSP. They insisted that this is not the same as extra-sensory perception, or ESP, though the differences seem largely semantic.

This, in five parts, is the story of that experiment, what events led up to it, what happened during, and the sad but predictable aftermath.

I have written about this experiment before, but this is the first time I present all details. This will take us some time, and might even be tedious at times, but it is, you will see, necessary to be explicit as I can be.

BACKGROUND

In the fall of 2004, Dr Sung Won Lee was an attending physician in the department of General Internal Medicine and Center for Complementary and Integrative Medicine at the Cornell University Medical School. I was there, too, as an Assistant Professor of Biostatistics.

Sung and I were friends. He is one of the sweetest humans on the planet. We had been to each other’s homes for meals, compared grievances about our boss, and so on. We used to argue the physics and biology and what it means to be conscious, what constitutes evidence for or against a theory, the strictures of philosophical systems, what is and isn’t “science”. Sung put a lot of thought into his beliefs and was suspicious of logic.

Dr S. Lee got involved, a couple of years earlier, with Dr Ilchi Lee (no relation), KIBS, and Dahn Yoga. As part of an official experiment he was conducting at Cornell, Dr S. Lee was teaching adults Dahn Yoga to measure their improvement in various quality of life measures. Sung had many close social associations with other members of Dahn Yoga.

Sung knew that I had written a book about testing psychic phenomena—So, You Think You’re Psychic?—and that I was interested in magic, particularly mentalism, which is the art of fooling people into thinking you have psychic powers. So he asked whether I’d be interested in designing a test for KIBS.

He explained that his friend, Dr Ilchi Lee, had developed a method called brain respiration that would allow people to develop extraordinary sensory powers. For examples of these claims, click here. Note that this is only one of the many dozens of sites run by that group. More on this later.

Sung thought that training in brain respiration methods would allow kids to be able to “feel” colors—without, of course, using their eyes. He said that KIBS-trained kids could, while blindfolded, hold colored cards in their hands and correctly identify the color of those cards by touch. Sung had a promotional video tape from KIBS showing these miracles.

He brought the tape to my apartment and we watched it. The tape was in Korean, but Sung provided the translation. It started by showing a batch of children about six to twelve years old attending a brain respiration orientation. Before they knew its secrets, they tried reading the colors of cards blindfolded, but failed. They also tried to stick spoons on their face by “magnetizing” themselves but failed that, too. The kids then underwent training and afterwards could read colors blindfolded and could stick spoons to their faces. There might have been other abilities, but I’ve forgotten them.

I was exasperated by the performance because I thought the kids were obviously cheating. Sung believed they were not. I got a spoon from the kitchen and demonstrated it not sticking. I then “magnetizing” myself and—lo!—the spoon stuck. This is a simple trick, the simplest maybe, of all magic tricks. To get a spoon to stick, simply press it hard so that as large a surface area of the spoon touches your skin as possible. Make sure the bowl sticks outwards. It helps if your skin is not dry. I put mine on my forehead and nose.

I fashioned a blindfold and Sung handed me colored cards. I could easily “feel” their colors by cheating. Blindfold tricks are the next easiest and cheating is so easy a KIBS child could do it. Try it and you’ll find how simple it is to see anything, as long as you’re able to move the object to an advantageous spot, which the KIBS kids could.

I warned Sung that there were two ways to be fooled in psychic tests: the investigator could see things he wanted to see but weren’t there, and the person under study could cheat, knowingly or not. Psychic phenomena are supposed to occur by other than normal means, but most people do not appreciate how perceptive we are, and how we can sense subtle clues from our environment that might not allow us to, for example, say exactly what object is held in an opaque bag, but will let us guess better than randomly.

The temptation to cheat is strong, especially in kids, who are anxious to please adults and who enjoy praise when successful.

The stronger an experimenter wants to believe the more likely he will miss clues that his subjects are cheating, or to dismiss the times he did catch his subjects cheating as aberrations, or even as necessary to “relieve testing stress.” This has happened countless times in psychic testing.

In short, psychic tests are difficult because you have to work hard to eliminate all possible avenues of sensory leakage, remove the subtle clues that people unconsciously use, and make it impossible for cheating to occur.

You already believe, I told Sung, but others do not. The only way to convince outsiders that brain respiration was real was to produce incontrovertible evidence of its abilities under the strictest possible conditions. Sung agreed with this.

I told him that if a controlled experiment succeeded, I would be willing to believe in brain respiration, but given my experience with situations like this, I did not think it would work.

Then came my big warning. I explained what happens when people who believe strongly in psychic powers are presented with positive evidence that the powers are false. They refuse to acknowledge that evidence. They dismiss it, explain it away, insist it is flawed, or a fluke. In no way do they lessen their belief in the power; if anything, their belief strengthens.

They also blame others for the failure.

Sung said none of this would happen. He was confident that the KIBS trained kids would pass any test, and was excited about the prospect of demonstrating to the world the amazing possibilities of Dahn Yoga and brain respiration and training.

I said, OK, I would design the experiment.

Next: The design

Part I, II, III, IV, V