THE EXPERIMENT
The test was filmed by several cameras, most from the KIBS people (I have been unable to find any videos on line). The KIBS staff itself was large, about a dozen or two. The public was not barred from filming or photographing any part of the proceedings, and many did. I did not meet many people, but I was told prominent physics and medicine professors from Harvard and MIT were invited, as were other “thought leaders”. There were about forty people not from KIBS in attendance.
There were snacks and drinks provided. I set up in a corner and tried to be be non obtrusive. From what I could gather from the chatter, the audience was anxious but expected good things. Everybody, staff included, was all smiles. Each kid was more adorable than the last, and we were all pulling for them.
Ilchi Lee gave a standard opening speech in Korean, which was simultaneously translated by Sung Lee. Sung also gave a speech explaining the history and goals of brain respiration and what was expected that night. I was introduced, as were the proctors, but none of us spoke.
Sung explained that, in practice, the kids had hit rates of 80-90% with the test envelopes during the previous two weeks. As I said before, I have no clear idea how they did these unsupervised trials. We were thus expecting each kid to guess about 10 more envelopes correctly.
Sung outlined the experimental protocol, assured the audience we would see some special that night, and the test began.
Blindfold phase
The three kids sat together at a table, but only one at a time was blindfolded. Not surprisingly, the kids got all the blindfold guesses correct. I’m pretty sure they didn’t peek down their nose, though they could have above it because they held the cards up high in the air to indicate that they were ready to announce their guess.
But they didn’t have to peek to cheat. I and two other proctors, who also had experience in magic, felt that the kids were cluing each other (the third proctor admitted the possibility but did not care to guess); which is to say they were either telling the blindfolded kid openly what color card he held, or using other signaling methods. There were certainly countless ways the kids could have communicated to each other. All in the room could see the card except the blindfolded kid. The kids were fairly close to one another and they were allowed to move about and talk when the other of them was blindfolded. I’m fairly sure on the kids and the KIBS staff spoke Korean, so there was no way for the proctors to definitively know what was said. I suggested to one Sung about the potential ways the kids might have cheated, and that there were ways that we can check this. But he was not interested in finding out!
Interestingly, a good portion of the audience cheered and applauded when the correct guesses were revealed. One woman, not affiliated with KIBS, was moved to tears.
This phase of the experiment went exactly as I had expected it to. The kids in the KIBS tape never missed when wearing blindfolds, and neither did they now in front of an audience.
Envelopes phase
After a five-minute pause, the main trial began. As mentioned, each kid had a separate proctor, and I watched from the sidelines. The kids took up to 20 minutes for each guess, announced his guess in English, the proctor repeated the guess, the kid confirmed it loud enough for all to hear, the proctor wrote down the guess then checked the envelope for holes or tears which would have allowed the kid to cheat, then the proctor opened the envelope and wrote down the actual color.
We never found any holes, though one kid’s envelopes were consistently wet as if he tried to lick them and perhaps allow a tear—but the paper was too thick to allow him to see anything. The dampness could also have been sweat as this kid occasionally held the cards to his face tightly. He might have also been trying to see through the envelope as he consistently pressed the envelope to his face and pointed his face towards the ceiling lights. But my prior tests of directly holding the envelopes on the surface of an incandescent light bulb assured that it would not be possible to see through the envelope using a ceiling-mounted fluorescent bulb.
The first set of cards took about eight minutes. The kids grunted and sweat, ate come chocolates, did some exercises, and announced their guesses—not all at once, but when each was ready individually.
There was tension in the room when the first kid spoke his guess. The proctor confirmed it and then began to open the envelope. Remember, nobody, not me, not anybody, knew what the card’s color was going to be. People held their breath. I held my breath.
It was a miss. The color did not match the kid’s guess.
No problem, 35 more cards to go.
After a few more misses—and open groans and even hand wringing—there was finally a hit, followed by applause and cheers. Had things finally turned around?
Most of the guesses were wrong, and the pressure started to mount on the kids. Each kid was to get 12 envelopes, and by chance we’d expect they’d get 2 or so correct. But the hits—the correct guesses—were slow in coming.
The kids knew they were failing, the audience had become mostly silent, or sat quietly talking to one another. The kids began to get up more, ate even more candy, exercised more. But no change. Most guesses were misses.
Eventually, after it became clear that nothing more was going to happen, Sung told me he was going to stop the experiment out of concern for the kids’ anxiety. The remaining trials would be marked down as misses.
Sung stepped up and announced the trial’s ending. The audience understood, and clearly felt for the kids.
One kid did 7 trials, the other two did 6 before the experiment was stopped . They were scheduled to do 12 trials each. They got 4 hits during these 19 trials, right what chance would predict: kid one got 1, kid two got 1, kid three got 2.
Recall that before the trial started, KIBS staff members were confident each kid would get at least 10 out 12 hits.
Immediate sequel
Because the test was a failure, the oldest kid wanted to do another blindfold demonstration. I should not have allowed it as it was not part of the official protocol agreed to before the test. But I weakened and said Okay.
This demonstration went the same as before: he got both new cards right. It was then suggested be a proctor that all three kids be blindfolded at once, and that, respectfully, no noise be made. This was because the Sung and the KIBS people finally took seriously our suggestion that the kids were cuing each other when one was blindfolded and the others were not.
I still didn’t love this idea because there were many other people in the room who could have cheated if they wanted to (the KIBS staff, audience members, and so on), but the audience insisted on it. Again, I should not have allowed this, but I was weak.
Only kids two and three attempted a reading, but all were blindfolded anyway. In the previous blindfold demonstration, each kid made their guesses in under a minute. This time—when none of the kids could see each other—it took about five to seven minutes until kid number three started to complain of a stomachache. And so, even the last blindfold demonstration was called off with no guesses made.
The end
The evening ended with both Sung Lee and Dr. Ilchi Lee giving rambling, long, impromptu speeches saying things like, “Absence of evidence is not evidence of absence”, they hoped that more trials would be done, none of us really understood what is going on, and something about “mysterious Z-rays that are absorbed or emitted by the human brain.”
None of it made any sense, and the audience began shifting their traps by the end, clearly ready to leave.
Nobody from KIBS spoke to me after the test. I walked out with Mark Glickman and said goodbye.
Except for Sung Lee, that was the last time I talked to any KIBS member.
Next: the aftermath
This is just fascinating!
Seems pretty conclusive to me that nothing special is taking place. I look forward to the aftermath.
Bet Itchy is still conning people.
I had never heard of Dahn Yoga before, it seems to be a booming business in the USA. Knowing there is an aftermath, my guess is that the objections to the experiment are:
– the boys had a bad start, this led to stress, and this led to a suboptimal use of their brain.
– the original claim was that the children can “sense” colors with their eyes closed, the hard nosed statistician missed this nuance in his design.
For those of you who want to see and hear the main players:
Dr. Sung Lee on ABC News “Healthy Life” Feb. 28 2008
http://www.youtube.com/watch?v=3kYyeijZKA8
Dr. Ilchi Lee
http://www.youtube.com/watch?v=yv1fV6CJjXo&feature=related
(at 6.30 talk about his University, at 7.00 about “sensing” colors)
hmm.
so what exactly were you expecting ? The kids to choose 10/12 cards correctly ?
or what transpired ?
What’s the point of guessing colors? Can Drs Lee and Lee train the kids to pick stocks? Or tell me when to buy puts and calls on the S&P 500?
My heart goes out to the children.
JH – me too! And especially because they were Korean.
I have had some experiences with Korean students who attended summer language classes in the US. I must say that my contacts were with a small subset of Korean kids under special circumstances, and so by no means constitute a survey or broad sampling.
But I found them uniformly to be suffering from inferiority complexes. They were desperate to fit in, to be seen as “equals” to American kids, and complete failures at that. They became surly quite quickly. The same pattern appeared with three different groups in different years with different language instructors.
Expectations did not match outcomes and much psychological trauma resulted. The school eventually abandoned the program because year after year the kids became so unhappy so quickly.
Just my limited experience. It may not be true for all or even most Korean kids. But when I first read of this study in the earlier post, I got a bad feeling. Terrible thing to mess with emotionally sensitive kids in this manner. Cruel.
Happy new Year TCO!
Harold:
Those clips! Especially the first one was painful to watch. Whatever the character of the man, it concerns me that he is a physician. His anatomy and physiology knowledge would be a ’fail’ as far as his description of brain blood flow with the head shaking nonsense. His description of relaxed breathing is nothing new, but is expressed in such a way as to give the impression that the autonomic nervous system (sympathetic/parasympathetic) is like a tap that can be easily switched on and off at will. This is a gross exaggeration and is misleading. I would expect this sort of talk from an alternative health practitioner, not from someone trained in medicine to any basic level.
I thought a bit before posting this but can’t let it pass without comment.
When giving advice on remedies for stress, who cares how ridiculous it sounds, as long as it’s safe- if it works, don’t knock it, placebo is a powerful thing. However, to describe the mechanism for how something might work where there is no clinical evidence or reasoning is wrong. If he said,
“Sit there and tap your head with your fingertips†“we don’t know why this works but it often does and it won’t hurt to try since you’ve exhausted all conventional possibilitiesâ€, that would be intellectually honest at the very least.
Lucia,
Probably not! But look at it another way: they’d probably be no worse than the people we already pay to do it for us!
Well done! The Amazing Randi would be impressed by this work.
I’m also reminded of the experiment described by R. A. Fisher in his “Design of Experiments” to examine the claim of a lady that she can determine by taste whether the infusion of milk came before or after the tea was poured in the cup.
An interesting twist on the experiment would have been to randomly select 3 audience members as a control. But it is all a bit sad; no need to further add to the embarassment of the children.
Mike B:
Haha! I can tell whether the milk was poured first or second. No special HSP is necessary, just sensitive olfactory function.
The reason is obvious but works assuming only that the tea is made in the ’optimum’ manner. I.e. the water is at least 96 degrees when poured over the tea leaves and the tea is left to brew for two minutes in a pre-warmed pot. In other words, if the tea is hot enough when poured into the cup, as it should be, then the milk in first method tastes different to the milk in last method. If the tea is hot, when the milk is poured in it denatures, taking on a slight cooked flavour which is not desirable in tea.
Unless you drink chai that they drink in India or any permutation of it in which case then you will like the taste of boiled milk with your tea.
Each to their own, but I reckon I can tell the difference. Same goes for tea that has been stirred with a spoon from a sugared tea or even a spoon that has been used in a coffee cup. Furthermore I can tell if the tea is made using a tea bag in the cup instead of the teapot.
When I was in the states I noticed that the coffee makers were being used to make tea, no good. The water needs to be a different temperature for coffee than tea although I love both.
I’m afraid I’ve guessed what comes next in the saga, but don’t want to spoil it for your readers. I’ll write my guess down and put it in an envelope and hold it up to my PC. Your more psychically talented readers will be able to read it. But then, they’ll already know what comes next.
While the results are convincing to those of sceptical disposition, it is unfortunate in hindsight that the results were revealed progressively, before the test was over.
I can think of an excuse: “After an unfortunate error at the first guess, the pressure and stress of performing in front of a large audience affected the children, and their powers temporarily vanished”.
To me it would not be a convincing excuse, but to someone who believed in this sort of stuff it would probably be acceptable. As such, these results probably would not change many minds.
@Joy
There is no ethical way you can knowingly administer a placebo treatment to a patient. Firstly the deception required violates patient autonomy. Secondly it validates the perception that there may be a treatment for the condition you can’t treat, which is again unethical. Thirdly, conventional treatments have a built-in placebo effect, and competent reassurance and skillful listening from a health practitioner should render the actual need for administration of a physical placebo unneccessary. Fourthly there are useful and proven treatments for managing stress, and if the person you are dealing with doesn’t like that, you are not obliged to alter your professional opinion to suit the belief system of your patient. You sjhouldn’t knowingly accept being party to treatment you know to be futile.
Mick — healthier is healthier. Who is to say what a placebo is if it works. Whether it’s voodoo or swift kick, if the treatment works it cannot be said to be non-efficacious. Non-medicinal, perhaps, but not fraudulent. Conversely, accepted medical treatments often make the patient sicker while the physician fiddles with the dosage. Outcome-based treatments are the most ethical.
The Bad Science blog has a lot of interesting discussion on this sort of thing.
http://www.badscience.net/category/placebo/
In the video clip, Dr. S.W. Lee had a calm smile and came across as a sincere individual who would like to improve people’s life. I don’t think the clip and this story can define who and how he is as a person and as a doctor.
JH,
We are not free to comment on his character, and believe me, there are a lot worse things said directly to patients by Doctors at the top of their profession. However, his description cranial/brain blood flow was ridiculous. He ought to have just done the demonstration and left it there.
Mic,
Altering professional opinion to suit the patient?
Professional opinion is always patient centred. The clinical findings, subjective and objective drive the diagnosis, not the intervention. The diagnosis does not determine the intervention either. It may only show what treatment is indicated and contra-indicated. A diagnosis is not enough on its own. There is not a recipe for every condition, apart from outlines in textbooks, and patients have rarely read the same one.
Placebo:
Read what I wrote carefully again. Placebo is not always as in a clinical drug trial, a pill, where the administrator fools the consumer. I ought to make that clear. I use this word in a broader way. Clinicians are often fooled by their own treatment, more so than the patient; in some cases the patient fools the clinician. A patient may tell you they are better for a variety of reasons. The opposite is also true. The Placebo I refer to is not just a sugar pill, which I assume is what you mean when you say “physical placebo†with all it’s Machiavellian connotations. On that subject, the following is interesting:
“It is a known fact that the shape of the tablet plays a part in the effectiveness of the drug. Transparent capsules with coloured beads work better than capsules with white beads, which work better than coloured tablets, which work better than square tablets with the corners missing, which work better than round tablets†(David Butler, “Explain pain†p17 ISBN0-9750910-0-X)
So then, why are not ALL drugs the optimum shape and colour? Is it unethical for any practitioner to administer any other shape knowing this? Of course not, if all tablets were the same, they would then be ordinary and another shape would take on the special placebo qualities for it’s new appearance.
It is not the place of any practitioner to tell a patient to stop taking part in such an activity as this yoga, for example, if the patient tells them that they derive benefit. If the behaviour is not unsafe or in any way delaying or negating some other intervention that will be effective in their case. Unfortunately, in many cases, especially of a chronic nature, there are no alternatives. The patient has reached the end of the conventional road. It is a callus individual that would tell a patient
“Don’t do that, it’s having no effect at all!†This WOULD destroy the placebo that was until now alleviating suffering of whatever kind. Conversely, if, on questioning, it is clear that the patient is wasting money and is not convinced that their yoga is helping, but is simply hoping that it will, or requested a straight forward answer, I would give my opinion, as I did about the man in the video.
A lovely lady that had undergone radical bowel surgery and chemotherapy, was told she was a miracle for her survival, had used a special diet during her road to recovery that was somewhat unproven and definitely alternative. She was particularly impressed by this diet and believed that it had had a lot to do with her successful outcome: Her Oncologist had given her short shrift about this diet. “It won’t do any good, I don’t think a lot of that, what do you want me to say?†or words to that effect, to which the intelligent lady replied,
“I wasn’t seeking your approval, I thought you ought to know.†This attitude from the Oncologist was deeply painful for this lady and is all too common an attitude from certain types of Doctor.
You see, patients will do and have the right to do what they like when they like. The Doctor is NOT in charge; of course it is as well in some cases to try and take charge, but cases rule.
Other placebos off the top of my head:
Consider the dentist when he gives the child a sticker.
Children’s plasters in fancy colours, that have characters on them.
Consider the consultant that administers a tiny amount of sugar solution to neonates who are in pain…they stop crying and give all outward signs of improved comfort. I know a consultant that did/does this. She had neurotic colleagues that told her she was destroying the child’s future teeth! My consultant friend was more concerned about their immediate and future well being…and please, of course they received all medicine that was due.
What is more important, a happy child that developed a sweet tooth from the earliest age possible, or a child that is troubled but has potentially no difficulty resisting sweets!
Adults have similar needs. Some people only understand when it comes to children. When a person reaches a certain age, they somehow believe that equivalent measures are unnecessary. Until, of course, They are the patient.
Stress:
is not a defined medical condition, but a collection of symptoms due to another problem that may or may not be due to pathology. Stress is a cause or aggravating factor.
There are no proven treatments for “stress†if you go to your Doctor and tell him “I’m stressed†he will endeavour to find out how this manifests and what might be the likely cause. Obviously, areas that might be addressed may take the form of advice, counselling or clinical investigation to check for any medical reason why the patient was feeling heightened agitation or symptoms caused by stress.
There are no magic bullets for stress! Anyone who says there is, is not in command of the facts.
“…right what chance would predict…”
This is exactly the “due to chance” idea that you often express displeasure/rant about.
Justin
Justin,
Yep, it is. It came before my enlightenment (which I subsequently blogged about, years ago).
I was born a frequentist!