STUDY DESIGN
As I mentioned last time, it is crucial to design a test that eliminates all possibility and suspicion of cheating. If the KIBS kids were going to demonstrate extraordinary powers, I had to be sure they could not use any of their ordinary powers.
Luckily, this is easy to do for this kind of alleged ability. Blindfolds were definitely out, because nobody would ever—or should ever—believe results from a blindfold test. It is just too easy to cheat.
Still, Sung said that the kids were trained using blindfolds, were comfortable with them, and would like the chance to demonstrate their talents with them. I agreed to this, for one because there was no other way—they insisted on the blindfold demonstration—but also because a contrast between a way in which it was possible to cheat and one in which it was not would strengthen the evidence if the test failed.
Here, in our original draft of the experimental protocol, is what Sung and others said about brain respiration:
Throughout history, unusual skills have been claimed to occur as a by-product of meditative training. “Brain Respiration” (BR) is a mind-body training program that has been practiced by nearly one million persons in South Korea, Japan, North and South America, and Europe since the early 1980s, as a means to maximize brain function for the purpose of peace. More recently, the curriculum of this practice has included methods for developing “heightened sensory perception” (HSP). HSP is a putative perceptual faculty that allows identification of colors, shapes, letters or other forms of information without use of vision. HSP is described as a normal but relatively neglected neural capacity present in all humans. Use of HSP is felt to facilitate emotional and moral development, as well as to assist in discriminating among competing items of mutually exclusive information. Children are said to be more easily trained in HSP because of their fewer preconceptions against the possibility of such skills, as well as their greater neuroplasticity.
A draft document was written explaining the protocol to which all members agreed. That document was authored by Sung W. Lee, M.D., M.Sc., Instructor, Weill Medical College, Cornell University; Joseph Ingelfinger, M.D., Instructor, Harvard Medical School.; William Briggs, PhD., Assistant Prof. of Biostatistics, Weill Medical College, Cornell U.; JooRi Jun, B.S., Director of Special Projects, Korea Institute of Brain Science; Ul-Soon Lee, Vice President and Chief HSP Instructor, Korea Institute of Brain Science; Chang-Su Ryu, Director of Research, Korea Institute of Brain Science; Ilchi Lee, PhD. (honorary only), President, Korea Institute of Brain Science.
Overall
The test was to be in two phases: blindfold, and opaque envelope. Three Korean-national boys, aged 10 to 15, would take part. They were flown in a week before the test to acclimate to the new time and place. None were to take part in the test unless they expressed confidence they could perform.
During the blindfold phase, the kids would be handed cards and had to guess their colors. All in the audience could see the colors of the cards. The kids could take as long as they wanted, move in any way they wanted, eat chocolates or other candy, stand up and do exercises, until they were comfortable enough to take a guess.
In the opaque envelope phase, the kids took off the blindfolds and were presented with colored cards inside sealed envelops. The kids again could move around, eat, even talk to anybody they wanted. The only stipulation was that the envelope had to remain in sight on the table. They could take as long as they wanted until they were comfortable enough to take a guess.
During both phases, a separate proctor would watch each child. I recruited fellow professor of statistics Mark Glickman from Boston University to be one proctor. Sung found two others, both physics graduate students. I was to watch the proctors, and Sung ran the experiment.
When a child announced his guess, the proctor would hold up either the card or envelop and repeat the color. For example, if the child said “blue” (they spoke English for the color names), the proctor would say, “Blue?” or “Did you say blue?” The child could either confirm or change his mind. Once the entire audience, proctor, and child were satisfied that the guess was indeed “blue”, the proctor would record the guess on a piece of paper, and then the child would be told if he was correct or the envelope would be opened and revealed to all. The true color was then recorded on paper, and the next test would begin. If the kid got the color right, it was recorded as a “hit” else it was a “miss.” By agreement, if the kid did not know the color that trial was set to a “miss”.
After all the guesses were complete, then and there we would announce the results. We did not want anybody to go away not knowing what had happened.
The KIBS trainers were in charge of fashioning and fitting the blindfolds. Neither the proctors nor I verified their opacity or integrity. The blindfold phase was explicitly meant to be a “warm up” and would not be used as the official test results. Only the opaque envelope phase of the test was to be counted towards or against proof of HSP.
Sung and I agreed that either or both of us could write or publish anything we wanted about the test. I was very clear about this because of my concern of what might happen in the test was a failure. Our agreement was a gentleman’s one—and Sung is a gentleman—so there was and is no need of him to have written out a legal document.
Envelopes phase
Each child was to be presented with 12 cards contained in opaque envelopes. The cards were various colors: yellow, red, etc. Before the test, Dr S. Lee and I went to several office supply stores in order to find adequate envelopes. I held each sample envelope up to a 60-watt bulb to ensure there was no way they could be seen through. Once we were satisfied with the choice, Dr Lee ordered a large batch from a manufacturer in New Jersey. A quantity of these were sent to Korea by Dr S. Lee for the children to practice on. I took the rest. The envelopes were 65# grey, 6×9 inches, and colored paper card stock was cut to just fit inside each envelope. The example envelopes and colored stock were sent to Korea at least three weeks prior to the experiment, and none of these samples were used for the actual experiment.
I generated a random list of cards using the R statistical software platform. It picked the numbers 1 through 6, assigned each number to a color (1 was always “red”, for example), and then listed those colors. Computer random number generators take as initial input a “seed”, a large integer that is used to commence the sequence. I wrote the code to produce the numbers but I did not know the seed.
I gave the program to my number two son and he picked a seed, ran the program and generated the list. Then he and my number one son stuffed each of the envelopes. I did not know, before or during the test, which envelope held which color. My two sons remained in New York City while I made the trip to Boston. They sealed, in a separate envelope, the randomization sheet, which matched each envelope, which were numbered, with the colors generated by the computer. I did not open this envelope until after the experiment was over.
These precautions were necessary to ensure that there was no way I could offer any visual clues, consciously or not, to the kids during the experiment. They could not look to me to see whether their “Maybe blue?” guesses were right or not. The only two people in the world who knew the contents of the envelopes were nowhere near Boston, nor did they have any communication with anybody about the contents of those envelopes.
To be clear, the randomization procedure could have produced a list of cards that was, for example, all red, or “yellow, blue, yellow, blue,…” There was no way to know in advance—or during the test—what sequence would show itself to the kids. As it turned out, the list looked “random”, that is, it presented an equal mix of all 6 colors.
Each envelope was “licked” and then liberally glue-sticked closed. My sons and I experimented with this to see if we could find a way to cheat by, say, picking the corner loose and peeking inside. Of course, we cannot guarantee that cheating was impossible, but we felt it was extremely unlikely.
I had, and have, no idea how the kids used the envelopes to practice in Korea, but I could sense surprise when the envelopes proved difficult to open, even by the proctors.
Statistics
Calculating probabilities of “hits” and “misses” in this experiment is particularly easy. Since there is a 1 in 6 chance of getting any card right just by guessing, we can then calculate the probability, over 12 trials, the probability of 0 hits, 1 hit, 2 hits, and so on up to 12. For example, the KIBS people were confident that the kids would get around 10 hits each. The probability of guessing by chance 10 or more cards right is about 8 x 10^-7, or about 1 in 1.3 million, odds sufficiently low to give evidence that brain respiration worked if the kids scored that high.
For reference, the probability of scoring at least K hits out of 12 guesses is
| K | Prob(Hits >= K | 1/6 chance) |
|---|---|
| 0 | 1 |
| 1 | 0.89 |
| 2 | 0.62 |
| 3 | 0.32 |
| 4 | 0.13 |
| 5 | 0.036 |
| 6 | 0.0079 |
| 7 | 0.0013 |
| 8 | 0.00016 |
| 9 | 1.3 x 10^-5 |
| 10 | 7.9 x 10^-7 |
| 11 | 2.8 x 10^-8 |
| 12 | 4.6 x 10^-10 |
However, do not forget that we are testing 3 kids, not just one. Imagine we are testing 1 million such kids. Do you think it would be unusual if at least 1 of those million got 10 or more hits? No. In fact, there is a 54% chance that at least 1 will score 10 or greater. Thus, we wouldn’t be surprised to see it happen. In our case, there is probability of 2.4 x 10^-6 that at least 1 kid will score 10 or greater. Still low enough that it would be surprising enough if it happened.
(To learn how to calculate these kinds of probabilities, turn to Chapters 3 and 5 of that wonderful book Breaking the Law of Averages: Real-Life Probability and Statistics in Plain English.)
Next: the day of the test
Hello, just some thoughts on aspects of the outline:
Why do you let the Experimental Group ” practice “with the cards, it will be an advantage over a Control Group, or is this group also allowed to “practice”?Ooh I see there is no Control Group, but then are there there not two variables being tested? In my view this “practicing” is asking for mischief. The kind of envelope used seems important to me, the “written card inside, 60W ” criterion was not used.
The draft of the experimental protocol reminds me of the book When the Wind Blows by James Patterson in which babies were experimented on and altered scientifically to grow wings and to develop heightened instinct abilities. And the children were all kept in a secret school.
This probably had caused a certain degree of damaging (or beneficial) effects on his HSP at once, and hence influenced the result of the next test, i.e, non-independent trials. 🙂
Were there any specific reasons why three boys were used? Why not two girls? Girls have HSP, and I suspect this was the biggest flaw in the study design… hahah.
Yes, I am trying extremely hard to find a bone (fault) inside of an egg (the design)… a Chinese idioms in honor of the coming Chinese New Year (Jan. 26).
This is the flaw:
Color is a state that only exists in the presence of light.
In opaque envelopes, color does not exist.
Wouldn’t HSP based on the color of the cards in opaque envelopes be impossible? What are the kids supposed to be picking up on in that case – a property that doesn’t currently exist? IF this card was exposed to light, what color would it be? Meaning, what wavelength of light would this material reflect, if it were exposed to light, which it currently is not? I am surprised the KIBS people agreed to opaque envelopes.
It seems completely impossible that anyone could detect that, no matter what their powers. The wavelength needs to be reflected, at the very least, for it to be true. In the blindfold test, light hits the cards, so color exists. Perhaps the kids can see the color, perhaps they can’t; at least they’d have a chance – even if that chance was based on picking up something (extra-sensory or otherwise) from the audience and judges perceiving the color at that instant.
Ultimately, I don’t really understand why they are testing for color. They should be testing for mass. The mass of an object will not change, whether its visible or not, illuminated or not, etc. It’s intrinsic, a real property of matter, with real effects on physical laws – not dependent on human perceptions. Detecting a cube of lead vs. a cube of glass, either hidden from view or both painted the same color so as to appear indistinguishable, would be a much better test of some kind of sixth sense.
Dr Briggs,
After all James Randi has written, I’m amazed you didn’t invite a magician skilled in close-up magic to help with your experimental design.
I’m happy to see that your group expects a “real” blind reader to get 10 of 12. I thought it was ridiculous for Rhyne to make so much of “better than statistical expectation.” We expect an unblinded reader 10 years of age to get colors 100%. If a blind reader is real, we should actually be more interested in his misses!
Regards,
Bill Drissel
PS How did anyone come to believe in this effect in the first place?
JH,
Good point, Briggs knew it would fail if he picked boys.
but seriously,
Citizen,
For the idea about the colour to be true, one must assume that there is some physical mechanism that links the differing wavelength of light with some exact part of the body. No such wave-length discrimination is known to exist. If one did, then blind children would use this to sort their socks! However I can confirm that as we speak there are blind children all over the world wearing odd socks, some care and some have to be made to care in order to fit into the world around them. Your point is a philosophical one. To say that the envelopes would cause the experiment to fail is to bring in another unproven dimension of physical possibility or to say, without proof that envelopes block the hsp pathway. Maybe something that a believer might say, or hope.
The body reacts to light, artificial and natural, however there is one organ that is able to distinguish wavelength and bring this to a conscious level, the eye.
Thanks, Joy.
I wonder if PF lens can see through the envelops.
Why were three boys, aged from 10 to 15, chosen? They were probably the best three kids. Why weren’t there any girls? It just occurred to me that it probably had something to do with cheating.
1) Boys were more often guilty than girls.
2) Cheating behaviors are well-established by high school.
When I was a youngster I gave chess lessons to small children, I let them sit on their hands and told them to think with their brain. Please disregard my previous comment, I still have doubts concerning the opaqueness of the envelopes, but the design seems fine.
JH,
They only needed three, They could have used one of each!
Citizen – GREAT point (from one physics grad to another I am guessing).
Joy, you seem to miss Citizens point. The test was to determine if these children could determine the colour of something without using their eyes after having received “special” training to do this.
If their body somehow detected colour in another way, then this test would fail because there was no colour being emitted for them to detect. Under this situation EVEN IF they had some weird ability they would fail.
I have no doubt they were cheating anyway.
Anon:
No, there are two ends to the interaction of the colour and the colour detection. The wavelength solution is implied by the fact that what is being blocked is in fact the light. Whether this is done with a proper blind fold or by an opaque envelope. Sensation light detection is but one possibility. Citizen’s point assumes that it is the visual light spectrum doing the magic (or that this is why the experiment is flawed). Your idea about light being absent inside the envelope is one aspect of the origin of the colour. What property makes a green card green and a pink one pink? Clearly colour reflection is a function of the pigment or chemical that does the reflecting or absorbing. I.e. the property that makes a card green (out of the range of cards in question) is the same regardless of whether there is any light shining on it. Imagine chlorophyll in a leaf. Is it’s only property that it absorbs certain wavelengths and reflects others? Hence my point that the idea of the visible light wavelength being relevant is simply one explanation that is a “get out†without any evidence that the wavelength can, in fact, be detected by a body part other than the eyes.
Think “vibesâ€, â€vibrationsâ€, “smellâ€, touch, (special ability to feel different pigment), perspiration from fingertips mixed with pigment, entering the body and being detected by taste centres in the brain; or any other made up word or mechanism you could come up with….â€HSPâ€?
Of course I think the whole idea is nonsense anyway but if someone is claiming HSP without any explanation of how, then they are free to claim any reason for failure in the event that they are shown not to have the special power.
Again, blindness is not a new phenominon, if such power were possible, blind people would developit as children and use it.
This Itchy man is a charlatan, wasting people’s time, money and using children in the process.
Reminds me of Fagin.
JH,
What is a PF lens?
Joy,
Selection of samples and randomization of assignments are fundamental questions in the design of an experiment. They can be complicated.
A PF lens supposedly allows a camera man to see through someone”s clothes. I have no idea if this is true.
I don’t believe an ordinarily reasonable person, when asked, “What is the colour of the card in this envelope?” would answer, “Since it is unilluminated it has no colour”. This would also lead to the absurdity that we can never check the assertion that it has no colour since we cannot inspect it without illuminating it.
The KIBS people and the kids clearly interpreted the question in the everyday sense so that was the test they agreed to.
Not that I mind philosophical pedantry particularly …
I really don’t think it’s pedantry. Just because the kids think they have the ability, doesn’t mean they understand how it works. Just because they agreed to the envelopes doesn’t mean they understood the implications.
My argument isn’t about whether the kids have the ability. It isn’t about whether or not the eye is the only sense we have for detecting light. It isn’t if this was/wasn’t the test the kids agreed to. My argument is: this was not a valid test of a non-visual faculty for detecting color.
Color, as we know it, is a reflected wavelength of light. If the light isn’t reflecting, that color does not exist.
To say the kids should be able to predict the “color” of something that at the moment has NO color, i.e. what wavelength of light a substance would reflect, if light were to hit it – detecting the ‘property’ that causes it to reflect that wavelength – is something quite different than saying they can sense color through non-visual means. Something that, intuitively, seems much more obscure: to detect the potential for reflecting light of an arrangement of the molecules on the surface of the card (the ink or dye) as distinct from the molecules of the card itself.
As opposed to: The wavelength of light reflecting off the card at the moment of detection.
I’m not saying that these kids had powers. I’m saying that to test someone’s ability to detect color, in the absence of any color, is flawed. Just as detecting color in a pitch-black room would be, or the ‘color’ of an object in an opaque envelope. Color is not an intrinsic quality. A ‘blue’ object will change color based on the type of light being reflected – for example, under a red light. It’s kind of a mistake to say something ‘has’ a color, objectively. But you can say that at whatever moment, it is reflecting a certain wavelength. It is my impression that the reflected wavelength is what the kids claimed to be “detecting” during the blindfold test.
whoops, sorry, the comment didn’t show up the first time.
I think Citizen makes a good point, except that in this situation, Ilchi Lee claimed that the children could identify the colors of the cards inside the envelopes, and that they had succeeded in doing this using the same batch of envelopes when they practiced before the experiment. In other words, the kids were said to be capable of something along the lines of psychic ability, knowing what color the cards would be if illuminated. (He also has claimed that his BR trained kids could read pages through the covers of books and make silverware stick to their faces using psychic energy.)
Also, Dr. Briggs allowed, even if he did feel some reservations about it, the second blindfold trials to proceed, in silence, testing his hypothesis that the children were getting verbal cues from others in the room. Lo and behold, the blindfolded kids weren’t able to identify the colors even when the cards were illuminated.
Ilchi Lee is a significant figure, especially in Korea, but in the US and other countries too. His KIBS, a so-called brain institute, seems to be highly regarded by the Korean government, as is Brain Respiration, Brain Education, Dahn Yoga, and their HSP claims. KIBS was accepted by the UN for roster consultative status based on their purported ability to create peaceful brains among the masses with their programs. The reach of this spiritual leader and purported brain scientist is large and growing!
Even as Dr. Briggs couldn’t persuade his friend and colleague with the evidence, or lack thereof, I think he helps some potential Dahn followers to question the paranormal claims of a charismatic leader of what many report to be a manipulative cult that uses its Dahn Yoga centers and university clubs as outreach. Thanks, Dr. Briggs, and sorry you lost your friend over this. It was inevitable, I think.
“10 7.9 x 10^-7
11 2.8 x 10^-8
12 4.6 x 10^-10
However, do not forget that we are testing 3 kids, not just one. Imagine we are testing 1 million such kids. Do you think it would be unusual if at least 1 of those million got 10 or more hits? No. In fact, there is a 54% chance that at least 1 will score 10 or greater. Thus, we wouldn’t be surprised to see it happen. In our case, there is probability of 2.4 x 10^-6 that at least 1 kid will score 10 or greater. Still low enough that it would be surprising enough if it happened.”
Congrats. You demonstrated that concepts of alpha and p-value work.
Justin
“I generated a random list of cards using the R statistical software platform. It picked the numbers 1 through 6, assigned each number to a color (1 was always “red”, for example), and then listed those colors. Computer random number generators take as initial input a “seed”, a large integer that is used to commence the sequence. I wrote the code to produce the numbers but I did not know the seed.
I gave the program to my number two son and he picked a seed, ran the program and generated the list.”
Also, a good demo that (pseudo)random numbers work.
In short, in this ‘testing Korean HSP claims’ series, you’ve demonstrated the efficacy of the things you frequently write against:
-random numbers (pseudorandom)
-alpha (essentially the bar for the claimant to pass their claim)
-p-value (essentially the test statistic)
Your use of them made sense though and demonstrated their effectiveness, such as Randi had in similar tests, like of ESP, dowsers, etc. Of course, a well-designed experiment(s) is first and foremost.
Or is your ‘wee p’ and alpha here valid but those of us using them in similar tests is not valid?
Justin
Justin,
Yep, you got it. Never use p-values. This was all done long before my road to Damascus.