January 5, 2009 | 16 Comments
As I mentioned last time, it is crucial to design a test that eliminates all possibility and suspicion of cheating. If the KIBS kids were going to demonstrate extraordinary powers, I had to be sure they could not use any of their ordinary powers.
Luckily, this is easy to do for this kind of alleged ability. Blindfolds were definitely out, because nobody would ever—or should ever—believe results from a blindfold test. It is just too easy to cheat.
Still, Sung said that the kids were trained using blindfolds, were comfortable with them, and would like the chance to demonstrate their talents with them. I agreed to this, for one because there was no other way—they insisted on the blindfold demonstration—but also because a contrast between a way in which it was possible to cheat and one in which it was not would strengthen the evidence if the test failed.
Here, in our original draft of the experimental protocol, is what Sung and others said about brain respiration:
Throughout history, unusual skills have been claimed to occur as a by-product of meditative training. “Brain Respiration” (BR) is a mind-body training program that has been practiced by nearly one million persons in South Korea, Japan, North and South America, and Europe since the early 1980s, as a means to maximize brain function for the purpose of peace. More recently, the curriculum of this practice has included methods for developing “heightened sensory perception” (HSP). HSP is a putative perceptual faculty that allows identification of colors, shapes, letters or other forms of information without use of vision. HSP is described as a normal but relatively neglected neural capacity present in all humans. Use of HSP is felt to facilitate emotional and moral development, as well as to assist in discriminating among competing items of mutually exclusive information. Children are said to be more easily trained in HSP because of their fewer preconceptions against the possibility of such skills, as well as their greater neuroplasticity.
A draft document was written explaining the protocol to which all members agreed. That document was authored by Sung W. Lee, M.D., M.Sc., Instructor, Weill Medical College, Cornell University; Joseph Ingelfinger, M.D., Instructor, Harvard Medical School.; William Briggs, PhD., Assistant Prof. of Biostatistics, Weill Medical College, Cornell U.; JooRi Jun, B.S., Director of Special Projects, Korea Institute of Brain Science; Ul-Soon Lee, Vice President and Chief HSP Instructor, Korea Institute of Brain Science; Chang-Su Ryu, Director of Research, Korea Institute of Brain Science; Ilchi Lee, PhD. (honorary only), President, Korea Institute of Brain Science.
The test was to be in two phases: blindfold, and opaque envelope. Three Korean-national boys, aged 10 to 15, would take part. They were flown in a week before the test to acclimate to the new time and place. None were to take part in the test unless they expressed confidence they could perform.
During the blindfold phase, the kids would be handed cards and had to guess their colors. All in the audience could see the colors of the cards. The kids could take as long as they wanted, move in any way they wanted, eat chocolates or other candy, stand up and do exercises, until they were comfortable enough to take a guess.
In the opaque envelope phase, the kids took off the blindfolds and were presented with colored cards inside sealed envelops. The kids again could move around, eat, even talk to anybody they wanted. The only stipulation was that the envelope had to remain in sight on the table. They could take as long as they wanted until they were comfortable enough to take a guess.
During both phases, a separate proctor would watch each child. I recruited fellow professor of statistics Mark Glickman from Boston University to be one proctor. Sung found two others, both physics graduate students. I was to watch the proctors, and Sung ran the experiment.
When a child announced his guess, the proctor would hold up either the card or envelop and repeat the color. For example, if the child said “blue” (they spoke English for the color names), the proctor would say, “Blue?” or “Did you say blue?” The child could either confirm or change his mind. Once the entire audience, proctor, and child were satisfied that the guess was indeed “blue”, the proctor would record the guess on a piece of paper, and then the child would be told if he was correct or the envelope would be opened and revealed to all. The true color was then recorded on paper, and the next test would begin. If the kid got the color right, it was recorded as a “hit” else it was a “miss.” By agreement, if the kid did not know the color that trial was set to a “miss”.
After all the guesses were complete, then and there we would announce the results. We did not want anybody to go away not knowing what had happened.
The KIBS trainers were in charge of fashioning and fitting the blindfolds. Neither the proctors nor I verified their opacity or integrity. The blindfold phase was explicitly meant to be a “warm up” and would not be used as the official test results. Only the opaque envelope phase of the test was to be counted towards or against proof of HSP.
Sung and I agreed that either or both of us could write or publish anything we wanted about the test. I was very clear about this because of my concern of what might happen in the test was a failure. Our agreement was a gentleman’s one—and Sung is a gentleman—so there was and is no need of him to have written out a legal document.
Each child was to be presented with 12 cards contained in opaque envelopes. The cards were various colors: yellow, red, etc. Before the test, Dr S. Lee and I went to several office supply stores in order to find adequate envelopes. I held each sample envelope up to a 60-watt bulb to ensure there was no way they could be seen through. Once we were satisfied with the choice, Dr Lee ordered a large batch from a manufacturer in New Jersey. A quantity of these were sent to Korea by Dr S. Lee for the children to practice on. I took the rest. The envelopes were 65# grey, 6×9 inches, and colored paper card stock was cut to just fit inside each envelope. The example envelopes and colored stock were sent to Korea at least three weeks prior to the experiment, and none of these samples were used for the actual experiment.
I generated a random list of cards using the R statistical software platform. It picked the numbers 1 through 6, assigned each number to a color (1 was always “red”, for example), and then listed those colors. Computer random number generators take as initial input a “seed”, a large integer that is used to commence the sequence. I wrote the code to produce the numbers but I did not know the seed.
I gave the program to my number two son and he picked a seed, ran the program and generated the list. Then he and my number one son stuffed each of the envelopes. I did not know, before or during the test, which envelope held which color. My two sons remained in New York City while I made the trip to Boston. They sealed, in a separate envelope, the randomization sheet, which matched each envelope, which were numbered, with the colors generated by the computer. I did not open this envelope until after the experiment was over.
These precautions were necessary to ensure that there was no way I could offer any visual clues, consciously or not, to the kids during the experiment. They could not look to me to see whether their “Maybe blue?” guesses were right or not. The only two people in the world who knew the contents of the envelopes were nowhere near Boston, nor did they have any communication with anybody about the contents of those envelopes.
To be clear, the randomization procedure could have produced a list of cards that was, for example, all red, or “yellow, blue, yellow, blue,…” There was no way to know in advance—or during the test—what sequence would show itself to the kids. As it turned out, the list looked “random”, that is, it presented an equal mix of all 6 colors.
Each envelope was “licked” and then liberally glue-sticked closed. My sons and I experimented with this to see if we could find a way to cheat by, say, picking the corner loose and peeking inside. Of course, we cannot guarantee that cheating was impossible, but we felt it was extremely unlikely.
I had, and have, no idea how the kids used the envelopes to practice in Korea, but I could sense surprise when the envelopes proved difficult to open, even by the proctors.
Calculating probabilities of “hits” and “misses” in this experiment is particularly easy. Since there is a 1 in 6 chance of getting any card right just by guessing, we can then calculate the probability, over 12 trials, the probability of 0 hits, 1 hit, 2 hits, and so on up to 12. For example, the KIBS people were confident that the kids would get around 10 hits each. The probability of guessing by chance 10 or more cards right is about 8 x 10^-7, or about 1 in 1.3 million, odds sufficiently low to give evidence that brain respiration worked if the kids scored that high.
For reference, the probability of scoring at least K hits out of 12 guesses is
||Prob(Hits >= K | 1/6 chance)
||1.3 x 10^-5
||7.9 x 10^-7
||2.8 x 10^-8
||4.6 x 10^-10
However, do not forget that we are testing 3 kids, not just one. Imagine we are testing 1 million such kids. Do you think it would be unusual if at least 1 of those million got 10 or more hits? No. In fact, there is a 54% chance that at least 1 will score 10 or greater. Thus, we wouldn’t be surprised to see it happen. In our case, there is probability of 2.4 x 10^-6 that at least 1 kid will score 10 or greater. Still low enough that it would be surprising enough if it happened.
(To learn how to calculate these kinds of probabilities, turn to Chapters 3 and 5 of that wonderful book Breaking the Law of Averages: Real-Life Probability and Statistics in Plain English.)
Next: the day of the test
Part I, II, III, IV, V