If you’re a scientist, soft or hard, who routinely uses statistics, it’s likely that your funding, and therefore your career, the very wellness of your being hinges on discovering statistically significant results. Small p-values, that is.
Sometimes, however, the data just won’t cooperate. You conjured a theory that just must be true, set up your experiment, and collected your data. You ran your test and—drat!—a p-value larger than the publishable limit!
Do you despair? Do you, God forbid, abandon your oh-so-beautiful theory? Do you chuck it all and even change your mind?
No, sir! Friend, I’m here to tell you there is hope. There is a cure for what ails you. There is sunshine waiting. Introducing the Sample Size Extender™, the only frequentist statistical procedure guaranteed—did he say guaranteed?—to produce a publishable p-value each and every time.
As long as your funding holds the Sample Size Extender™ will provide you a small p-value or we will double your money back.
The Sample Size Extender™ is so easy a child could use it. Here’s how it works. All you need is a theory: any will do, the wilder the better. Begin collecting your data, put them through the works one at a time, then just wait for the publishable p-value, which is sure to come.
Here’s an example. Your theory supposes that a certain measure will be different between two groups. You’ll confirm this using a t-test—though any test will do. You begin by measuring two people, one from each group1. Put the measurements of this pair into the t-test and check whether your p-value is less than the magic number. If it is: stop! Your theory is proven. Go forth and write your paper.
But if the p-value is not small enough, measure two more people, one more from each group. Then run the augmented sample through the t-test and check for a publishable p-value again.
Iterate the Sample Size Extender™ and you will always—absolutely always—find a p-value less than the magic number. Yes, sir, friends: this method is foolproof. Fools prove it every day!
Now I know you’re doubtful, neighbor. I know you don’t believe. Why you should trust old Honest Matt? He’s trying to sell you something! Friend, I’m not asking you to believe. I’m not asking for your trust. I want you to convince yourself. I want you to see the truth with your own eyes.
Look and behold!
The picture is the Sample Size Extender™ in action. What do you suppose would happen if you grab two numbers from the air, numbers which are fiction, which are real as bigfoot, numbers which have no relation to one another? Would a t-test based on these numbers give you a small p-value? It might, friend, it might. But not often. It’s supposed to only happen one out of every twenty attempts.
But with the Sample Size Extender™ it can always happen.
What I did was to grab numbers from the air, one pair at a time and run them through a t-test. If the first pair through the t-test gave joy, then I stopped. If not, then I added another pair and checked again. I kept doing this until I got a publishable p-value, and then I noted how many pairs it took. That’s the sample size.
Then I did the whole thing over. I started with one pair, then two, and so on. I again noted how many pairs (up to 1000) it took to get a publishable p-value. I did the whole procedure 500 times and plotted up the results.
Over 10% of the time it only took five or fewer pairs to prove my theory—which is no theory at all! Remember these are entirely made up numbers! How much easier will it be, friend, to prove your theory which must be true!
Talk about simple, neighbor. Talk about progress. Talk about limitless possibilities! Why, 20% of the time, it took only ten or fewer pairs to prove Nothing. Forty-percent of the time it only takes less than 100 pairs. How little is needed to do so much!
This is statistics, friend. This is what it’s all about. No need to use that tired old phrase “More research is needed” when research each and every time will prove what you want—but only if you use the Sample Size Extender™.
The sharp-eyed among you will note the strangeness at the end of the graph, the spike at 1000. Well, friends, this is where I tuckered out. I could have gone on generating pairs beyond 1000 and—you’ll have to trust me now—eventually these experiments would all give me a small p-value. You’ll always get one, as long as you can keep taking measurements.
What’s that I hear? You can’t afford to take measurements indefinitely? Are you sure you can’t get a bigger grant? No? Then let me tell you about the miracle of our patented Sub-Group Analyzer™…
1Yes, sourpuss, technically you need to start with four people for a t-test. Yet another reason to make the switch to Bayes!
This post was inspired by reader Andrew Kennett and his link to the article Re-examining Significant Research: The Problem of False-Positives .