When I was a kid they had these things called drinking fountains in public places. Schools had ’em, too. They stood about yea-high. You’d push a button and water would come out which you could drink. This was before scientists figured out how to sell plastic-encapsulated water at enormously inflated rates. Progress.
There’s been so much progress that it’s hard to find water fountains these days. The scarcity got some scientists thinking: what would happen if water were available to school kids? And not just water. Water jets. Yes, water which has been “electrically cooled [in] large clear jugs with a push lever for fast dispensing”! Which sounds even more expensive than bottled water.
Enter Amy Ellen Schwartz and three others who give us the peer-reviewed study “Effect of a School-Based Water Intervention on Child Body Mass Index and Obesity” in JAMA Pediatrics, in which we learn that, scientifically speaking, “Water is essential for human function”. This is why “Previous studies found providing water to students in schools may be beneficial.” May. Scientists are known for their caution.
What Schwartz et al. did was to look at the age-sex normalized BMIs of about a million New York school kids (the government helpfully tracks this none-of-their-business information). If and on what date each kid’s school got a $1,000 water jet was noted. They also found out how much milk was sold at each school’s cafeteria (chocolate or plain was also noted).
Here comes the science: “We identified a student as being ‘treated’ by a water jet if he or she spent 60 or more cumulative school days in a school with a water jet.”
In plain English, this translates to: “We have no idea how much water any kid drank. But we like to say water jets. Try it. Water jets. It’s fun and expensive! And it’s scientific.”
The science didn’t end there. They also scrutinized “changes in milk purchases between schools that had a water jet and schools that did not have a water jet, before and after introduction of a water jet.”
At this point, Schwartz and pals could have noted the differences in BMIs and milk purchases between water jet and non-water jet kids. But no. Instead, they stuck all the data inside a series of regression models. Because regression models are more scientific than just looking at numbers.
Turns out by “policy design, water jets in New York increased over time during our analysis period, and machines were often placed in schools in waves”. Sounds like somebody’s brother-in-law is in the water jet business, no? Skip it.
The shocking main results. “There was a significant main effect of water jets on zBMI, such that the adoption of water jets was associated with a 0.025 (95% CI, ?0.038 to ?0.011) reduction of zBMI for boys and a 0.022 (95% CI, ?0.035 to ?0.008) reduction for girls (P?<?.01).” Wee p-values!
Did you see that? A whopping 0.025 reduction in age-sex normalized BMI for boys. And a stunning 0.022 cliff dive for girls. Scientifically speaking, these differences fall into the Preposterously Trivial classification, which is one step above What Do I Look Like, An Idiot? level. (The same kind of finding for milk was presented.)
Then again, since these are actually parameter estimates from the series of regression models and not actual reductions, Preposterously Trivial is probably an overstatement. And since nobody has any idea whatsoever how much water any kid drank, even What Do I Look Like, An Idiot? is too high. We’re probably much closer to I Weep For The Future Of Humanity classification.
Let’s see how Schwartz describe their success: “Analyses revealed that zBMI and overweight decreased significantly for boys, and zBMI and overweight also decreased significantly for girls.” Note that, despite appearances, “significantly” has no connection to any English word: it is merely to note the p-values were smaller than the magic number. The authors also note that “some children might not like white milk.”
A “potential mechanism” to explain the dramatic changes was that, possibly, “students consumed fewer [sugar-sweetened beverages] brought from outside school”. But the authors have no idea, because they have zero clue what any kid drank.
What we have here is the epidemiologist fallacy in all its glory, married to misused and misunderstood statistical models. The epidemiologist fallacy is when a scientists says X causes Y but where X was never measured. Here X = water jets, and Y is the stupid small change in BMI, and even that was exaggerated by the statistical model.
Just to show you how harmful this fallacy is, one newspaper said the “study found making water available through self-serve dispensers in school cafeterias results in student weight loss.” Sheesh.
This is an official entrant for 2016’s WMBriggs.com Bad Science of the Year Award.