There is a new Research Shows paper on whether drugs like Ozempic (semaglutide) lower cardiac muscle mass. It makes a good lesson in how to read statistical claims. The paper is “Semaglutide Reduces Cardiomyocyte Size and Cardiac Mass in Lean and Obese Mice” by Martens and others in JACC: Back to Translational Science. It’s on line and you can follow along.
All their graphics are crammed into their “Figure 1”. I only reproduce the most relevant here. Here is a video discussion that walks through all the pics:
Our first clue is the paper itself is thin—a hilarious pun. Meaning that if we were trying to reproduce the results, we’d have to do a lot of guessing about the experiment itself. In mice.
The idea was simple. The fed male mice, not people, a high sugar, high fat diet and fattened them up over time. Then they fed them regular mice chow, with half receiving something like a placebo (control or “vehicle”) and half semaglutide. All the mice lost some weight, they said, because they took all off the high fat diet.
Here they lost an opportunity, only showing us body weight and other measures at the end of the experiment. It would have been better to show us three points for each mouse: beginning, after the fattening diet, and after their treatment (vehicle or semaglutide). We don’t know where any mouse started, and it could be the fatter ones were in the control group. They say the diet wouldn’t affect mice hearts, but how do we know? It we could see the before, during, and after all at once, we could judge for ourselves.
In Figure 1’s A(ii) and A(iii) it certainly seems like more mice had less body mass, though some had more, and less fat mass, though some had more, in the semaglutide group. But we don’t know where they started from.
This failing appears, or rather doesn’t appear, in any of the graphs, in any of the measures. This means that we should be less certain of the results because we never know the proper baseline. This is not to say the authors are wrong, but we don’t know they’re right.
They say things like this, “No differences in left ventricular (LV) mass between groups were observed.” Read that again while looking at A(iv). It looks like there were differences in Ejection Fraction (more is better, up to a point, after which more is worse; in people this point is around 70-75%; I have no idea what it is in mice). The peculiar way of speaking, i.e. that “No differences”, is in reference to a “hypothesis test.” It means they didn’t get a wee P-value. Which is meaningless in the real world. As I have been trying to teach us in the Class.
We don’t need any of those damned asterisks, or P-values. We have the data, which shows what happened. Whether we can infer the cause, or causes behind that data, is entirely independent of the P-values. ALWAYS keep this in mind when reading ANY Research Shows paper. They will speak in that funny way “we found differences” or “we found no differences” are always with respect to hypothesis tests. And often they won’t show the data, and only give the hypothesis test! Which is another reason we have to kill hypothesis testing. It leads to wicked over-certainty.
Now the real meat of the study—two gorgeous puns in one post—comes in the claim there is less meat in the heart for semaglutide-eating mice (which they normalized by mouse size, using tibia length as the guage; larger mice have bigger hearts). See A(vi) and A(vii). Although we don’t know where any mouse started, it looks like most semaglutide mice have lower LV mass and that many have lower heart weight.
These measures were determined by echocardiography. Meaning there is some plus-or-minus associated with these, but they are only shown as certain dots. They didn’t kill the mice and do exact measures. This opens—a third pun!—the possibility of bias, because there doesn’t appear to be blinding. It’s all too easy to see what you want to see. I do not claim there is bias, but there might be, and we have to add this to the “fuzz” around the results.
They also looked at A(ix) cardiomyocte area: these are the contracting muscles. Here a picture with overlapping histograms would be easier to interpret, instead of squat blobs. You’d be surprised how easy it is to fool yourself when looking at blobs. They found similar results in human cells, but these were in (so to speak) petri dishes, not in bodies, and they were force fed semaglutide, which wouldn’t happen in real life.
To check, they look at expressions of some genes “associated with” these things, and said they found “no difference”. But we look and see real differences, with the semaglutide mice having the highest expression values in the fibrosis genes (this is when connective tissue replaces damaged normal cells, leading to lowered ability to squeeze).
What’s odd is that they repeated the experiment in already lean mice, and these expression level maxes largely switched to the control group! Is this acausal variability? Or does it mean something? I don’t know.
Their conclusion: “Overall, these data show that semaglutide has an effect on the cardiomyocyte resulting in reduced cardiomyocyte size and cardiac mass in lean and obese mice…” There’s that dreary exaggerated hypothesis-testing language again. How much better to say something like, maybe, “Given this data and protocol, x% of mice had y% smaller hearts when fed semaglutide” or whatever. Or make that into a probability-prediction for future populations. In mice.
There’s more in the video discussion.
Subscribe or donate to support this site and its wholly independent host using credit card click here. Or use the paid subscription at Substack. Cash App: \$WilliamMBriggs. For Zelle, use my email: matt@wmbriggs.com, and please include yours so I know who to thank. BUY ME A COFFEE
