[ad_1]
Most nutrition research studies are designed to see what will happen if we change something.
- What happens to muscle synthesis if we add more protein to the diet?
- What happens to cholesterol levels if we increase vitamin E intake?
- What happens to blood sugar levels if we decrease carbohydrate intake?
- What happens to immune function if we add more vitamin D?
Sometimes, of course, nothing happens. We often refer to that as a null result. And it’s easy to see a null result as a failed experiment. But actually, it’s not.
Joining me today to talk about null results is Greg Lopez. Greg is the lead scientific editor at examine.com, where he and his team collect, assess, and summarize a staggering amount of nutrition research. Below are some highlights from our conversation but please click on the audio player to hear all the juicy details.
Monica Reinagel: Examine.com is one of my go-to resources for nutrition research. Not long ago, you added a new section to the Nutrition Examination Research Digest (affectionately known as NERD), dedicated to reporting null results—or research where nothing happens. Why did you feel that this was important to shine a light on?
If there are strong null results, then people can avoid wasting their time and money on ineffective nutrition and supplementation.
Greg Lopez: People are turning to us in order to find out what works and what doesn’t. And if there are strong null results, then people can avoid wasting their time and money on ineffective nutrition and supplementation.
MR: You mention the value that this might have for consumers who are trying to decide whether or not to try something. But I would think that for researchers, it would also be important to know about null results that have already been found so that they can focus their research attention more effectively.
GL: Exactly. It’s a big deal for researchers for a few reasons. When you have a whole bunch of small studies, you may see some null results because the studies are too small to catch small-size effects. But if you put all these results together into a meta-analysis, you can see more subtle effects. But if only positive results are published, then you’re getting a biased snapshot of what the literature is actually saying. So meta-analysis conclusions won’t be as reliable as they would be if more null results were published.
Ruling out ideas is how science works. If you already knew what the result was going to be, then you wouldn’t need to do the experiment!
Plus, there’s a lot of interesting hypotheses that are being tested in nutrition and supplementation, and ruling out those hypotheses can actually push basic research and translational research forward. Ruling out ideas is how science works. If you already knew what the result was going to be, then you wouldn’t need to do the experiment!
MR: You mentioned a publication bias where results that have a positive finding are more likely to end up published. Why is it that research journals are less interested in publishing these null results? Especially because, as you say, not publishing them really gives us a skewed idea of what we know.
GL: Results that are null could be seen as failures or as a waste of research dollars. At the end of the day, I think that it is probably a mix of prestige and bringing in the research dollars. You want to make sure that you can show steady scientific progress by publishing positive findings.
MR: It’s interesting that we use the term “negative results” and “null results” interchangeably. But is that accurate? A null result shows that an intervention did not have a significant effect. Well, that’s not a negative result. Right?
GL: It is a strike against the hypothesis that you’re testing. But the null hypothesis doesn’t necessarily mean nothing. You can set it to be whatever you want it to be. So for instance, say you’re looking at weight loss. Both the control group and the intervention group lose two kilograms over the course of a month. That could be a null result even though both groups lost two kilograms. Because there’s no difference between the groups.
A null hypothesis doesn’t necessarily mean nothing; it means ‘no different.’
Or maybe you just want to observe people on a specific diet and their weight loss over time. You can state, in your null hypothesis, how much would matter to you, and if you don’t reach that threshold, call that no different. So a null hypothesis doesn’t necessarily mean nothing; it means “no different.” And what “no difference” means is embedded in what the null hypothesis is.
MR: As you mentioned a little bit earlier, we are supposed to state what we’re looking for, what we’re going to consider to be significant, what we’re measuring at the beginning of the study. But sometimes people go back and tweak those retrospectively to make their results a little bit more interesting. That’s kind of breaking the rules, isn’t it?
GL: I have a strong respect for exploratory research. And if researchers start out saying “we don’t know what’s going on, we’re going to test a whole bunch of things with this intervention and see what little signals we find”—if they’re upfront about it, then explore away. The problem is that a lot of people treat exploratory research as if it’s going to confirm a specific hypothesis. And that is harder to do when you test a whole bunch of hypotheses, especially with small samples.
ND: When we see an initial small study that gets a positive result, the next thing we always say is “we need more research to replicate this result before we can really have a lot of confidence in it.” Should null results also be replicated?
A bunch of null results in a bunch of small studies, as long as they’re all similar enough to each other, can provide more and more evidence that there’s not much going on.
GL: Yes, for a couple of reasons. We’ve already mentioned that, especially in small studies, it is hard to see a signal. And so if you get a null result once, replication can help confirm it. A whole bunch of null results in a whole bunch of small studies, as long as they’re all similar enough to each other, can provide more and more evidence that there’s not much going on there.
There was an interesting example I recently came across when I was taking a look at vitamin D supplementation and its effect on colorectal cancer. There was a meta-analysis that took five studies, four of which found no effect on colorectal cancer mortality, and one which did. You could say, well, four studies came up null and one study came up positive. So it probably doesn’t work.
But when you put all of those together, they did find that vitamin D supplementation did reduce death from colorectal cancer. The four null results, when you combine them together, gave enough information to give that signal. And that’s a second reason why it could be useful.
MR: I want to circle back to why null results can be tricky to interpret. Can you talk about that a little bit more?
GL: There’s a lot going into interpreting whether a result is really null or not. In late 2020, we looked at a secondary analysis of the VITAL trial, which is a big trial that took a look at vitamin D and Omega-3 supplementation. And they found that there was no effect on depression. But the population was mostly vitamin D sufficient. But these null results might not apply to people who are deficient in vitamin D. So you just can’t say there’s no effect, full stop. You have to take a look at the population.
There was another study that looked at reducing entrée portion size and seeing whether, if you shrink lunch by 25%, will people eat more dessert? And this came up with a null result. Their conclusion was that entrée portion size does not affect a dessert intake. But the test meal was low in protein, and they had a highly palatable dessert. So maybe in these circumstances, entrée portion size didn’t affect dessert intake. But does that apply to entrées with different macro distributions or different desserts? Null results in one particular circumstance may not generalize to other circumstances.
MR: Now, the fault is not necessarily in a research design that only tested a particular meal and particular type of dessert. It’s just the obligation of those of us who are translating and reporting that research to make sure that we supply that context around the conclusions. So exactly as you just illustrated, we’re not over-generalizing a result, whether it’s a positive result or a null result.
GL: Sometimes limited scope is extremely important. If you want to see specific metabolic differences, for instance, in ketogenic versus high carbohydrate diets, then you put people in a metabolic chamber and measure their food precisely. That is not the real world, but you have to have that strict control to see exactly what’s going on and see any small differences that you can see. All of these kinds of methods have their place and all of us have a responsibility in order to make sure that they’re interpreted properly.
[ad_2]
