Quick Guide to Evaluating Weight Science Research

This is the Weight and Healthcare newsletter! If you like what you are reading, please consider subscribing and/or sharing!

If you are looking at a study (or an article about a study) that is making claims about weight, weight loss, and/or health, here are some questions to ask.

Note that this is by no means an exhaustive list, but just a place to start. (Feel free to add other questions in the comments!) Also, I’m going to use “study” as shorthand for weight-science publications including clinical trials, research, reports, reviews, expert opinions etc.

QUESTIONS FOR ALL STUDIES

How do I find the original source?

If the article you are reading doesn’t provide a link to the study or the study name, you can look for searchable clues. Often they have a quote from the lead author or one of the study authors. If the article says it’s a recent study you can look at the year of the article. They may offer the name of the journal where it was published. Put together all the clues you can find and then search the articles that come up for the right one.

Where was this published?

The first question I ask is if the publication is peer-reviewed. Lack of peer review doesn’t necessarily mean a poor study, but it’s a reason for scrutiny. Peer review is seen as the gold standard (meaning that proper research methods were used and that you can, for example, trust that the conclusions match the data) but in my experience, it simply doesn’t prove true for weight science pieces which often flagrantly fail to meet the basic requirements of research methods, and still get published. Lucy Aphramor has a great paper about this here.

Who funded this?

Funding by a company (or organization representing those) with a profit interest in the study outcome doesn’t automatically prove bias, but it’s a good reason to give the study methods and conclusions further scrutiny.

Whose payroll are the researchers on?

Again, this doesn’t prove bias but it’s a question worth asking. Sometimes it’s available in the disclosures, but I’ve seen some nonsense where that is concerned (heads of commercial diet programs claiming no conflicts of interest in diet studies, for example.) Sometimes a little digging will show that all of the researchers own/run weight loss clinics or programs and/or that their entire livelihoods and careers are tied to the idea of weight-loss as healthcare. It’s worth knowing if that’s their perspective.

There’s also the question of what companies/interests they are taking payments from for things like consulting, speaking, travel/meals/alcohol etc. The site https://openpaymentsdata.cms.gov/ is a searchable database of payments made by drug and medical device companies to physicians, physician assistants, advanced practice nurses and teaching hospitals from January, 2015 through December, 2021.

Again, none of this proves bias, but it can indicate a higher chance of both extrinsic and intrinsic bias.

Who was (and wasn’t included)?

One of the first things I look for in a study is who was included in the study population. Were marginalized people (people of color, trans and nonbinary, fat people etc.) represented? At what percentage (does the representation, for example, reflect the rate at which these folks exist in the world?)

QUESTIONS FOR SPECIFIC TYPES OF RESEARCH

For claims linking being higher-weight to health issues:

Was a causal mechanism identified or is it just correlation?

• Terms like linked, connected, risk factor, and correlated are good clues that a causal mechanism has not been identified and that the conclusion is that being higher-weight occurs at the same time as the health issue with some frequency. That absolutely does not prove that being higher-weight causes the health issue or, for that matter, that losing weight would reduce the incidence of the health issue. More about that here.

Does this study control for the impacts of weight stigma, weight cycling, and healthcare inequalities?

• In healthcare research, correlation is often used even if a causal mechanism is not identified, but that’s only appropriate after investigating possible confounding variables. In the case of weight loss, weight stigma, weight cycling, and healthcare inequalities are well-researched confounding variables that are typically ignored in research. Given that we know that these three things are correlated to many of the same health issues to which being higher-weight is correlated, there is no excuse for claiming correlation without, at the very least, acknowledging that these could be to blame. More about this here.

For studies with weight loss claims:

What was the drop-out rate?

• Often these studies have high dropout rates and those who dropped out are not accounted for (so the conclusion says some form of “everyone lost weight” but in fact a lot of people - I’ve seen up to 2/3 - in the sample dropped out with no follow-up.)

How much weight was actually lost?

• A lot of the time these studies seem to be going to great lengths to hide the amount of weight that was actually lost. Instead using broad generalizations like “significant” or “successful” without attaching numbers. When you dig for this information you’ll often find that there wasn’t much weight lost at all.

Are they using “significant” when they mean statistically significant?

• If a result is “statistically significant,” it means that it’s more likely that the result was caused by the study intervention than by chance. So participants could have lost an average of two pounds, but if it’s determined that it’s more likely that those two lost pounds were due to the study intervention than by chance, then that two-pound loss is statistically significant. What sometimes happens with weight science is that the conclusion states that participants lost “a significant amount of weight” when what it means is that the small amount of weight that was lost was statistically significant. Whether accidentally or on purpose, this misleads people (including healthcare practitioners) due to the colloquial meaning of significant, to believe that the intervention is far more successful than it actually is.

Does this study have at least five years of follow-up?

• Not only is there typically a fairly small amount of weight actually lost, but the research we have shows that the vast majority of people lose weight short-term and gain it back within 2-5 years. The weight-loss industry has long been taking advantage of this by running studies that are short-term (sometimes just a few weeks long, other times up to two years.) With the few week studies, the claim is made (or strongly implied) that the weight lost is not just sustainable, but that people could use this intervention to keep losing weight indefinitely. With the two-year studies, the conclusion often says something like “all participants regained weight, but remained below their starting weight” without pointing out that the weight regain arrow was going straight up before they stopped tracking it. The idea that if people in the study could lose a little weight in a short time then anybody could lose as much as they want over a long time period is NOT supported by the evidence that exists.  

Were the subjects human?

• I see this a lot in articles, the study subjects were not human but the article is…less than forthcoming about it. If the article only uses the term “subjects” and only talks about the percentage of weight lost, you might want to do some digging to see if these were, in fact, animal subjects.

For health benefit claims:

Has this study separated the impacts of behavior change from the impacts of weight loss?

• This goes back to the dubious claim that 5-10% weight loss creates health benefits. The short story is that if an intervention includes behavior changes, that were then followed by both (at least short-term) weight loss, and health changes, we cannot conclude that the weight loss is what created the health changes, since it could easily have been the behavior changes, as has been found in research that I wrote about in-depth here.

Does this study compare weight-suppressed people to people who were never fat?

• Comparisons of thin people to fat people who have lost weight are of questionable validity. So, if a study suggests that weight loss should be undertaken because thin people experience a health issue less often than fat people, then that claim is questionable, at best. People who have never been fat and fat people who lose weight (at least temporarily) are not an apples-to-apples comparison for many reasons including everything from potential genetic differences to possible lasting impacts of weight stigma, weight-cycling, and healthcare inequalities that fat people have experienced.

Does this study assume that weight loss increases health?

• Weight loss and health improvements are two separate things. Often studies conflate the two, suggesting (or assuming) that if weight was lost, then health was improved, with no actual proof of that.

Does this show benefits above and beyond those seen with weight-neutral interventions?

• There is good evidence that weight-neutral interventions provide strong benefits with few risks. Given the high long-term failure rate and risks of weight loss attempts, this is an important question to ask.

For pharmaceuticals:

In addition to the questions above, does approval/recommendation predicate risk on body size?

• These drugs are approved based on a risk/benefit analysis. If the approval is based on weight (i.e. it’s approved only for people with a BMI over X and/or a lower BMI with an additional “co-morbidity”) then they are suggesting that higher-weight people should be exposed to more danger. In the history of weight loss drugs, this has done significant harm.

For Bariatric Surgeries

Does this study have more than 10 years of follow-up?

• These procedures move a healthy digestive system from a perfectly functioning state into a disease state (often irreversibly) with little to no long-term data about the impact. Adding more short-term data doesn’t provide information about what life will be like for those who survive the surgery after ten or more years.

What was the framework for investigating adverse effects?

• Often I find that these studies do a horrible job (or no job at all) tracking negative outcomes and adverse effects, basing their conclusions on the idea that if the patient lost weight and didn’t die (at least within the relatively short follow-up,) then the procedure was a “success.”

Were participants blamed for adverse effects?

• Sometimes if the study does take adverse events and effects into account, the authors work hard to suggest that the blame for any poor outcomes should be placed on the survivor of the procedure rather than the procedure itself.

Do they compare people who didn’t qualify for the surgery with people who did?

• This is a subtle bias, but a bias nonetheless (and one that I think I first heard about from Deb Burgard.) It’s possible that the reason that those people didn’t quality for the surgery is also the reason that they had poorer outcomes.

For more in-depth information about these surgeries I have a three-part series here, as well as a piece specifically discussing the different types of procedures.

Did you find this post helpful? You can subscribe for free to get future posts delivered direct to your inbox, or choose a paid subscription to support the newsletter (and the work that goes into it!) and get special benefits! Click the Subscribe button below for details:

More research and resources:
https://haeshealthsheets.com/resources/

*Note on language: I use “fat” as a neutral descriptor as used by the fat activist community, I use “ob*se” and “overw*ight” to acknowledge that these are terms that were created to medicalize and pathologize fat bodies, with roots in racism and specifically anti-Blackness. Please read Sabrina Strings’ Fearing the Black Body – the Racial Origins of Fat Phobia and Da’Shaun Harrison’s Belly of the Beast: The Politics of Anti-Fatness as Anti-Blackness for more on this.