Analyzing Studies

[1986]

very interesting thread. it is important for people to think critically when reading studies rather than blindly following what the researcher says.

[EricT]

I thought that this letter, posted by the New England Journal of Medicine, was very appropriate here and a good example of some of the things discussed. Specifically, the inappropriate use of studies to cite the effectiveness of certain 'supplements'. In this case, so-called human growth hormone sups.

Inappropriate Advertising of Dietary Supplements

Jeffrey M. Drazen, M.D.

Related Article
by Vance, M. L.

PubMed Citation


Information about health is worthwhile only if it is accurate and fairly communicated. Recently, a number of advertisements on the Internet have contained statements about dietary supplements that could lead one to believe, erroneously, that information published in the Journal supported the claims made by the manufacturers.

As the editor-in-chief of the Journal, I spend the bulk of my time working with our editors and outside reviewers to ensure the accuracy of results presented in the articles we publish. The task is complicated by the fact that the work we publish is often at the cutting edge of research, where medical science meets clinical practice, and therefore facts are often not immediately in clear focus. Our reputation has been built, however, on taking a scientifically conservative stance. We often choose not to publish a paper because we believe that doubt remains about the veracity of its findings. When we decide to publish, we work hard to make sure that the statements made by the authors are supported by the evidence, clearly communicated, and fairly presented. Balance and objectivity are our goals.

Sadly, caution and balance are not universal characteristics in advertising about health. New legislation has made it legal for companies to market dietary supplements without approval of the Food and Drug Administration, and on the Internet they can do so at very little cost. Anyone with a public e-mail account is likely to receive hundreds, if not thousands, of messages a year in which unproven claims are made about dietary supplements or other products and the medical wonders they can achieve. Most of these claims seem too good to be true, and nearly all are.

We find one case particularly unnerving. A number of advertisements for human growth hormone or for dietary supplements said to be "human growth hormone releasers" refer to statements made in our pages as evidence of the value of these products. The advertisers cite a study reported in the Journal in 1990 by Rudman et al.1 In this study, 12 men older than 60 years of age who had low plasma levels of insulin-like growth factor 1 (representing about one third of otherwise healthy men 60 to 80 years of age) were treated with injections of human growth hormone three times weekly for six months; these men had statistically significant increases in lean body mass and bone mineral, unlike a group of 9 similar men who received no treatment.

Although the findings of the study were biologically interesting, the duration of treatment was so short that side effects were unlikely to have emerged, and it was clear that the results were not sufficient to serve as a basis for treatment recommendations. Indeed, Mary Lee Vance of the University of Virginia said in an accompanying editorial,2 "Because there are so many unanswered questions about the use of growth hormone in the elderly and in adults with growth hormone deficiency, its general use now or in the immediate future is not justified." Dr. Vance restates her views on the study in this issue of the Journal (pages 779–780); they remain fundamentally unchanged.

In some of the cases of which we have recently become aware, the advertised product is not human growth hormone but, rather, a mixture of substances that is claimed only to stimulate the body to release human growth hormone. There are no data in the advertisements or accompanying the products themselves, so far as we have seen, that support this claim or prove that the substance has no harmful side effects. We have reported our concern to the offices of the attorneys general in two states and are awaiting the outcome of their investigations.

We are especially concerned because the promotional e-mails are apparently sending readers to our Web site; the 1990 article by Rudman et al. receives as many "hits" in a week as other 1990 articles do in a year. If people are induced to buy a "human growth hormone releaser" on the basis of research published in the Journal, they are being misled. In order to warn those who visit our Web site for this reason, this Perspective article and Dr. Vance's commentaries will from now on appear with the article by Rudman et al. each time it is downloaded. The point, as stressed by an earlier editor-in-chief of the Journal, Franz Ingelfinger, is that "advertising is advertising" and nothing more. Readers need to know that the editors of the Journal do not endorse any product for any commercial use. We strongly believe that when scientific information is communicated clearly and fairly, it speaks for itself.

References


Rudman D, Feller AG, Nagraj HS, et al. Effects of human growth hormone in men over 60 years old. N Engl J Med 1990;323:1-6. [Free Full Text]
Vance ML. Growth hormone for the elderly? N Engl J Med 1990;323:52-54. [Free Full Text]

[EricT]

I got around to reading more of Bryan Chung's blog,
Evidence Based Fitness, which I linked at the beginning as well. Really a lot of generous info that should help cut throught the science mumbo jumbo. This is a way of arming yourself against the "experts" whose "science", whether you realize or not, isn't always as sensible as it sounds.

April, 2007 (filled with good stuff), has a thing on abstracts which I found very educational.

April, 2007

Hrdgain may like all his writings on Beta Alanine (but it's not the beta alanine blog says Bryan, haha).

Atually one of my favorite strength, conditioning and corrective coaches recently mentioned as "abstract" in one of his writings as support and, even without knowing much, I was a little put off by a guy in his position using just an abstract in research support, especially since this thing I was reading was NOT free. But what he was talking about was sort of an aside and not so important and he may have just mentioned the abstract because he knew his readers would have easy access to it as opposed to a full paper. I don't know but it's something to watch for.

[BG5150]

I don't read studies. I get all my information from the forums at bodybuilding.com.

[EricT]

Right. And all the yahoos, at bb.com DO read studies (or at least abstracts). Or they regurgitate stuff from authors who supposedly read studies (or at least abstracts)

Listen to people who spout science while not trying to verify the science is basically going on faith.

[Ross86]

That's why I don't like BB.com........................................

[EricT]

I think next I will find one of the many articles that cite studies where some of the studies don't have any actual thing to do with the subject matter and point it out.

A favorite trick is to say some general statement and then cite like "12-23". Then you look at some of the studies and you really don't have a clue what they are ACTUALLY using to support their statement. And most likely they just included ever "related" paper they saw listed under the first one they found.

[Ross86]

I have seen some articles like that about sesamin oil and ecdysterone. It's so bogus. That's what got me to at least glance at the sources list before moving on every time after I read an article.

[EricT]

Do tell. If you come across them again an example would be great.

Here is an example that I still remember. Most people have seen the thread and the artices on "30 Grams of Protein Myth". There is a Will Brink article and another one as well that basically says....no studies or evidence has ever been found on protein limit by the authors. The conclusion is not to worry about it and of course it sounds pretty silly anyway. But just because you don't believe it doesn't mean you have the right to act like there is some type of PROOF of how much protein everyone can assimilate and then post bogus sources.

Here is the example I'm speaking of from Bryan Haycock:

Fact: The body has the ability to digest and assimilate much more than 30 grams of protein from a single meal.

Speaking of high intakes of protein, people have been perpetuating the myth that you can only assimilate ~30 grams of protein at a time, making protein meals any greater than a 6 oz. chicken breast a waste. This is anything but true. For example, the digestibility of meat (i.e. beef, poultry, pork and fish) is about 97% efficient. If you eat 25 grams of beef, you will absorb into the blood stream 97% of the protein in that piece of meat. If, on the other hand, you eat a 10 oz steak containing about 60 grams of protein, you will again digest and absorb 97% of the protein. If you could only assimilate 30 grams of protein at a time, why would researchers be using in excess of 40 grams of protein to stimulate muscle growth?1

And here is the "1" citation he is listing:

http://ajpendo.physiology.org/cgi/content/full/276/4/E628

It is one of those Tipton things on post exercise amino acids (solution). There is NOTHING in it that has anything to do with how much steak you can assimilate, etc. It has no bearing whatsoever on the "30 grams myth" that I can see.

Very bogus. Sorry to any Haycock fans but it is what it is. What he is really doing is making a statement about 97 percent that cannot be verified and then using the study as an example of why he asks the question "why would researchers use in excess of 40 grams"...as if that somehow proves something. The effect is that most people will see a "study" listed and take the whole paragraph as "truth".

[widdoes2504]

People hear what they want to hear and pretty much ignore the rest IMO. When someone who desperately wants to justify something (like the article you mentioned) and they find something that seems to fit their position, they tend to use it as absolute fact without even investigating for themselves to see if it truly is worth the read.

People are also very gullible. Just watch some of the commercials out there....scary stuff.