The Fallacies of Comparative Studies

Skin Care


Hmm, which side to take?
Hmm, which side to take?

When I refer to comparative studies, I’m talking about ones that evaluate two (or more) ingredients in terms of the same clinical characteristic, such as reducing inflammation and fading hyperpigmentation. There’s no doubt that these kinds of studies have their benefit: they quantify the relative efficacy of one ingredient to another, and are frequently used to demonstrate how a novel ingredient can be nearly or just as good as a tried-and-true one. However, it’s important to recognize that these results shouldn’t be taken at face value.

Fallacy #1: Lack of Vehicle Control

Talk about no vehicle control.
Talk about no vehicle control.

Most comparative studies only compare two different “active” or “featured” ingredients in two separate vehicles. So when you read that the two ingredients perform similarly, you accept that as the truth. In fact, I myself have done this multiple times in the past. And like I said above, it’s not wrong to do so. You just have to remember to not jump to conclusions.

For those of you who already spotted the potential caveat in this scenario, good job! For everyone else, the first source of error is that the two vehicles, in which the two “active” ingredients are suspended, have not been tested to see if they contributed to the final results; basically the studies are not vehicle-controlled. In order to eliminate any doubt, each (vehicle) would first have to be juxtaposed against its corresponding (vehicle + “active” ingredient) in terms of the same clinical characteristic as that in the comparative study. Then, the “active” formulation must demonstrate a statistically significant improvement (in the same characteristic) in comparison to its vehicle alone.

See how this can create feelings of doubt?

Lesson #1: Check and see if the two “sides” of any particular comparative studies have been vehicle-tested for efficacy.    

Fallacy #2: Different Mechanisms of Action

Benzoyl peroxide targets P. Acnes.
Benzoyl peroxide targets P. Acnes.

While the potential ramifications of not having studies with vehicle-controlled formulations are a very real issue, their overall effect on the validity of any comparative study and on their relevancy to real-life scenarios is small. Because virtually every study that’s accessible to the public on Pub Med originated from a published medical journal—meaning that it had to withstand close scrutiny by a large group of qualified individuals, it is unlikely (though still possible) that they all failed to notice the inclusion of a vehicular ingredient that has the ability to meaningfully skew the test results.

Therefore, it is this second “fallacy” that is of greater concern, which is that, the differing mechanisms of action between the “active” ingredients. Take this study for example. It suggests that 14% gluconolactone is about as effective as 5% benzoyl peroxide at reducing the number of inflammatory lesions (think papules and pustules) and non-inflammatory ones (think whiteheads and blackheads). Furthermore, the study concludes that fewer negative side effects were experienced by patients treated with gluconolactone. By reading this, many people will simply assume that gluconolactone is just the better treatment, and that the study proves that there’s no real need for the “antiquated use of benzoyl peroxide.”

Gluconolactone targets corneocytes.
Gluconolactone targets corneocytes.

However, what these people fail to recognize is that these two ingredients reduce acne via very different mechanisms of action. Benzoyl peroxide targets the acne-promoting anaerobic P. Acnes bacteria by flooding the skin with oxygen. Gluconolactone targets the excess dead skin cells (corneocytes) and forces them to fall off the skin. However, while benzoyl peroxide tends to penetrate deeply into the lower epidermis, gluconolactone tends to remain in the stratum corneum and the upper epidermis. Therefore, for deeper acne lesions such as cysts and nodules, benzoyl peroxide would be more effective.  The results of the comparative study reveal none of this. So again, take nothing at face value.

Think of it like this: One ingredient or person in this example can bend a spoon with his hands. Another person can bend a spoon with his mind. If you only looked at the results–bent spoons, you’d consider both persons to be pretty much equal, which is clearly not the case. Yes, that’s an extreme example, but you get the idea.

Lesson #2: Make sure that you thoroughly understand the mechanisms of action, because they can make a huge impact on your skin overall, depending on your needs.

Comparative Studies: Looking Ahead

Mhm. Let's sing Kumbaya and do it together!
Mhm. Let’s sing Kumbaya and do it together!

So keep in mind the two lessons discussed here as you peruse relevant studies, and/or when you see someone else cite a comparison study. Make sure to examine and question the appropriateness and correctness of the author’s claims and supporting documentation.

Finally, like I said before, comparative studies have many clinical and therapeutic benefits. However, it’s best to not try and look for the “best” ingredient for your skin/condition. As with anything, employ a multifaceted approach to treating your skin/condition by using ingredients that do the same thing, but via different mechanisms of action. So if you’ve got hyperpigmentation, use ingredients like retinoids, hydroquinone, niacinamide, and glycolic acid. If you’ve got acne, use ingredients like salicylic acid, retinoids, benzoyl peroxide, and azelaic acid.

Find this all confusing? Well, stay tuned because on my blog, I’ll currently writing a massive 30-part Page on what the Ideal Skin Care Routine is and how you can get there! Gosh, I sound like an infomercial… Anyways, here’s a sneak peek at how the Page will be organized.


Check our bestsellers!

  • @janine

    Why, thank you. 😉

    Goal… AH-chieved!

  • janine

    It is so easy for product junkies like me to only hear what we want to hear out of some of these “comparative studies”.
    I really need posts like this one to keep me in check and making discerning, intelligent choices with my skin care.
    As always, THANK YOU.

  • @Jessica Allison

    Glad to heart it! I appreciate you taking the time to express your thoughts.

  • I love your articles, John! Through the expansion of Future Derm, I’ll admit, the first thing I do when I see a new post is look at the byline. You, and of course, Nikki, are always must reads!

  • @Eileen

    Thanks! 🙂

    You make an excellent point that a lot of people don’t get the results they desire, due to a lack of understanding.

  • Eileen

    It’s consumer friendly articles like these that keep me coming back to Future Derm. Your lesson #2 about knowing your skin care needs is often overlooked by people in their search for the holy grail product that will transform their skin and render it flawless. A disconnect between the skin’s needs and the manner in which a product works is responsible for a lot of frustration on the part of consumers. I took a look at your “sneak peek” and am looking forward to the completed article. Well done, John!

Recent Posts