Clinically proven

My eye was caught the other day by this advert:

Boots

Quite a bold claim, I thought. “Defends against cold and flu” would indeed be impressive, if it were true. Though I also noticed the somewhat meaningless verb “defend”. What does that mean exactly? Does it stop you getting a cold or flu in the first place? Or does it just help you recover faster if you get a cold or flu?

I had a look at the relevant page on the Boots website to see if I could find out more. It told me

“Boots Pharmaceuticals Cold & Flu Defence Nasal Spray is an easy to use nasal spray with antiviral properties containing clinically proven Carragelose to defend against colds and flu, as well as help shorten the duration and severity of both colds and flu.”

It then went on to say

“Use three times a day to help prevent a cold or flu, or several times a day at the first signs helping reduce the severity and duration of both colds and flu.”

OK, so Boots obviously want us to think that it can do both: prevent colds and flu and help treat them.

So what is the evidence? Neither the advert nor the web page had any links to any of the evidence backing up the claim that these properties were “clinically proven”. So I tweeted to Boots to ask them.

To their credit, Boots did reply to me (oddly by direct message, in case you’re wondering why I’m not linking to their tweets) with 4 papers in peer reviewed journals.

So how does the evidence stack up?

Well, the first thing to note is that although there were 4 papers, there were only 3 clinical trials: one of the papers is a combined analysis of 2 of the others. The next thing to note is that all 3 trials were of patients in the early stages of a common cold. So right away we can see that we have no evidence whatsoever that the product can help prevent a cold or flu, and no evidence whatsoever that it can treat flu.

The “clinically proven” claim is starting to look at little shaky.

But can it at least treat a common cold? That would be pretty impressive if it could. The common cold has proved remarkably resilient to anything medical science can throw at it. A treatment that actually worked against the common cold would indeed be good news.

The first of the trials was published in 2010. It was an exploratory study in 35 patients who were in the first 48 hours of a cold, but otherwise healthy. It was randomised and double-blind, and as far as I can tell from the paper, seems to have been reasonably carefully conducted. The study showed a significant benefit of the nasal spray on the primary outcome measure, namely the average of a total symptom score on days 2 to 4 after the start of dosing.

Well, I say significant. It met the conventional level of statistical significance, but only just, at P = 0.046 (that means that there’s about a 1 in 20 chance you could have seen results like this if the product were in fact completely ineffective: not a particularly high bar). The size of the effect also wasn’t very impressive: the symptom score was 4.6 out of a possible 24 in the active treatment group and 6.3 in the placebo group. Not only that, but it seems symptom scores were higher in the placebo group at baseline as well, and no attempt was made to adjust for that.

So not wholly convincing, really. On the other hand, the study did show quite an impressive effect on the secondary outcome of viral load, with a 6-fold increase from baseline to day 3 or 4 in the placebo group, but a 92% decrease in the active group. This was statistically significant at P = 0.009.

So we have some preliminary evidence of efficacy, but with such a small study and such unconvincing results on the primary outcome of symptoms, I think we’re going to have to do a lot better.

The next study was published in 2012, and included children (ages 1 to 18 years) in the early stages of a common cold. It was also randomised and double blind. The study randomised 213 patients, but only reported efficacy data for 153 of them, so that’s not a good start. It also completely failed to show any difference between the active and placebo treatments on the primary outcome measure, the symptom score from days 2 to 7. Again, there was a significant effect on viral load, but given the lack of an effect on the symptom score, it’s probably fair to say the product doesn’t work very well, if at all, in children.

The final study was published in 2013. It was again randomised and double blind, and like the first study included otherwise healthy adults in the first 48 h of a common cold. The primary endpoint was different this time, and was the duration of disease. This was a larger study than the first one, and included 211 patients.

The results were far from impressive. One of the big problems with this study was that they restricted their efficacy analysis to the subset of 118 patients with laboratory confirmed viral infection. Losing half your patients from the analysis like this is a huge problem. If you have a cold and are tempted to buy this product, you won’t know whether you have laboratory confirmed viral infection, so the results of this study may not apply to you.

But even then, the results were distinctly underwhelming. The active and placebo treatments were only significantly different in the virus-positive per-protocol population, a set of just 103 patients: less than half the total number recruited. And even then, the results were only just statistically significant, at P = 0.037. The duration of disease was reduced from 13.7 days in the placebo group to 11.6 days in the active group.

So, do I think that Boots Cold and Flu Defence is “clinically proven”? Absolutely not. There is no evidence whatsoever that it prevents a cold. There is no evidence whatsoever that it either prevents or treats flu.

There is some evidence that it may help treat a cold. It’s really hard to know whether it does or not from the studies that have been done so far. Larger studies will be needed to confirm or refute the claims. If it does help to treat a cold, it probably doesn’t help very much.

The moral of this story is that if you see the words “clinically proven” in an advert, please be aware that that phrase is completely meaningless.

21 thoughts on “Clinically proven”

  1. Hello,

    From my own statistical sample of one (which i know is pretty meaningless) I’ve had a great result from both Vic first then Boots cold defense use. Not had a cold this year for the first time in around 20 years. I travel by train and tube every day and have been surrounded by people who’ve had the cold/flu in the last 5 months. Usually I’d have had it at leat 2 or 3 times so far. So it’s working pretty well for me. I’d be happy to participate in a decent trial for this stuff, but even withought that so far it’s done pretty well for me, or maybe i’ve just been lucky! I’m going to keep using it and see how long I can go before I get a cold.

  2. “First Defence” as a cold preventative. Does it work?

    This paper reports the development and implementation of a unique research project designed to investigate the effects of “First Defence” as a preventative for colds.
    Previous studies investigating the effects of cold and flu nasal sprays employed participants already at the onset of an upper respiratory tract infection and their results have been inconclusive.
    This unique study investigates the effectiveness of a nasal spray used as a daily preventative.
    One female singing teacher (age: 42-44yrs) completed a 2-year longitudinal empirical study. Data regarding her lifetime experience was noted before the intervention where 4-6 colds a year were reported.
    “First Defence” was taken nightly after teaching, 3 squirts up each nostril. A slight tingly sensation was noticed.
    An ongoing diary method was employed to register day-to-day effects and the participant reported no side-effects or illness, including colds or flu, for the full 2 year experiment.
    The implication from this groundbreaking research is that vocal students, teachers of voice and singing performers at any level would benefit greatly from taking “First Defence” to avoid catching colds.
    Limitations of the research were the amount of participants and the fact the participant administered the research. There was no control, no randomization and no ethics approval was sought. No conflict of interest was reported.
    Further non-serious research is being conducted to investigate the effects on a wider demographic of singing students.

    😉

  3. I have had around 50 colds over the last 4 years and I have taken BOOTS cold & flu defence and each time, it has effectively cleared it up. 100% got rid of it. I took it when I had the flu and it cleared it within 2 days.

    So who gives a sh*t about written studies and evidence!!! If you just took it yourself when you had a cold or the flu, you would see for yourself that it WORKS!!!!! You have to take it 3 times a day, 2 squirts in each nostral for a week !!!

    1. How do you know your colds wouldn’t have got better if you didn’t take it?

      If the product really worked as well as you think it does, then that would have shown up in the clinical trials. But it didn’t.

      That, my friend, is why we have to do properly controlled clinical trials. It’s pretty much impossible to know whether something works otherwise.

  4. These products do not contain active ingredients and instead are salt solution with some polysaccharide which forms a barrier preventing the virus from penetrating the nasal mucosa- a process needed for systemic infection, spread and development of a cold. Not sure therefore if you can reliably do a clinical trial or state clinically proven. I use these products when I feel that pain in my nose that there is an inflammatory reaction occurring (a possible cold forming) and if dodged a few colds in the last few years, but this (like others) is very subjective evidence. I use the product as to me the principle of how it works makes sense and it appears to work- with the caveat that this is very subjective. If someone told me that other therapies worked I would go through the same exercise before regularly using them.

    1. Having read the packaging, I believe they also contain Eucalyptus, Zinc, and something Acidic. Zinc was found, by the common cold unit, to inhibit cold progression if taken in large enough quantities so I think there is, at least, some sort of experimental background to the ingredients.

  5. This stuff is the advertising industry’s dream: if you don’t get a cold, the medicine worked; if you do get a cold, you took it too late. The same argument could apply to all those hair-restoring medications such as Regaine.

  6. I would not be without Boots spray. I use it both when I have been with someone with a cold, or if I feel the very first symptoms. I now seldom if ever get a full blown cold. It is miraculous as far as I am concerned. (The Vick spray hurts my nose so I abandoned that) .

  7. I have non CF Bronchiectasis and suffer severe side effects to most antibiotics. It is really important that if I am going to stay out of hospital that I do not pick up bacterial/viral infections. This Spray was originally created by a company based in Vienna. To find out more about trials it is best to go to their website. There is a strong tradition in folk medicine in the UK that red seaweed- carrageen is good for upper respiratory tract and lung infections. Right now there are clinical trials going on in Wales using red seaweed as a new treatment for Cystic Fibrosis.
    Since the end of September 2015 I have had no Exacerbations. I am putting this down to using BOOTS Cold and Flu Defense before and during outings on public transport/public places, avoiding visits from people with colds, hand washing and relying on a face mask when I have to. This winter has been really bad for virus/ bacteria infections and right now at least half the population are coughing and sneezing. I would recommend this product and I strongly believe that due to the strategy I have adopted I have managed to stay in better health than many people without the chronic lung condition I have.

    1. So let me get this straight. You have been avoiding people with colds, been careful about washing your hands, and wearing a face mask, and you think it’s the Boots spray that’s the reason you’ve avoided getting a cold?

  8. The data you provided initially confirms it must work only 20 to 1 chance it doesnt From having colds/manflu no idea how many times best guess 2 to three times per year to no colds in 3 years I’m certain it works as are loads of people everyone I know who has tried it swears by it I don’t think it can cure manflu that sounds implausible +it tastes horrible /stings I always take a halls menthol lyptus drop to mask the taste I also drink whisky, stop all cold drinks, and dress warm when I feel a cold coming on none of which is clinically proven but why not who wants a cold

    1. When you say “only 20 to 1 chance it doesn’t”, you are making a very common error in the interpretation of P values. A P value of 0.05 does not mean that there is only a 1 in 20 chance that a product is ineffective. It means that there is a 1 in 20 chance that, if the product is ineffective, you would have seen the results that you did. Those are not the same thing.

      The wonderful XKCD explains why they are not the same thing better than I could:

      https://xkcd.com/1132/

  9. Thank you for this blog. Until now I had been blindly sniffing the stuff as I thought there was peer reviewed science behind it. What a joke.

    Until they actually conduct some proper studies, I guess I’ll just stick to tissues and a grumpy face.-

  10. It is confusing that there are not more studies being conducted to support the use of these sprays because my experience (and many other people i know) is that they have proven highly effective at halting the development of colds. Working in an environment with lots of unwell people was resulting in 1-3 colds per year for me, but since i bought my first bottle 3 years ago i have not had a single full blown cold following the early signs. I agree that the empirical evidence does not appear to support the claims, but my experiences absolutely do.

  11. Was it your complaint that took the product off the shelves for a period this year? Having worked well for me in the past, I tried to get it but was unable, I then had a full on cold over an important engagement that probably could have been prevented or made a lot less severe. Whether or not it was adequately clinically proven to be advertised as such, its not snake oil and does work. Did you spot the advert while raising your head to sneeze over everyone else in the carriage? Because that would infer a similar indifference to the conequences of your actions for other people.

    1. No, my complaint was only about the advert. Boots are still perfectly entitled to sell the product if they wish. If they are not selling it, you’ll have to ask them why not.

Leave a Reply

Your email address will not be published. Required fields are marked *