I’ve written before about how the statistic “50% of all clinical trials are unpublished”, much beloved of the All Trials campaign, is simply not evidence based.
The charity Sense About Science, who run the All Trials campaign, also run a rather splendid website called “ask for evidence“, which encourages people to ask for evidence when people make dodgy claims.
So I used Sense About Science’s website to ask for evidence from Sense About Science for their claim that 50% of all trials are unpublished.
To their credit, they responded promptly, and pointed me to this article, which they claimed provided the evidence behind their claim.
So how well does it support their claim?
Interestingly, the very first sentence states that they don’t really have evidence. The first paragraph of the document reads as follows:
“We may never know the answer to this question. In some ways, maybe it doesn’t matter. Even one clinical trial left unreported is unacceptable.”
So if they don’t know, why are they making such a confident claim?
As an aside, they are of course spot on in the rest of that paragraph. Even if the proportion of unpublished trials is substantially less than 50%, if it’s greater than zero it’s still too high.
They go on to further emphasise the point that they really don’t know what proportion of trials are unpublished:
“It is clearly not a statistic, and we wouldn’t advocate trying to roll up the results of all the studies listed below to produce something spuriously precise.”
They then go on to explain the complexity of estimating the proportion of unpublished trials. It certainly is complex, and they give a good explanation of why. It’s not a bad document, and even includes some studies showing much higher rates of disclosure that they don’t admit to on their main website article.
But if they understand, as they clearly do, that the claim that half of all trials are unpublished is spuriously precise and that it would be wrong to claim that, why do they do so anyway?
Not only do they make the claim very confidently on their own website, the claim also often appears in the very many articles that their PR machine churns out. These articles state it as fact, and do not acknowledge the problems that they describe in their background document. You will never see those articles citing the most recent research showing greater than 90% disclosure rates.
This still seems dishonest to me.