Category Archives: Politics

The Trials Tracker and post-truth politics

The All Trials campaign was founded in 2013 with the stated aim of ensuring that all clinical trials are disclosed in the public domain. This is, of course, an entirely worthy aim. There is no doubt that sponsors of clinical trials have an ethical responsibility to make sure that the results of their trials are made public.

However, as I have written before, I am not impressed by the way the All Trials campaign misuses statistics in pursuit of its aims. Specifically, the statistic they keep promoting, “about half of all clinical trials are unpublished”, is simply not evidence based. Most recent studies show that the extent of trials that are undisclosed is more like 20% than 50%.

The latest initiative by the All Trials campaign is the Trials Tracker. This is an automated tool that looks at all trials registered on clinicaltrials.gov since 2006 and determines, using an automated algorithm, which of them have been disclosed. They found 45% were undisclosed (27% of industry sponsored-trials and 54% of non-industry trials). So, surely this is evidence to support the All Trials claim that about half of trials are undisclosed, right?

Wrong.

In fact it looks like the true figure for undisclosed trials is not 45%, but at most 21%. Let me explain.

The problem is that an automated algorithm is not very good at determining whether trials are disclosed or not. The algorithm can tell if results have been posted on clinicaltrials.gov, and also searches PubMed for publications with a matching clinicaltrials.gov ID number. You can probably see the flaw in this already. There are many ways that results could be disclosed that would not be picked up by that algorithm.

Many pharmaceutical companies make results of clinical trials available on their own websites. The algorithm would not pick that up. Also, although journal publications of clinical trials should ideally make sure they are indexed by the clinicaltrials.gov ID number, in practice that system is imperfect. So the automated algorithm misses many journal articles that aren’t indexed correctly with their ID number.

So how bad is the algorithm?

The sponsor with the greatest number of unreported trials, according to the algorithm, is Sanofi. I started by downloading the raw data, picked the first 10 trials sponsored by Sanofi that were supposedly “undisclosed”, and tried searching for results manually.

As an aside, the Trials Tracker team get 7/10 for transparency. They make their raw data available for download, which is great, but they don’t disclose their metadata (descriptions of what each variable in the dataset represents), so it was rather hard work figuring out how to use the data. But I think I figured it out in the end, as after trying a few combinations of interpretations I was able to replicate their published results exactly.

Anyway, of those 10 “undisclosed” trials by Sanofi, 8 of them were reported on Sanofi’s own website, and one of the remaining 2 was published in a journal. So in fact only 1 of the 10 was actually undisclosed. I posted this information in a comment on the journal article in which the Trials Tracker is described, and it prompted another reader, Tamas Ferenci, to investigate the Sanofi trials more systematically. He found that 227 of the 285 Sanofi trials (80%) listed as undisclosed by Trials Tracker were in fact published on Sanofi’s website. He then went on to look at “undisclosed” trials sponsored by AstraZeneca, and found that 38 of the 68 supposedly undisclosed trials (56%) were actually published on AstraZeneca’s website. Ferenci’s search only looked at company websites, so it’s possible that more of the trials were reported in journal articles.

The above analyses only looked at a couple of sponsors, and we don’t know if they are representative. So to investigate more systematically the extent to which the Trials Tracker algorithm underestimates disclosure, I searched for results manually for 100 trials: a random selection of 50 industry trials and a random selection of 50 non-industry trials.

I found that 54% (95% confidence interval 40-68%) of industry trials and 52% (95% CI 38-66%) of non-industry trials that had been classified as undisclosed by Trials Tracker were available in the public domain. This might be an underestimate, as my search was not especially thorough. I searched Google, Google Scholar, and PubMed, and if I couldn’t find any results in a few minutes then I gave up. A more systematic search might have found more articles.

If you’d like to check the results yourself, my findings are in a csv file here. This follows the same structure as the original dataset (I’d love to be able to give you the metadata for that, but as mentioned above, I can’t), but with the addition of 3 variables at the end. “Disclosed” specifies whether the trial was disclosed, and if so, how (journal, company website, etc). It’s possible that trials were disclosed in more than one place, but once I’d found a trial in one place I stopped searching. “Link” is a link to the results if available, and “Comment” is any other information that struck me as relevant, such as whether a trial was terminated prematurely or was of a product which has since been discontinued.

Putting these figures together with the Trials Tracker main results, this suggests that only 12% of industry trials and 26% of non-industry trials are undisclosed, or 21% overall (34% of the trials were sponsored by industry). And given the rough and ready nature of my search strategy, this is probably an upper bound for the proportion of undisclosed trials. A far cry from “about half”, and in fact broadly consistent with the recent studies showing that about 80% of trials are disclosed. It’s also worth noting that industry are clearly doing better at disclosure than academia. Much of the narrative that the All Trials campaign has encouraged is of the form “evil secretive Big Pharma deliberately withholding their results”. The data don’t seem to support this. It seems far more likely that trials are undisclosed simply because triallists lack the resources to write them up for publication. Research in industry is generally better funded than research in academia, and my guess is that the better funding explains why industry do better at disclosing their results. I and some colleagues have previously suggested that one way to increase trial disclosure rates would be to ensure that funders of research ringfence a part of their budget specifically for the costs of publication.

There are some interesting features of the 23 out of the 50 industry-sponsored trials that really did seem to be undisclosed. 9 of them were not trials of a drug intervention. Of the 14 undisclosed drug trials, 4 were of products that had been discontinued and a further 3 had sample sizes less than 12 subjects, so none of those 7 studies are likely to be relevant to clinical practice. It seems that undisclosed industry-sponsored drug trials of relevance to clinical practice are very rare indeed.

The Trials Tracker team would no doubt respond by saying that the trials missed by their algorithm have been badly indexed, which is bad in itself. And they would be right about that. Trial sponsors should update clinicaltrials.gov with their results. They should also make sure that the clinicaltrials.gov ID number is included in the publication (although in several cases of published trials that were missed by the algorithm, the ID number was in fact included in the abstract of the paper, so this seems to be a fault of Medline indexing rather than any fault of the triallists).

However, the claim made by the Trials Tracker is not that trials are badly indexed. If they stuck to making only that claim, then the Trials Tracker would be a perfectly worthy and admirable project. But the problem is they go beyond that, and claim something which their data simply do not show. Their claim is that the trials are undisclosed. This is just wrong. It is another example of what seems to be all the rage these days, namely “post-truth politics”. It is no different from when the Brexit campaign said “We spend £350 million a week on the EU and could spend it on the NHS instead” or when Donald Trump said, well, pretty much every time his lips moved really.

Welcome to the post-truth world.

 

Sugar tax

One of the most newsworthy features of yesterday’s budget was the announcement that the UK will introduce a tax on sugary drinks.

There is reason to think this may have primarily been done as a dead cat move, to draw attention away from the fact the the Chancellor is missing all his deficit reduction targets and cutting disability benefits (though apparently he can still afford tax cuts for higher rate tax payers).

But what effect might a tax on sugary drinks have?

Obviously it will reduce consumption of sugary drinks:  it’s economics 101 that when the price of something goes up, consumption falls. But that by itself is not interesting or useful. The question is what effect will that have on health and well-being?

The only honest answer to that is we don’t know, as few countries have tried such a tax, and we do not have good data on what the effects have been in countries that have.

For millionaires such as George Osborne and Jamie Oliver, the tax is unlikely to make much difference. Sugary drinks are such a tiny part of their expenditure, they will probably not notice.

But what about those at the other end of the income scale? While George Osborne may not realise this, there are some people for whom the weekly grocery shop is a significant proportion of their total expenditure. For such people, taxing sugary drinks may well have a noticeable effect.

For a family who currently spends money on sugary drinks, 3 outcomes are possible.

The first possibility is that they continue to buy the same quantity of sugary drinks as before (or sufficiently close to the same quantity that their total expenditure still rises). They will then be worse off, as they will have less money to spend on other things. This is bad in itself, but also one of the strongest determinants of ill health is poverty, so taking money away from people is unlikely to make them healthier.

The second possibility is that they reduce their consumption of sugary drinks by an amount roughly equivalent to the increased price. They will then be no better or worse off in terms of the money left in their pocket after the weekly grocery shopping, but they will be worse off in welfare terms, as they will have less of something that they value (sugary drinks). We know that they value sugary drinks, because if they didn’t, they wouldn’t buy them in the first place.

Proponents of the sugar tax will argue that they will be better off in health terms, as sugary drinks are bad for you, and they are now consuming less of them. Well, maybe. But that really needs a great big [citation needed]. This would be a relatively modest decrease in sugary drink consumption, and personally I would be surprised if it made much difference to health. There is certainly no good evidence that it would have benefits on health, and given that you are harming people by depriving them of something they value, I think it is up to proponents of the sugar tax to come up with evidence that the benefits outweigh those harms. It seems rather simplistic to suppose that obesity, diabetes, and the other things the the sugar tax is supposed to benefit are primarily a function of sugary drink consumption, when there are so many other aspects of diet, and of course exercise, which the sugar tax will not affect.

The third possibility is that they reduce their consumption by more than the amount of the price increase. They will now have more money in their pocket at the end of the weekly grocery shop. Perhaps they will spend that money on vegan tofu health drinks and gym membership, and be healthier as a result, as the supporters of the sugar tax seem to believe. Or maybe they’ll spend it on cigarettes and boiled sweets. We simply don’t know, as there are no data to show what happens here. The supposed health benefits of the sugar tax are at this stage entirely hypothetical.

But whatever they spend it on, they would have preferred to spend it on sugary drinks, so we are again making them worse off in terms of the things that they value.

All these considerations are trivial for people on high incomes. They may not be for people on low incomes. What seems certain is that the costs of the sugar tax will fall disproportionately on the poor.

You may think that’s a good idea. George Osborne obviously does. But personally, I’m not a fan of regressive taxation.

The amazing magic Saatchi Bill

Yesterday saw the dangerous and misguided Saatchi Bill (now reincarnated as the Access to Medical Treatments (Innovation) Bill) debated in the House of Commons.

The bill started out as an attempt by the Conservative peer Lord Saatchi to write a new law to encourage innovation in medical research. I have no doubt that the motivation for doing so was based entirely on good intentions, but sadly the attempt was badly misguided. Although many people explained to Lord Saatchi why he was wrong to tackle the problem in the way he did, it turns out that listening to experts is not Saatchi’s strong suit, and he blundered on with his flawed plan anyway.

If you want to know what is wrong with the bill I can do no better than direct you to the Stop the Saatchi Bill website, which explains the problems with the bill very clearly. But briefly, it sets out to solve a problem that does not exist, and causes harm at the same time. It attempts to promote innovation in medical research by removing the fear of litigation from doctors who innovate, despite the fact that fear of litigation is not what stops doctors innovating. But worse, it removes important legal protection for patients. Although the vast majority of doctors put their patients’ best interests firmly at the heart of everything they do, there will always be a small number of unscrupulous quacks who will be only too eager to hoodwink patients into paying for ineffective or dangerous treatments if they think there is money in it.

If the bill is passed, any patients harmed by unscrupulous quacks will find it harder to get redress through the legal system. That does not protect patients.

Although the bill as originally introduced by Saatchi failed to make sufficient progress through Parliament, it has now been resurrected in a new, though essentially similar, form as a private member’s bill in the House of Commons.

I’m afraid to say that the debate in the House of Commons did not show our lawmakers in a good light.

We were treated to several speeches by people who clearly either didn’t understand what the bill was about or were being dishonest. The two notable exceptions were Heidi Alexander, the Shadow Health Secretary, and Sarah Wollaston, chair of the Health Select Committee and a doctor herself in a previous career. Both Alexander and Wollaston clearly showed that they had taken the trouble to read the bill and other relevant information carefully, and based their contributions on facts rather than empty rhetoric.

I won’t go into detail on all the speeches, but if you want to read them you can do so in Hansard.

The one speech I want to focus on is by George Freeman, the Parliamentary Under-Secretary of State for Life Sciences. As he is a government minister, his speech gives us a clue about the government’s official thinking on the bill. Remember that it is a private member’s bill, so government support is crucial if it is to have a chance of becoming law. Sadly, Freeman seems to have swallowed the PR surrounding the bill and was in favour of it.

Although Freeman said many things, many of which showed either a poor understanding of the issues or blatant dishonesty, the one I particularly want to focus on is where he imbued the bill with magic powers.

He repeated the myths about fear of litigation holding back medical research. He was challenged in those claims by both Sarah Wollaston and Heidi Alexander.

When he reeled off a whole bunch of statistics about how much money medical litigation cost the NHS, Wollaston asked him how much of that was specifically related to complaints about innovative treatments. His reply was telling:

“Most of the cases are a result of other contexts— as my hon. Friend will know, obstetrics is a big part of that—rather than innovation. I am happy to write to her with the actual figure as I do not have it to hand.”

Surely that is the one statistic he should have had to hand if he’d wanted to appear even remotely prepared for his speech? What is the point of being able to quote all sorts of irrelevant statistics about the total cost of litigation in the NHS if he didn’t know the one statistic that actually mattered? Could it be that he knew it was so tiny it would completely undermine his case?

He then proceeded to talk about the fear of litigation, at which point Heidi Alexander asked him what evidence he had. He had to admit that he had none, and muttered something about “anecdotally”.

But anyway, despite having failed to make a convincing case that fear of litigation was holding back innovation, he was very clear that he thought the bill would remove that fear.

And now we come to the magic bit.

How exactly was that fear of litigation to be removed? Was it by changing the law on medical negligence to make it harder to sue “innovative” doctors? This is what Freeman said:

“As currently drafted the Bill provides no change to existing protections on medical negligence, and that is important. It sets out the power to create a database, and a mechanism to make clear to clinicians how they can demonstrate compliance with existing legal protection—the Bolam test has been referred to—and allow innovations to be recorded for the benefit of other clinicians and their patients. Importantly for the Government, that does not change existing protections on medical negligence, and it is crucial to understand that.”

So the bill makes no change whatsoever to the law on medical negligence, but removes the fear that doctors will be sued for negligence. If you can think of a way that that could work other than by magic, I’m all ears.

In the end, the bill passed its second reading by 32 votes to 19. Yes, that’s right: 599 well over 500* MPs didn’t think protection of vulnerable patients from unscrupulous quacks was worth turning up to vote about.

I find it very sad that such a misguided bill can make progress through Parliament on the basis of at best misunderstandings and at worst deliberate lies.

Although the bill has passed its second reading, it has not yet become law. It needs to go through its committee stage and then return to the House of Commons for its third reading first. It is to be hoped that common sense will prevail some time during that process, or patients harmed by unscrupulous quacks will find that the law does not protect them as much as it does now.

If you want to write to your MP to urge them to turn up and vote against this dreadful bill when it comes back for its third reading, now would be a good time.

* Many thanks to @_mattl on Twitter for pointing out the flaw in my original figure of 599: I hadn’t taken into account that the Speaker doesn’t vote, the Tellers aren’t counted in the totals, Sinn Fein MPs never turn up at all, and SNP MPs are unlikely to vote as this bill doesn’t apply to Scotland.

Equality of opportunity

Although this is primarily a blog about medical stuff, I did warn you that there might be the occasional social science themed post. This is one such post.

In his recent speech to the Conservative Party conference, David Cameron came up with many fine words about equality of opportunity. He led us to believe that he was for it. Here is an extract from the relevant part of his speech:

If we tackle the causes of poverty, we can make our country greater.

But there’s another big social problem we need to fix.

In politicians’ speak: a “lack of social mobility”.

In normal language: people unable to rise from the bottom to the top, or even from the middle to the top, because of their background.

Listen to this: Britain has the lowest social mobility in the developed world.

Here, the salary you earn is more linked to what your father got paid than in any other major country.

I’m sorry, for us Conservatives, the party of aspiration, we cannot accept that.

We know that education is the springboard to opportunity.

Fine words indeed. Cameron is quite right to identify lack of social mobility as a major problem. It cannot be right that your life chances should depend so much on who your parents are.

Cameron is also quite right to highlight the important role of education. Inequality of opportunity starts at school. If you have pushy middle class parents who get you into a good school, then you are likely to do better than if you have disadvantaged parents and end up in a poor school.

But it is very hard to reconcile Cameron’s fine words with today’s announcement of a new grammar school. In theory, grammar schools are supposed to aid social mobility by allowing bright kids from disadvantaged backgrounds to have a great education.

But in practice, they do no such thing.

In practice, grammar schools perpetuate social inequalities. Grammar schools are largely the preserve of the middle classes. According to research from the Institute for Fiscal studies, children from disadvantaged backgrounds are less likely than their better off peers to get into grammar schools, even if they have the same level of academic achievement.

It’s almost as if Cameron says one thing but means something else entirely, isn’t it?

If Cameron is serious about equality of opportunity, I have one little trick from the statistician’s toolkit which I think could help, namely randomisation.

My suggestion is this. All children should be randomly allocated to a school. Parents would have no say in which school their child goes to: it would be determined purely by randomisation. The available pool of schools would of course need to be within reasonable travelling distance of where the child lives, but that distance could be defined quite generously, so that you wouldn’t still have cosy middle class schools in cosy middle class neighbourhoods and poor schools in disadvantaged neighbourhoods.

At the moment, it is perfectly accepted by the political classes that some schools are good schools and others are poor. Once the middle classes realise that their own children might have to go to the poor schools, my guess is that the acceptance of the existence of poor schools would rapidly diminish. Political pressure would soon make sure that all schools are good schools.

That way, all children would have an equal start in life, no matter how rich their parents were.

This suggestion is, of course, pure fantasy. There is absolutely no way that our political classes would ever allow it. Under a system like that, their own children might have to go to school with the plebs, and that would never do, would it?

But please don’t expect me to take any politician seriously if they talk about equality of opportunity on the one hand but still support a system in which the school that kids go to is determined mainly by the socioeconomic status of their parents.

Energy prices rip off

Today we have learned that the big six energy providers have been overcharging customers to the tune of over £1 billion per year.

Obviously your first thought on this story is “And what will we learn next week? Which religion the Pope follows? Or perhaps what do bears do in the woods?” But I think it’s worth taking a moment to think about why the energy companies have got away with this, and what might be done about it.

Energy companies were privatised by the Thatcher government back in the late 1980s and early 1990s, based on the ideological belief that competition would make the market more efficient. I’m not sure I’d call overcharging consumers by over £1 billion efficient.

It’s as if Thatcher had read the first few pages of an economics textbook that talks about the advantages of competition and the free market, and then gave up on the book without reading the rest of it to find out what can go wrong with free markets in practice.

Many things can go wrong with free markets, but the big one here is information asymmetry. It’s an important assumption of free market competition that buyers and sellers have perfect information. If buyers do not know how much something is costing them, how can they choose the cheapest supplier?

It is extraordinarily difficult to compare prices among energy suppliers. When I last switched my energy supplier, I spent well over an hour constructing a spreadsheet to figure out which supplier would be cheapest for me. And I’m a professional statistician, so I’m probably better equipped to do that task than most.

Even finding out the prices is a struggle. Here is what I was presented with after I looked on NPower’s website to try to find the prices of their energy:

Screenshot from 2015-07-07 08:24:19

It seems that they want to know everything about me before they’ll reveal their prices. And I’d already had to give them my postcode before I even got that far. Not exactly transparent, is it?

It was similarly impossible to find out Eon’s prices without giving them my entire life history. EDF and SSE were a bit more transparent, though both of them needed to know my postcode before they’d reveal their prices.

Here are EDF’s rates:

Screenshot from 2015-07-07 08:31:06

And here are SSE’s rates:

Screenshot from 2015-07-07 08:30:20

Which of those is cheaper? Without going through that spreadsheet exercise, I have no idea. And that’s just the electricity prices. Obviously I have to do the same calculations for gas, and given that they all give dual fuel discounts, I then have to calculate a total as well as figuring out whether I would be better off going with separate suppliers for gas and electricity to take the cheapest deal on each and whether that would compensate for the dual fuel discount.

And then of course I also have to take into account how long prices are fixed for, what the exit charges are, etc etc.

Seriously, if I as a professional statistician find this impossibly difficult, how is anyone else supposed to figure it out? There are price comparison websites that are supposed to help people compare prices, but of course they have to make a living, and have their own problems.

It’s no wonder that competition is not working for the benefit of consumers.

So what is to be done about it?

I think there is a simple solution here. All suppliers should be required to charge in a simple and transparent way. The standing charge should go. Suppliers should be required simply to quote a price per unit, and should also be required to publish those prices prominently on their website without consumers having to give their inside leg measurements first. If different rates are given for day and night use, a common ratio of day rate to night rate should be required (the ratio used could be reviewed annually in response to market conditions).

Suppliers will no doubt argue that a flat price per unit is inefficient, as there are costs involved in simply having a customer even before any energy is used, and a customer who uses twice as much energy as another does not cost them twice as much.

Tough. The energy companies have had over 20 years to sort out their act, and have failed. While I’m not a fan of governments intervening in markets as a general principle, there are times when it is useful, and this is one of them. I don’t see how anyone can argue that an industry that overcharges consumers by over £1 billion per year is efficient. No one energy company would be at  a disadvantage, as all their competitors would be in the same position.

There would be a further benefit to this idea, in that it would add an element of progressiveness to energy pricing. At the moment, poor people who don’t use much energy pay more per unit than rich people. That doesn’t really seem fair, does it?

This is such a simple and workable idea it is hard to understand why it hasn’t already been implemented. Unless, of course, recent governments were somehow on the side of big business and cared far less about ordinary consumers.

But that can’t be true, can it?

What my hip tells me about the Saatchi bill

I have a hospital appointment tomorrow, at which I shall have a non-evidence-based treatment.

This is something I find somewhat troubling. I’m a medical statistician: I should know about evidence for the efficacy of medical interventions. And yet even I find myself ignoring the lack of good evidence when it comes to my own health.

I have had pain in my hip for the last few months. It’s been diagnosed by one doctor as trochanteric bursitis and by another as gluteus medius tendinopathy. Either way, something in my hip is inflammed, and is taking longer than it should to settle down.

So tomorrow, I’m having a steroid injection. This seems to be the consensus among those treating me. My physiotherapist was very keen that I should have it. My GP thought it would be a good idea. The consultant sports physician I saw last week thought it was the obvious next step.

And yet there is no good evidence that steroid injections work. I found a couple of open label randomised trials which showed reasonably good short-term effects for steroid injections, albeit little evidence of benefit in the long term. Here’s one of them. The results look impressive on a cursory glance, but something that really sticks out at me is that the trials weren’t blinded. Pain is subjective, and I fear the results are entirely compatible with a placebo effect. Perhaps my literature searching skills are going the same way as my hip, but I really couldn’t find any double-blind trials.

So in other words, I have no confidence whatsoever that a steroid injection is effective for inflammation in the hip.

So why am I doing this? To be honest, I’m really not sure. I’m bored of the pain, and even more bored of not being able to go running, and I’m hoping something will help. I guess I like to think that the health professionals treating me know what they’re doing, though I really don’t see how they can know, given the lack of good evidence from double blind trials.

What this little episode has taught me is how powerful the desire is to have some sort of treatment when you’re ill. I have some pain in my hip, which is pretty insignificant in the grand scheme of things, and yet even I’m getting a treatment which I have no particular reason to think is effective. Just imagine how much more powerful that desire must be if you’re really ill, for example with cancer. I have no reason to doubt that the health professionals treating me are highly competent and well qualified professionals who have my best interests at heart. But it has made me think how easy it must be to follow advice from whichever doctor is treating you, even if that doctor might be less scrupulous.

This has made me even more sure than ever that the Saatchi bill is a really bad thing. If a medical statistician who thinks quite carefully about these things is prepared to undergo a non-evidence-based treatment for what is really quite a trivial condition, just think how much the average person with a serious disease is going to be at the mercy of anyone treating them. The last thing we want to do is give a free pass for quacks to push completely cranky treatments at anyone who will have them.

And that’s exactly what the Saatchi bill will facilitate.

Hospital special measures and regression to the mean

Forgive me for writing 2 posts in a row about regression to the mean. But it’s an important statistical concept, which also happens to be widely misunderstood. Sometimes with important consequences.

Last week, I blogged about a claim that student tuition fees had not put off disadvantaged applicants. The research was flawed, because it defined disadvantage on the basis of postcode areas, and not on the individual characteristics of applicants. This means that an increase in university applications from disadvantaged areas could have simply been due to regression to the mean (ie the most disadvantaged areas becoming less disadvantaged) rather than more disadvantaged individual students applying to university.

Today, we have a story in the news where exactly the same statistical phenomenon is occurring. The story is that putting hospitals into “special measures” has been effective in reducing their death rates, according to new research by Dr Foster.

The research shows no such thing, of course.

The full report, “Is [sic] special measures working?” is available here. I’m afraid the authors’ statistical expertise is no better than their grammar.

The research looked at 11 hospital trusts that had been put into special measures, and found that their mortality rates fell faster than hospitals on average. They thus concluded that special measures were effective in reducing mortality.

Wrong, wrong, wrong. The 11 hospital trusts had been put into special measures not at random, but precisely because they had higher than expected mortality. If you take 11 hospital trusts on the basis of a high mortality rate and then look at them again a couple of years later, you would expect the mortality rate to have fallen more than in other hospitals simply because of regression to the mean.

Maybe those 11 hospitals were particularly bad, but maybe they were just unlucky. Perhaps it’s a combination of both. But if they were unusually unlucky one year, you wouldn’t expect them to be as unlucky the next year. If you take the hospitals with the worst mortality, or indeed the most extreme examples of anything, you would expect it to improve just by chance.

This is a classic example of regression to the mean. The research provides no evidence whatsoever that special measures are doing anything. To do that, you would need to take poorly performing hospitals and allocate them at random either to have special measures or to be in a control group. Simply observing that the worst trusts got better after going into special measures tells you nothing about whether special measures were responsible for the improvement.

Student tuition fees and disadvantaged applicants

Those of you who have known me for a while will remember that I used to blog on the now defunct Dianthus Medical website. The Internet Archive has kept some of those blogposts for posterity, but sadly not all of them. As I promised when I started this blog, I will get round to putting all those posts back on the internet one of these days, but I’m afraid I haven’t got round to that just yet.

But in the meantime, I’m going to repost one of those blogposts here, as it has just become beautifully relevant again. About this time last year, UCAS (the body responsible for university admissions in the UK) published a report which claimed to show that applications to university from disadvantaged young people  were increasing proportionately more than applications from the more affluent, or in other words, the gap between rich and poor was narrowing.

Sadly, the report showed no such thing. The claim was based on a schoolboy error in statistics.

Anyway, UCAS have recently published their next annual report. Again, this claims to show that the gap between rich and poor is narrowing, but doesn’t. Again, we see the same inaccurate headlines in the media that naively take the report’s conclusions at face value, and we see exactly the same schoolboy error in the way the statistics were analysed in the report.

So as what I wrote last year is still completely relevant today, here goes…

One of the most significant political events of the current Parliament has been the huge increase in student tuition fees, which mean that most university students now need to pay £9000 per year for their education.

One of the arguments against this rise used by its opponents was that it would put off young people from disadvantaged backgrounds from applying to university. Supporters of the new system argued that it would not, as students can borrow the money via a student loan to be paid back over a period of decades, so no-one would have to find the money up front.

The new fees came into effect in 2012, so we should now have some empirical data that should allow us to find out who was right. So what do the statistics show? Have people from disadvantaged backgrounds been deterred from applying to university?

A report was published earlier this year by UCAS, the organisation responsible for handling applications to university. This specifically addresses the question of applications from disadvantaged areas. This shows (see page 17 of the report) that although there was a small drop in application rates from the most disadvantaged areas immediately after the new fees came into effect, from 18.0% in 2011 to 17.5% in 2012, the rates have since risen to 20.5% in 2014. And the ratio of the rate of applications from the most advantaged areas to the most disadvantaged areas fell from 3.0 in 2011 to 2.5 in 2014.

So, case closed, then? Clearly the new fees have not stopped people from disadvantaged areas applying to university?

Actually, no. It’s really not that simple. You see, there is a big statistical problem with the data.

That problem is known as regression to the mean. This is a tendency of characteristics with particularly high or low values to become more like average values over time. It’s something we know all about in clinical trials, and is one of the reasons why clinical trials need to include control groups if they are going to give reliable data. For example, in a trial of a medication for high blood pressure, you would expect patients’ blood pressure to decrease during the trial no matter what you do to them, as they had to have high blood pressure at the start of the trial or they wouldn’t have been included in it in the first place.

In the case of the university admission statistics, the specific problem is the precise way in which “disadvantaged areas” and “advantaged areas” were defined.

The advantage or disadvantage of an area was defined by the proportion of young people participating in higher education during the period 2000 to 2004. Since the “disadvantaged” areas were specifically defined as those areas that had previously had the lowest participation rates, it is pretty much inevitable that those rates would increase, no matter what the underlying trends were.

Similarly, the most advantaged areas were almost certain to see decreases in participation rates (at least relatively speaking, though this is somewhat complicated by the fact that overall participation rates have increased since 2004).

So the finding that the ratio of applications from most advantaged areas to those from least advantaged areas has decreased was exactly what we would expect from regression to the mean. I’m afraid this does not provide evidence that the new tuition fee regime has been beneficial to people from disadvantaged backgrounds. It is very had to disentangle any real changes in participation rates from different backgrounds from the effects of regression to the mean.

Unless anyone can point me to any better statistics on university applications from disadvantaged backgrounds, I think the question of whether the new tuition fee regime has helped or hindered social inequalities in higher education remains open.

The Saatchi Bill

I was disappointed to see yesterday that the Saatchi Bill (or Medical Innovations Bill, to give it its official name) passed its third reading in the House of Lords.

The Saatchi Bill, if passed, will be a dreadful piece of legislation. The arguments against it have been well rehearsed elsewhere, so I won’t go into them in detail here. But briefly, the bill sets out to solve a problem that doesn’t exist, and then offers solutions that wouldn’t solve it even if it did exist.

It is based on the premise that the main reason no progress is ever made in medical research (which is nonsense to start with, of course, because progress made all the time) is because doctors are afraid to try innovative treatments in case they get sued. There is, however, absolutely no evidence that that’s true, and in any case, the bill would not help promote real innovation, as it specifically excludes the use of treatments as part of research. Without research, there is no meaningful innovation.

If the bill were simply ineffective, that would be one thing, but it’s also actively harmful. By removing the legal protection that patients  currently enjoy against doctors acting irresponsibly, the bill will be a quack’s charter. It would certainly make it more likely that someone like Stanislaw Burzynski, an out-and-out quack who makes his fortune from fleecing cancer patients by offering them ineffective and dangerous treatments, could operate legally in the UK. That would not be a good thing.

One thing that has struck me about the sorry story of the Saatchi bill is just how dishonest Maurice Saatchi and his team have been. A particularly dishonourable mention goes to the Daily Telegraph, who have been the bill’s “official media partner“. Seriously? Since when did bills going through parliament have an official media partner? Some of the articles they have written have been breathtakingly dishonest. They wrote recently that the bill had “won over its critics“,  which is very far from the truth. Pretty much the entire medical profession is against it: this response from the Academy of Royal Medical Colleges is typical. The same article says that one way the bill had won over its critics was by amending it to require that doctors treating patients under this law must publish their research. There are 2 problems with that: first, the law doesn’t apply to research, and second, it doesn’t say anything about a requirement to publish results.

In an article in the Telegraph today, Saatchi himself continued the dishonesty. As well as continuing to pretend that the bill is now widely supported, he also claimed that more than 18,000 patients responded to the Department of Health’s consultation on the bill. In fact, the total number of responses to the consultation was only 170.

The dishonesty behind the promotion of the Saatchi bill has been well documented by David Hills (aka “the Wandering Teacake”), and I’d encourage you to read his detailed blogpost.

The question that I want to ask about all this is why? Why is Maurice Saatchi doing all this? What does he have to gain from promoting a bill that’s going to be bad for patients but good for unscrupulous quacks?

I cannot know the answers to any of those questions, of course. Only Saatchi himself can know, and even he may not really know: we are not always fully aware of our own motivations. The rest of us can only speculate. But nonetheless, I think it’s interesting to speculate, so I hope you’ll bear with me while I do so.

The original impetus for the Saatchi bill came when Saatchi lost his wife to ovarian cancer. Losing a loved one to cancer is always difficult, and ovarian cancer is a particularly nasty disease. There can be no doubt that Saatchi was genuinely distressed by the experience, and deserves our sympathy.

No doubt it seemed like a good idea to try to do something about this. After all, as a member of the House of Lords, he has the opportunity to propose new legislation. It is completely understandable that if he thought a new law could help people who were dying of cancer, he would be highly motivated to introduce one.

All of that is very plausible and easy to understand. What has happened subsequently, however, is a little harder to understand.

It can’t have been very long after Saatchi proposed the bill that many people who know more about medicine than he does told him why it simply wouldn’t work, and would have harmful consequences. So I think what is harder to understand is why he persisted with the bill after all the problems with it had been explained to him.

It has been suggested that this is about personal financial gain: his advertising company works for various pharmaceutical companies, and pharmaceutical companies will gain from the bill.

However, I don’t believe that that is a plausible explanation for Saatchi’s behaviour.

For a start, I’m pretty sure that the emotional impact of losing a beloved wife is a far stronger motivator than money, particularly for someone who is already extremely rich. It’s not as if Saatchi needs more money. He’s already rich enough to buy the support of a major national newspaper and to get a truly dreadful bill through parliament.

And for another thing, I’m not at all sure that pharmaceutical companies would do particularly well out of the bill anyway. They are mostly interested in getting their drugs licensed so that they can sell them in large quantities. Selling them as a one-off to individual patients is unlikely to be at the top of their list of priorities.

For what it’s worth, my guess is that Saatchi just has difficulty admitting that he was wrong. It’s not a particularly rare personality trait. He originally thought the bill would genuinely help cancer patients, and when told otherwise, he simply ignored that information. You might see this as an example of the Dunning Kruger effect, and it’s certainly consistent with the widely accepted phenomenon of confirmation bias.

Granted, what we’re seeing here is a pretty extreme case of confirmation bias, and has required some spectacular dishonesty on the part of Saatchi to maintain the illusion that he was right all along. But Saatchi is a politician who originally made his money in advertising, and it would be hard to think of 2 more dishonest professions than politics and advertising. It perhaps shouldn’t be too surprising that dishonesty is something that comes naturally to him.

Whatever the reasons for Saatchi’s insistence on promoting the bill in the face of widespread opposition, this whole story has been a rather scary tale of how money and power can buy your way through the legislative process.

The bill still has to pass its third reading in the House of Commons before it becomes law. We can only hope that our elected MPs are smart enough to see what a travesty the bill is. If you want to write to your MP to ask them to vote against the bill, now would be a good time to do it.