Being sceptical about scepticism

I have an arthritic knee. When I went to see the experts on bad knees they told me confidently to take a couple of supplements - glucosamine and chondroitin - which, they said, had been demonstrated to help bad knees. So I went to the chemist outside the hospital, bought some, and eventually convinced myself that the knee was better if I took the supplements.

Then I read The Sceptic in the Guardian who said that this was all wrong, that a new, well-designed trial with a large sample, had found that this was just wishful thinking. The people on the supplements "did no better than those on placebos". There was apparently just one slight glimmer of hope - "in the subgroup whose arthritis was particularly bad, the glucosamin chondroiting made them feel a little bit better." But the sceptic, unfortunately, thought this was just a fluke. I found a similar message on Quackwatch.

But then I decided to check the source, something academics are always supposed to do, but rarely bother with because most things have to be taken on trust. I don't have access to the full text of the article on the new trial, but the abstract of the article by D O Clegg and his 25 colleagues (yes - 25!) is on the web. And this flatly contradicts what I had been led to believe by the Sceptic and Quackwatch:

"As compared with the rate of response to placebo (60.1 percent), the rate of response to glucosamine was 3.9 percentage points higher (P=0.30), the rate of response to chondroitin sulfate was 5.3 percentage points higher (P=0.17), and the rate of response to combined treatment was 6.5 percentage points higher (P=0.09). The rate of response in the celecoxib control group was 10.0 percentage points higher than that in the placebo control group (P=0.008). For patients with moderate-to-severe pain at baseline, the rate of response was significantly higher with combined therapy than with placebo (79.2 percent vs. 54.3 percent, P=0.002)."

In other words the Sceptic and Quackwatch were wrong - people on glucosamine did better than people on the placebo, those on chondroitin did better still, and those on both did even better.

But why the misreporting of this result? I think the problem has to be the old one of misunderstanding statistics. The results for glucosamine, chondroitin and the combined treatment were not statistically significant at the conventional level of 5% - therefore the commentators say there is no effect.

Not so. There is an effect, but a small one which might plausibly be due to chance.

However, chance strikes me as an unlikely explanation here. The relative results of the placebo treatment, the two supplements and the combined treatments is just what I would expect if the supplements each had a beneficial effect on the knee. The placebo comes bottom, and the treatment with both goodies comes top. Just as we would expect. If it was just a matter of chance it's easy to work out that the probability of this happening is 8.5%. Not less than the magical 5% but close. And if we take into account the evidence of the trials that originally convinced the doctors to tell me to tuck into the supplements, and the fact that the evidence for the subgroup with bad arthritis was convincing (P=0.002), the evidence looks pretty strong that there is something useful going on here.

But it looks a small effect - just 6.5%. I'm not sure (having not read the article) exactly what this means, but it's about two thirds of the effect due to the painkiller. Which doesn't seem too bad to me.

So scepticism about the sceptic seems to be justified. I think I've convinced myself that glucosamine and chondroitin might well work! I've been taking them for the last few days.

And apparently there were few side-effects, and they aren't nasty drugs anyway... so perhaps I should try a double dose? Then, perhaps I can put back the clock and run a ten minute mile again?