Oh what rubbish a lot of this research is

I’ve just had to dip into the research literature again. Always depressing.

It’s easy to mock a lot of the qualitative style of research. Talk to a few friends and write it up with a liberal sprinkling of long words like social constructionism, hermeneutics, ethnography, phenomenology etc. Then it counts as more than the prejudices of your friends, or an excuse to visit gastro pubs (the focus of one colleague’s research), rather, it’s empirical data and every word is recorded and transcribed and taken very seriously.

But at least the reader (a rare beast in the research game but we have to believe they exist) gets to know something about the topic. They can read about the beer and the food. The so called positivist style of research is usually far worse, partly because positivists assume it’s so much better. I’ve just come across one article which is fairly typical. It’s on the newly discovered broccoli-carrot ratio as a predictor of gym membership.

According to the article the researchers looked at lots of possible variables to see if they could relate them to gym membership. Number of children, marital status, education level, and so on and so forth. The only significant relationship was with this ratio between the amount of broccoli consumed and the amount of carrot. The p value cited was less than 0.01% - so they obviously think it’s pretty important.

It isn’t of course. The significance level just tells you that it’s not a chance thing, but reading the detailed statistics the relationship is actually a very small one. People who eat more broccoli than carrot are ever so slightly more likely to belong to a gym. That’s it.

They make no comments on why it might be so, or on what use knowing about this relationship might be, or on whether there’s some causal mechanism driving those who eat more broccoli than carrot to the gym. And no mention at all that this is a tiny correlation. They assume that because it uses esoteric statistical methods which show it’s statistically significant it must be important.

Not so. And unfortunately this paper is typical of the stuff we’ve got to force our students to read in the name of academic rigour