Wednesday, November 28, 2012

Peer regard for pot noodles: a critique of some qualitative research

Pot Noodles, Placements and Peer Regard: Creative Career Trajectories and Communities of Practice in the British Advertising Industry was published in the prestigious British Journal of Management in 2010. I chose it for the title, which looked fun, so from the point of view of the quality of the research, which is what I am interested in here, it is a more or less random choice. I think it is typical of many “qualitative” empirical papers in the literature.


“They can’t be serious,” was the first reaction of one of the students I asked to look critically at this paper. But of course they are, unfortunately. This paper does read like a parody of itself. The main difficulties are the unnecessarily convoluted language, the fact that some of the conclusions are blindingly obvious, the vagueness of the sample on which the research is based, the lack of any real information about whether the conclusions always apply or sometimes apply (and if so how often), or indeed any satisfactory audit trail to link the conclusions with the data on which they are supposedly based. And the conclusions are so vague it’s actually difficult to see what they are.

In short, I think this paper is a complete waste of time. Which is a pity, because a study of careers in the advertising industry could be very interesting and useful from the perspective of various stakeholders – the advertising creatives themselves, those managing the agencies, and society in general. Furthermore very similar issues are likely to apply to other business areas. The topic is a good one, worth researching.

The initial problem from the students’ point of view is the language used. For example: “People learn by participating in the shared practices of a community or ‘lived in world’ (Fuller and Unwin, 1998). A community of practice is a collectively developed understanding of the nature and identity of the community to which its members are accountable. It is sustained through norms and relationships of mutuality and a shared repertoire of communal resources, language routines, artefacts, tools and stories. Those who are involved in a particular ‘community’ understand its limits or boundaries. What is learned by participants within a community is identity formation, rather than knowledge per se: according to Lave and Wenger (1991, p. 53), ‘[l]earning implies becoming a different person with respect to the possibilities enabled by these systems of relations’. It is the gradual construction of an identity and learning to talk within a practice (rather than about it) that allows the novice to become part of a community.” (p. 115)

Does this mean “a community has its own know-how, jargon, assumptions and habits which people pick up as they participate”? If so why not use simpler language like this? And isn’t it actually rather obvious? Any group has its own jargon and idiosyncracies which beginners need to learn. We all know this. We don’t need research to tell us. Isn’t this just translating the obvious into long words so we don’t realize it’s obvious? Or have I misunderstood because the words are too long for me?

What conclusions does the article come to? The first paragraph of the concluding section is
“The focus of the paper has been on how career trajectories unfold within the advertising industry as a community of practice. It is within this context that anticipatory socialization, situated learning, and the move from periphery to centre takes place. Advertising's creative community is not simply a backdrop for learning: individuals created and developed the community through social action, and this inter-relationship was crucial in shaping career trajectories.”
And the last sentence of the article is:
“Nonetheless, this study of advertising creatives suggests that, although modern careers may be individualized undertakings, they increasingly unfold within norms and practices of multiple, inter-related occupational communities.”

These extracts give a good flavour of the style of conclusions derived from the research. But … surely all careers “unfold” within “multiple, inter-related occupational communities”, all communities are created by individuals “through social action”, and people with successful careers usually have little choice but to start on the periphery of their chosen field and then try to move to the centre. And so on.  In short, how could things be otherwise? We really do not need research to tell us this!

Yes, but, you might say, perhaps advertising creatives’ careers involve more interaction with multiple communities, and “anticipatory socialization” and “situated learning” are more important than they are in other careers. Perhaps. We are not told. These things are not measured so there is no possibility of comparison. Besides their obviousness, the lack of any indication of the magnitude of any of the features described, or their importance compared with other contexts, makes the conclusions too vague to be interesting.

We also really need more detail of how the research was done. A sample of 34 creatives was interviewed, which is fair enough. But who were they? We are told the sampling was “informed” by industry knowledge, and various selection criteria to get a sample of creatives in “different situations”. But we are not told how this was done, or any of the details that are necessary to understand the composition of the sample. A different sample would lead to different conclusions: the reader needs to be convinced that the sample is in some sense typical. This reader is not remotely close to being convinced.

Similarly, we are told about open coding, being “mindful of our theory”, a small set of key categories, theoretical codes, etc, etc. All the usual grounded theory stuff, but without any more detail to tell the reader what it actually means. And then we go straight into a narrative with statements like “creatives who had come into advertising through such specialist courses talked about the value of what they had learned in terms of craft skills, immersion in the community and insights into its employment patterns and work practices.” Does this mean all of them? Were they right? And where is the evidence, the codes, etc, etc? I find it hard to believe the grounded theory stuff was actually carried out, and harder still to believe it had any value. There’s certainly no trace of it in the paper.

Peer reviewed journal papers: are there any good ones?

Had a class yesterday when I got the MBA students to do critiques of papers in academic management journals with a view to making them more critical of what they read, and appreciating some of the problems they are likely to encounter in their own research.

First problem was to choose some suitable papers: hopefully some good, some bad and some OK but with obvious possibilities for improvement. Unfortunately I really couldn’t find any good ones. It’s easy enough to find examples of bad articles (one of my favourites is Can total quality management make small firms competitive?), but I haven’t managed to find any good articles on ordinary empirical research in management. There doubtless are articles putting forward interesting models and theories, and there are good articles in other fields like medicine, but I have yet to find what I consider a good published example of the kind of empirical research we expect our MBA students to do.

Last time I ran this session one of the student complained, very reasonably, that I had not provided an example of good practice to follow, so I sent round a circular to my colleagues asking for suggestions, but none of their suggestions really fitted the bill. So I’m still trying to teach the students how to do good research, but without any models of good practice. My ideas about how research in this area should be done are so different from what is actually done that I can’t mention what I really think for fear of undermining the literature that am supposed to be encouraging the students to study. Help!

Empirical papers in the management research literature usually come in one of two styles – often called qualitative and quantitative. I don’t think this is a sensible or useful distinction (see Are ‘Qualitative’ and ‘Quantitative’ Useful Terms for Describing Research?) but it is the way the literature divides up, so I’ll deal with the two types separately in two future posts. And in a further post I’ll try and say what I think management research should look like.

Monday, November 26, 2012

PhDs

PhDs are meant to make a significant contribution to knowledge. Who checks this contribution to see if it is genuine, valid and useful? This task is performed by just two examiners – often colleagues or friends of the supervisor. I wonder if this process is always as rigorous as we pretend it is?

If contribution to knowledge could be meaningfully assessed – and I can’t believe in it myself – you’d need a team of independent, anonymous assessors, and advice from an epistemologist.