There’s currently a lot of talk about REF 2020 across pretty much all of our necks of the woods here in the UK, and inevitably this seems to entail the fine dissection of the still-warm body of 2014. One of the things that keeps on cropping up is the whole issue of ‘rigour’ and, in particular, what that means in actual practice. A discourse that I’ve heard a couple of times now in casual contexts is that ‘it basically means hard-core quants’. As a post-positivist researcher, that obviously concerns me quite a bit, and so here are my thoughts on rigour.
Rigour relates to questions of methodology and so I should begin by noting that I see methodology as super-super important. I just don’t think that it’s possible to separate questions of how we produce knowledge from questions of what knowledge is produced. Indeed, one of the things that is guaranteed to make me have an epic rant to students, colleagues, my parents, and/or random passers-by is to read the words ‘the research finds that’ or ‘the study shows’, without any mention of the methodology involved. It makes me go a big-bit crazy when the ‘findings’ or ‘results’ just hang there, as if they’re floating in space, when the reality is that those findings/results have been produced by actual, embodied and fallible researchers. I go crazy because (to get a bit shouty and arm-wavey here) I find it duplicitous and irresponsible – it’s like a magician trying to hide how they do the trick; except with researchers no-one thinks we’re magicians. Quite the contrary: there’s often a perception that academics are in the business of discovering reality/truths and then reporting those reality/truths – our ‘findings’ – to other academics/the media/policy-makers/our students. Except we haven’t just ‘found’ the truths; we have actively produced them through the process of undertaking our research. We need, therefore, to be clear about exactly how we have undertaken that research. This is especially important given that our findings can be (and often are) taken up as transparent ‘facts’ by policy-makers, as if those ‘facts’ aren’t in reality always open to contestation. So, researchers have a huge responsibility to be upfront and honest about how we have produced what we have ‘found’. We have a duty of care to take methodology seriously, and it’s an ethical and political one as well as an intellectual one.
And, in fact, all of the above is actually what I understand to be meant by ‘rigour’ in the 2014 REF document. The generic criteria state that: ‘Rigour will be understood as the extent to which the purpose of the work is clearly articulated, an appropriate methodology for the research area has been adopted, and compelling evidence presented to show that the purpose has been achieved’. And the Panel 2C criteria in turn states: ‘Rigour will be understood in terms of the intellectual precision, robustness and appropriateness of the concepts, analyses, theories and methodologies deployed within a research output. Account will be taken of such qualities as the integrity, coherence and consistency of arguments and analysis, such as the due consideration of ethical issues’. There is no subsequent sentence that then reads: ‘N.B. This means do statistics’. Indeed, I would argue that it is only in a world where methodology isn’t being taken seriously that ‘rigour’ might mean ‘we should just do hard-core statistics and then that’ll mean 4*’.
To continue with the question of quants, I should add that I am not an ‘anti-quants’ person. I believe both that feminists can count and that (as Sara Ahmed notes) ‘numbers can be affective‘. But we all know that, ‘even’ with quants, there is absolutely no inherent guarantee of ‘rigour’ – indeed, as someone who researches on the sex industry, I am particularly mindful of the huge damage to actual people that dodgy and so-called ‘authoritative’ statistics can do. But, more broadly, we just cannot let ourselves off the hook like that. In all research – using whatever method/s – researchers have to be really clear about the claims, commitments and consequences of that research (i.e. its core claims; its underlying commitments; and its theoretical, political and ethical consequences). This means reflecting very carefully and very openly not just about the ‘limitations’ of the research (e.g. ‘I would have done more interviews if …’) but also about its own silences and gaps. What, in short, are the power relations being reproduced in and through the research itself? This is a matter of reflexivity, which I would argue is a condition of possibility for the ‘rigour’ that the REF document is asking for. Put another way (as we already tell our dissertation and research students): if you haven’t engaged in open and sustained reflection about how and why you’ve done the research in the very particular way that you have decided to do it, then you’ve got yourself one shoddy research methodology. This is just as true for quants as for any other method, for a research method is not the same thing as a methodology, and the latter cannot simply be reduced to the former.
I would love us to shift the discourse from ‘rigour’ to ‘reflexivity’ but, as I don’t think that’s an argument I have any chance of winning, this is instead a plea for rigour in the application of ‘rigour’. We just can’t use ‘rigour’ as any kind of shorthand for a specific type of method. If we do it then it will be rigor mortis that we get: intellectual, political, and ethical. I am sorry for that pun – but rigour is not synonymous with ‘statistics’ and this distinction matters a great deal.