The May 2013 issue of Political Studies Review is a symposium edition on the impact and relevance of political science. The symposium is freely available to download until 31st December. Here, Dr Nicholas Allen from Royal Holloway responds to the symposium and looks at the issue of validity – is it possible to measure impact and is impact even a good measure of the quality of research?
Most political scientists do not embark upon an academic career to change the world. Nor, must it be said, are concerns about impact and social relevance at the forefront of most political scientists’ minds. For many of us, therefore, the demands of the 2014 Research Excellence Framework (REF), the latest periodic audit of research quality in British higher-education institutions, and the requirement to produce ‘impact case studies’ have been something of a wake-up call.
Impact was not a feature of the precursor to the REF, the 2008 Research Assessment Exercise, which focused primarily on outputs like books and articles. In the REF, however, some 20 per cent of our assessed research quality—which will determine the flow of money into universities’ coffers—will relate to “the ‘reach and significance’ of impacts on the economy, society and/or culture that were underpinned by excellent research”, as well as departments’ approach to enabling impact. This change reflects the climate of expectations in which academics now operate. Our political masters are keener than ever for us to demonstrate that we’re deserving of public money.
Behavioural economics and ‘nudge’ are supposedly all the rave in some government circles. The 2014 REF is in many ways a classic case of nudge at work. If most of us were only slightly aware of impact and engagement when we started our careers, we are all fully aware of it now. Political science, like other disciplines, has been nudged to consider new measures of excellence. Whether that nudging is desirable, or whether the chosen measures are valid, is, of course, another matter.
I am personally supportive of the idea that professional students of politics should focus on real-world issues and problems, broadly defined. But then I don’t know many colleagues who think otherwise. There are doubtless a few in our profession who deserve to be labelled ‘theory fetishists’ or, perhaps worse, ‘methodological masturbators’, to invoke two colourful phrases that appear in the Political Studies Review symposium. But the more I think about it, the more difficult it is to think of any colleague who has absolutely no wish to engage with the outside world.
Much of the research undertaken by British political scientists is almost certainly peripheral to many politicians’ and policy makers’ interests. However, I would never say that my own research lacks social or political relevance, and I suspect that virtually every colleague would never say the same about theirs (although some may well say it about mine). Relevance, like everything else, is in the eye of the beholder; yet no colleague, as far as I am aware, aspires to social irrelevance.
Any measures of impact that focus on political scientists’ attempts to engage non-academic audiences are probably desirable, in that they may nudge academics to work a little harder to do what most would like to do in the first place. (See Rosie Campbell and Sarah Childs for practical advice on how to be more effective in engaging practitioners and the public.) Measures of impact that go far beyond this are less desirable, however, which leads me to the issue of validity.
The problems surrounding the measurement of impact are discussed at length in the various contributions to the symposium. Of these, most are pretty basic. For instance, it is one thing to hypothesise that a single piece of research at time t can directly affect the behaviour, broadly defined, of an individual or individuals at t+1. It is another thing to demonstrate this beyond reasonable doubt. How can you control for other potential influences on behaviour? As Peter John sets out, most political-science research is only likely to have an indirect impact by influencing the wider climate of ideas in which politicians and other policy makers operate. Measuring such impact with any degree of objectivity is, of course, virtually impossible.
Moreover, having impact is rarely within the immediate control of an academic researcher in any event. Much depends on the willingness of policy makers or others to listen and take on board ideas. Research can be of a very high quality, but, as Ronald Rogowski puts it, the messenger can always be ignored or shot. In the circumstances, any measure of impact may really only be an indicator of the preferences and prejudices of a given audience.
Academic research that contributes to wider public understandings of important matters of public policy has impact by a common-or-garden understanding of the term, and deserves to be rewarded. Unfortunately, such work could well become discounted over the long term if policy makers nudge us into pursuing narrower work with more discernible impact. Whether the ‘impact agenda’ triggers a fundamental shift in the culture and ethos of British political science remains to be seen. Much will depend on how the Politics and International Studies REF sub-panel assesses politics departments’ impact case studies—examples of impact—and impact templates—how departments have tried to facilitate the achievement of impact. For this, and other reasons, we await their judgements with baited breath.