By Troy Miller, Marketing and Communications Manager, Griffith Business School
With the inaugural assessment of engagement and impact for the Australian Research Council – Excellence in Research Awards (ERA) 2018, and a continuing focus on this area of assessment, some insights can be found in the UK where universities are rated on impact for their Research Excellence Framework (REF), in order to demonstrate the high quality and international standing of UK research and its contribution to society outside of academia.
An experienced REF assessor, Professor John Arnold from Loughborough University in the UK, recently visited Griffith and presented his insights, some examples of UK impact cases for the field of Business and Management Studies, and discussed the implications around impact. These cases were evaluated on two key components; Reach and Significance, and rated on a scale between unclassified, one-star and up to four-stars for the highest rating. These criteria are described more fully later.
Dr Arnold is Professor of Organisational Behaviour from the School of Business and Economics, the School’s Director of Research Impact, and a member of the School’s Research Centre for Professional Work and Society. It is his responsibility to try and foster research impact in the School of Business and Economics, and ensure Loughborough University performs well in the next REF in 2021, where a similar star rating will apply; four-star (world-leading), three-star (internationally excellent), two-star (internationally recognised), one-star (nationally recognised) and unclassified (below nationally recognised).
“Real effects on the real world – I think it is a plausible argument that the introduction of impact has had one of its desired effects which is to focus efforts on at least some of what we do in business schools, making a measurable difference outside academia.”
In 2014, four higher education funding bodies allocated around two billion pounds of research funding to UK universities and their aim was to support a dynamic and internationally competitive research sector that makes a major contribution to economic prosperity, national well-being and the expansion and dissemination of knowledge.
In Australia the federal government has recently adopted a similar approach, however the ERA’s impact rating scale is slightly different – with mechanisms to encourage the translation of research impacts beyond academia on a scale of; high (highly effective and well-integrated), medium (effective and integrated) and low (not effective and integrated) ratings – with the ultimate aim to improve the impact of research and its contribution to society that is similar to the UK’s purpose. Subsequently, there are important lessons that can be learned from UK’s impact cases and Professor Arnold’s hierarchy of impact indicators.
The presentation was moderated by Associate Professor Ki-Hoon Lee from the Department of Business Strategy and Innovation, who began by reminding the audience about the engagement and impact assessment being included in the Australian Research Council – Excellence in Research Awards going forward.
“Before we start I just want to highlight about the research impact and engagement issues. As you may know, the ERA assessment outcomes really started this year. From this year, business impact and engagement are included in the assessment,” said Associate Professor Lee.
“According to ERA, the engagement is the interaction between the researchers and the researcher’s power-ranges outside of academia for the mutually beneficial transfer of the knowledge, facilities, methods or resources. Research Impact, that is research maximised to the economy, to society, the environment or culture, beyond contribution to the academic solution.”
“This seminar is very timely, because we just received the ERA assessment outcomes (2018), especially including this impact and engagement outcomes. Griffith is very much in the middle-upper position in this impact and engagement assessment practice, and I believe there is much else we can improve on.”
“My aim is to give a brief overview of how research gets assessed in the UK, and it will feel fairly familiar to you, because I know you have got something pretty similar,” said Professor Arnold.
For UK universities one of the key reasons for accomplishing four-star impact scores in the REF is the financial benefits that are gained by higher-ranked universities, that evolve from the ability to attract future research funding and international students from countries such as China.
“The better you do in these assessments as a university, the more money you get. And because various newspapers and other agencies in the UK which compile league tables of universities take the REF scores into account, and weight them quite heavily, then it affects your league table position which in turn affects your ability to attract research funding and in particular overseas students,” said Professor Arnold.
UK REF Quality Profile – Business & Management
Professor Arnold explained the REF Assessment (2021) will be segmented again into three categories that contribute to the overall research quality score for a university. 1. Research outputs: In Business and Management Studies, this is mainly written: journal articles, books, chapters – weighting 60%. 2. Research impact: The demonstrable effects of research outside academia – weighting 25%. And 3. Research environment: Resources, activity and researcher development – weighting 15%. See Business and Management graph 2014.
“I am a seasoned campaigner in the sense, that in the last REF in 2014, I was on the Business and Management Assessing Panel, as indeed I was in 2008. But 2014 was the first time that impact came into it,” said Professor Arnold.
“So I have assessed a lot of cases that Business Schools make for the impact of their research. I have assessed lots of people’s impact, but I am not a seasoned campaigner in the sense that I have produced impact myself.”
“Each subject area within each university, and each university as a whole, gets a quality profile which indicates what percent of what we submitted counts as four-star (world-leading), what percent as three-star (internationally excellent), two-star (internationally recognised), one-star (nationally recognised) and unclassified (below nationally recognised).”
Professor Arnold outlined the wide variety of impact examples in the REF across six categories, and indicated that while he had seen cases across all categories from his school in 2014, his view was that Category 2 (economy, commercial organisational) was the most common, followed by Category 6 (public policy and law) and Category 5 (practitioners and professional services).
“Impact includes but not limited to, the activity, attitude, awareness, behaviour, etc. of audiences, beneficiaries, communities, and constituencies in any location. So it doesn’t have to be in the UK.”
“When I was an assessor in 2014, I assessed cases that fell into all of those categories from Business and Management Schools. And there is no requirement for impact to be in a certain sphere.”
“Impact has to be based on research done at the submitting university in the last 20 years, with the impact having occurred since 2013. The research underpinning the impact must be rated 2* or better. That’s a qualifying condition: 4* research is not necessary for a 4* impact case.”
To gain a better understanding of what goes into creating research impact performance, Professor Arnold has created a hierarchy of impact indicators that contribute in varying degrees towards impact scores, with element six (Outcome impact) providing higher scores for impact and element one (Dissemination) contributing to lower scores – other things being equal.
“I found it helpful having evaluated lots of cases and I think colleagues found it helpful, to think about almost like a hierarchy of impact, and the issue being how high up this hierarchy can you get with an impact case?” Professor Arnold asked.
UK REF Research Assessing Factors
In assessing research impact for 2021, the REF assessors will be considering two factors; 1. Reach: the extent and/or diversity of the beneficiaries of the impact as relevant to the nature of the impact, and 2. Significance: the degree to which the impact has enabled, enriched, influenced, informed or changed the performance, policies, practices, products, services, understanding, awareness or well-being of the beneficiaries.
“I noticed in REF 2014 there were quite a few impact cases which claimed impact on policy, and some of those were really strong – you could see the influence of the work in government green or white papers, even in legislation, and sometimes senior civil servants were prepared to write testimonials saying that it influenced their thinking, although that is quite difficult to get,” said Professor Arnold.
“Often though, some of the weaker cases would say well I had impact on policy and their evidence would be a series of references to where they were cited in various policy-related documents. But sometimes as an assessor, if you looked at those, they were one or two citations out of one hundred and fifty with no particular sign that this research had been specifically influential in the policy. So there was a lot of variation in the significance of different cases, particularly around policy.”
“Getting evidence of impact is often challenging and it’s all very well making claims, but how can you substantiate them?”
“But it’s very often through testimonials – very often the only way really you can say that you saved a company two million pounds or you affected their policy position, is to get someone senior in the company to say that,” said Professor Arnold.
“So a lot rests on relationships with the people in the context in which you have impact. And of course those people move on or the political climate changes and they don’t want to be aligned with something that was the previous government’s.”
In the UK, ownership of impact cases is granted to the institution over the researcher, which means if a researcher has a good history of cases and they move to a new institution, the impact cases remain with the founding institution. However, this is not the case for outputs (publications) that can travel with an academic to a new institution.
“But impact cases stay with the institution – so that limits the human capital value of having an impact case for your mobility as an individual academic.”
“Because you have got to spend some years at your next institution carrying on your research, carrying on your impact, before it can be claimed by your new institution.”
“Sometimes there are very obvious direct links between a research study or set of studies and impact. Most times however, it’s a little more diffuse than that,” said Professor Arnold.
“Right so this point which has been picked up – impact cases belong to the university, not the people who did the research. So although the rules are changing a bit, your outputs (your publications) if you move jobs, your publications go with you.”
Professor Arnold believes the introduction of research impact assessment has influenced UK Universities and Business Schools to contribute real-world practical outcomes outside of academia. And it has created staff development and career implications for the sector with a new specialism for professional services staff.
“I think there is a potential for non-research stars, by which I mean people that produce perfectly decent research, but they don’t hit the top journals. If they have impact, they can suddenly become very valuable for an institution. But the caveat is – they can’t take it with them,” said Professor Arnold.
“But on the other hand if your current employer is a bit short on impact cases and you are three years from the next REF and they really need you to keep working on this, they may be prepared to promote you or reward you in other ways. That is beginning to creep in.”
“But how valuable is impact for academics operating in an international labour market? I would suggest less so than top publications, but it may be changing slowly.”
“Along with some other things – like Research Councils requiring coverage of impact in grant proposals – I think it has made a difference. And it has made institutions more keen to do this, in collaboration with other organisations,” said Professor Arnold.
“As well as putting specific plans for impact in research proposals, there is a slow increase in attention paid to whether those plans come to fruition. There is more accountability at the back end of a research project than there used to be.”
“However, impact can also be seen as yet another managerial control on the activities of academics. Even though the REF does not require all academics to play the impact game, impact is becoming yet another performance measure with an infrastructure and therefore management cost to encourage (or enforce) impact activity.”
For more information on UK REF insights from Professor John Arnold please submit a request to Griffith’s Centre for Work, Organisation and Wellbeing through Ms Chantal Gallant or alternatively visit the REF 2014 Website or REF 2021 Website.
John Arnold is Professor of Organisational Behaviour from the School of Business and Economics, and a member of the School’s Research Centre for Professional Work and Society. Previously he was head of the Institute of Work Psychology (IWP), which is part of the Management School at the University of Sheffield. Before joining IWP in May 2011, he worked in Loughborough’s Business School for 15 years, and Manchester School of Management for 8 years. He is a Fellow and Chartered Psychologist of the British Psychological Society, and a Registered Psychologist with the Health and Care Professions Council.
Professor John Arnold is affiliated with Griffith Business School through the Centre for Work, Organisation and Wellbeing. This was a joint sponsored seminar from Griffith’s Centre for Work, Organisation and Wellbeing and the Office of the Dean (Reseach).
This article first appeared in the Griffith Business School Research Blog, by Troy Miller.
Recent Comments