The UK's research funding landscape is evolving, and with it, the challenge of managing an ever-increasing volume of grant applications. UK Research and Innovation (UKRI) is taking a bold step to tackle this issue by exploring the potential of AI in grant peer review.
UKRI, an umbrella organization, allocates a significant amount of funding, over £8 billion annually, to support research and innovation. However, the number of applications has skyrocketed, outpacing the growth in funded grants. This surge has prompted UKRI to seek innovative solutions to streamline its peer review process.
Enter Mike Thelwall, a data scientist from the University of Sheffield, leading a research team funded by the UK Metascience Unit. Their mission? To investigate the role of generative AI in grant peer review. With access to a substantial dataset of grant proposals, either funded or rejected by UKRI, the team aims to train large language models (LLMs) to predict peer review scores and decisions.
But here's where it gets controversial: Mohammad Hosseini, an AI ethics researcher at Northwestern University, raises concerns about LLMs' ability to generate novel ideas. He argues that if AI struggles with creativity, it may also fail to recognize truly innovative concepts, as it's trained on existing data. This perspective adds a layer of complexity to the discussion.
And this is the part most people miss: the potential impact on the grant application process. If funders aren't transparent about their AI criteria, researchers may feel left in the dark. On the other hand, if the process is open, applicants might start optimizing their proposals for AI feedback, potentially gaming the system.
So, how might UKRI implement generative AI? Thelwall suggests it could be a tiebreaker or an additional reviewer, even a fast-track desk-reject option. He cites the example of the la Caixa Foundation in Barcelona, where AI-assisted peer review still relies on human experts for the majority of applications.
Thelwall's team is taking a cautious approach, recognizing the need for AI systems to reach a high level of accuracy before assisting peer review. Their previous work on refereeing articles for the UK's Research Excellence Framework highlights the challenges and potential of AI in this domain.
As UKRI navigates this complex landscape, the use of AI in grant peer review is a fascinating development, raising important questions about the future of research funding and the role of technology in shaping it.
