Author: Davide Emanuele Iannace
Reviewers: Françoise Gouzi
Evaluation is a key to enlarging the capabilities of the RIs to be innovative actors in their own field. It gives them a key to understand their own kingdom and their reality. Again, it is a moment of reflection and self-assessment, useful to steer the course of action to new paths, following a clear, detailed and rigorous process.
A fundamental part of any infrastructural project is the evaluation. It can be seen as part of the project itself, to be performed as a pre-assessment, in-itinere action, or as a post-project evaluation. Whatever the moment chosen, evaluation is a key part of building an infrastructure. More than a screening process used to decide which project deserves to survive and which does not, evaluation should be seen and perceived as a tool, useful to analyse and comprehend the limit and the efficiency of a certain infrastructure or project.
How so? Evaluation processes push for self-reflection. Limiting the analysis to infrastructures operating in research and development, the evaluation process can become a mirror for the scholars and specialists involved. A mirror that is useful to face their own biases and look in a more neutral way toward both the achievements and the limits an infrastructure can have. Evaluation is also a moment of exchange of practices with comparable cases. No infrastructures or projects live in a bubble, isolated. Every element of the scientific community is in a continuous exchange with the space around itself.
A space shaped by other infrastructures, but also by funders and stakeholders not directly involved in the development of the RI (Research Infrastructure), and yet could have a relevant role in it. It is the case of universities not directly involved since the start of the RI, or research centres and even private companies that could actively support the infrastructure. Evaluation helps and supports the process of expanding the infrastructure network, but also as a credential of transparency of what has been done and will be done in the future.
Evaluation, in these terms, is a process that allows all the involved people to understand themselves and their project; howbeit, it is the opportunity for outsiders, and potentially interested new parties, to understand the project and even find a spot in it. Evaluation is a key to enlarging the capabilities of the RIs to be innovative actors in their own field. It gives them a key to understand their own kingdom and their reality. Again, it is a moment of reflection and self-assessment, useful to steer the course of action to new paths, following a clear, detailed, and rigorous process.
The RI-PATHS approach
The importance of evaluation, as briefly explained before has pushed for a better and new way to perform it. In this work, we are talking about and considering one context, the evaluation of Research Infrastructures. As massive enterprises, infrastructures require huge funds, an extensive network of institutions – i.e., public research centres, private companies, libraries, or laboratories – and the capability to plan their own future.
In this sense, evaluation is a fundamental tool to ensure the infrastructure is able to reach its KPI (Key Performance Indicators). It is useful to understand when the RI is reaching these useful indicators, failing to do so, and to explain and understand the reasons why.
Besides, the evaluation of an infrastructure is a process that allows scholars outside of the project itself to better understand what kind of effect these RIs have. These impacts, and the potentiality to realise a change in reality, is the core of the approach of EFIS RI-PATHS.
RI-PATHS (Research Infrastructure imPact Assessment paTHways) is an initiative sustained by the European Union and EFIS, to better understand and uniform the socio-economic impact evaluation of research infrastructures. It focuses on the set of infrastructures sponsored by the EU in its framework programs as Horizon, or the ones under the supervision of ESFRI (European Strategic Forum on Research Infrastructures). The scope of RI-PATHS is to present a plan for the evaluation of infrastructures, providing not only a guide for who must design an evaluation process, but also presenting a kit, a tool that could be used.
The approach of RI-PATHS is to encompass the variety of impacts, aspects covered by infrastructures, the effect on human resources, economy and innovation, society, and policy, under one umbrella. The toolkit goes beyond the simple identification of determinants. It creates instead its “pathways”, exploring the capability of a RI to influence certain aspects of scientific production – i.e., publications or citations – and of the society – i.e., the transfer of licenses, the condition of employment in academia. One of the main decisive outcomes of RI-PATHS are these “Pathways”. Not a plain sum of indicators, they reconstruct a logical course to understand the key aspects of the infrastructure. Three are the macro-areas: Enabling Science, Problem Solving and Science and Society. For each of these macro-areas, several pathways are present.
The numerous indicators will allow the evaluator, project manager, or the policy-maker, to immediately recognize a certain element they are interested in to better understand, and evaluate.
The stark simplicity of the construction of RI-PATHS allows this tool to show the impact in a well-tailored way, useful for any RI – from the physical to the digital or delocalized one. Moreover, it allows to creation a reflective feedback on the RI itself, on its capability to innovate and exist. Evaluation, in this sense, is a means to understand the RI’s potential capability to shape its field, while generating an array of positive effects that could accelerate innovation and impact the societal space around the RI.
Infrastructures themselves endure the evaluation as a process in which they confront themselves. The indicators and the pathways of this toolkit offer the possibility for infrastructure to also reflect on how the infrastructure is evolving, changing, and adapting to external and internal inputs.
This brief summary of RI-PATHS gives an insight into the double-edged innovative effect of the RI. On the one hand, an RI produces innovation by functioning as a pivotal element of the scientific field. On the other hand, the RI produces innovation through a feedback process, by reflection and its own evaluation, and while adapting to the world around itself.
The evaluation process, with these premises, is a producer of innovation. Considering as an example the pathways, evaluation is a process of understanding the nature of the actions of an RI, while establishing methods and flows of work that can become new practices in the scientific world. These processes, of learning-by-doing, adaptation through evaluation, are a focal point for the innovative capability of the RI itself and are flagrant when the evaluation is performed partially by the RI and partially by evaluators outside of it, in a mutual relations of responsibilities and practices, and in a a continuous exchange of multiple points of view.
A quali-quantitative approach
While the approach of Ri-PATHS finds its strength in the presence of clear indicators and logical pathways, a more de-structured approach could be useful to foster a self-reasoning stream in the RIs.
Directly involving the people part of an infrastructure in a method of evaluation could push toward a sort of reflective moment inside the infrastructure itself, creating space for change and new forms of innovation. In this sense, the structure of a quali-quantitative methodology in approaching the evaluation of infrastructures could be supportive.
This method, which is being developed as part of a broader doctoral project which I am working on, is twofold: firstly, a set of semi-structured interviews is conducted with key stakeholders and participants of the national nodes of an infrastructure. The semi-structured interview, following the PESTLE model of analysis, touches on different aspects of the life of infrastructure, from the formation of the node to the perspectives toward the new tools and the presence of some specific actors in the network. Semi-structured interviews are different from a simple stream of consciousness but allow a certain degree of freedom for a respondent to explore aspects the interviewer has not considered.
This is vital for the creation of the survey as a second part of the evaluation method. It allows exploring, qualitatively, the reality of the node and the infrastructure, their common evolution and their shared path. In this sense, the survey, directed to all the members of the teams involved in the infrastructure, is based not only on the existing and rich literature but also on how the participants interpret the infrastructure and their role in it.
A fundamental piece of this approach to evaluation is the interaction between the evaluator and the targeted infrastructure. Without a stable link of connection between these two sides of the process, the evaluation is naturally doomed to fail. Additionally, this interaction is useful also to generate feedback – dependent on both the quality of the survey and the responses in the qualitative moment – useful to generate and create possible new positive innovations inside the infrastructure to mend eventual shortcomings.
Evaluating infrastructures, or tools?
Both the pathways of RI-PATHS and the second method approach infrastructures as complex ecosystems, networks of stakeholders, and interconnected realities, and make sense out of it as their totality. This macro-approach, a top-down one, presents several limitations. Despite the advantage of considering the entirety of the infrastructure, it can be dispersive. National nodes could present serious differences among themselves. While the costs could be clear on the entirety of the RI, the benefits could be more theoretical than practical, because the point of view could fly too high with respect to the reality that the RI hopes to impact. A top-down approach may be the best way to keep a large view on the total capabilities of these infrastructures to introduce innovative tools. It can also be a way to rearrange their actions in a continuous feedback-action process, fostered by the evaluation, and acted on the entire network of the RI. It has the risk of being a far away eye from the reality of the everyday practices of the infrastructure and its node, and to miss the effects realised in the reality close to the RI but outside of it.
To amend such an issue, a different focus could be to concentrate the attention on one single aspect of the RI. This could be particularly effective for the RIs that are delocalized, digitalized, and, in some senses, an agglomeration of several sub-actions. Focusing on a single aspect – i.e., a tool, a project, an action – could foster a more profound understanding of it, and show the impacts existing on a more local, but yet relevant, level.
Taking OPERAS as an example, the evaluation of the totality of the infrastructure could present some of the aforementioned issues, due to the unique nature of the RI. To support and foster the innovative aspects of OPERAS itself, correlated to the tools developed under its macro-umbrella, analysing such tools – their history, their impacts, the challenges they met, and their strong points – could be a bottom-up way to understand the real wide capability of OPERAS to impact on the scientific community. It can be a way to understand how the innovation presented by such tools reflect on the RI and on its environment, but also how they are strongly related to both the local and digital spaces where such tools operate.
About the author
Davide Emanuele Iannace is a PhD student of the CNR-IRCRES and the University of Rome “La Sapienza”, specialized in evaluating policies on Research and Innovation. His research focuses on the role of research infrastructures in academia, their impacts and positive outcomes. He is part of the University of Coimbra’s team dedicated to PALOMERA”.