Authors: Maciej Maryl, Magdalena Wnuk

The OPERAS’ Lab Observatory sheds light on the innovative technological projects aimed at communicating research results in an untraditional way. The discussion over the needs of such projects is both interesting and inspiring, particularly the requirement of evaluation criteria. The ALLEA report on digital scholarly outputs gives us some useful insights on the matter.

Although books and articles remain crucial for academic evaluation systems in most of the European countries and beyond, it seems impossible to ignore the variety of publication types and forms of activities developed by the researchers in the last decade. However, most of the systems remain “blind” to many types of research outputs, leaving various authors and contributors of those projects undervalued which often diminishes their contribution and discourages them from further improvements (Avanço, Karla i in., 2021; Shaw i in., 2023; Tóth-Czifra i in., 2021). 

One of the problems pointed out by the OPERAS report Future of Scholarly Communication (Avanço, Karla i in., 2021), was the fact that scholars often revert to something we called “double publication” or “double citation”, i.e., even if engaging with an innovative genre, they would either publish or quote a more “traditional” version of the output, usually an article or a book. The reason is simple – research assessment criteria are often blind to innovative outputs and don’t treat them as valid scholarly work. Thus one of the challenges of modern scholarly communication is to overcome this duplication and provide fair rewards for new research practices and outputs.

The question is of importance to the OPERAS Innovation Team, which is conducting a case study on various innovative projects in the area of scholarly communication. Whether they are innovative journals or recommender tools for discovery services – although they are developed and maintained by scholars and for scholars, they do not get credit as independent research outputs. Our aim is to define evaluation criteria for such pursuits and develop assessment methods that will make them a valued output. 

How to evaluate innovation in scholarly communication? 

The ALLEA report on digital scholarly outputs takes on the above question. The latest report proposes a set of changes that should be made for each of the innovative genres and formats recognised, in order to make them valid in the evaluation system. What is especially interesting in terms of the Observatory are recommendations made for various types of digital outputs. Each case study proposed specific solutions for how to include the outputs in research assessment. 

The report identifies the cross-cutting issues pertinent to digital practices in the humanities, such as (1) Linking studies with underlying data, (2) Open-ended and updated outputs, (3) different levels of contribution and authorship, (4) The need for training and competence building, and (5) Reviewing and evaluating. These discussions are supplemented by the case studies, providing a hands-on discussion of particularities of innovative cases: (a) digital scholarly edition(s), (b) extended publications, (c) databases and datasets, (d) infographics, maps and visualisations, (e) code, (f) blogs, and (g) podcasts. The report is intended to serve as a reflective tool for researchers, research performing organisations, funders, and policy-makers to adjust their catalogues of good scholarly practices and attune them to modern research.

Example: evaluating code

Let’s take code as an example. Lately at the Observatory, Ronald Snijder published a post about the OAPEN recommender system for books, based on the proof of concept described in The Liber Quarterly:https://doi.org/10.53377/lq.10938. Ronald Snijder is an example of a researcher developing tools for the sake of improving scholarly communication. As an experienced software developer, he has a background in social sciences. Although his work has an impact on scholarly communication practices, it would not be considered by the evaluation systems in academia. Should we advocate for changing that? At the OPERAS Innovation Lab we answer “yes”, and the ALLEA report gives us some ideas on how to do that:

“Recognise the differences between generic (meeting general requirements of many customers) and bespoke (tailored to specific needs) code, and stress the importance of academic accountability and the ability to gain academic credit for bespoke code development by both RSEs [Researcher Software Engineers] and programming scholars” (ALLEA, All European Academies, 2023)

Although the difference between generic and bespoke code is not always obvious, it is a first step to making software development a distinguished scholarly practice. Thus it can be a separate research result, open to reviewing, funding, and measuring impact with ECTS. 

OPERAS Lab case studies

At the OPERAS Lab, we are working on research assessment of innovative projects as a part of a case study conducted in WP4 of the OPERAS-PLUS project. The focus is on the humanities and social sciences scholarly communication practices, tools, and services. We collected 4 cases, among which is the aforementioned recommender system for OAPEN, a newly designed DARIAH Overlay Journal and two others. All will be presented in the near future on the Observatory blog. 

The OPERAS Innovation Lab case study has two crucial objectives: 

• designing methods for providing individualised support to creators of innovative outputs;
• prototyping research assessment criteria for the selected cases. 

The effects of the study will be presented in April during the OPERAS Conference in Zadar. 

Literature

ALLEA, All European Academies. (2023). Recognising Digital Scholarly Outputs in the Humanities. ALLEA. https://doi.org/10.26356/OUTPUTS-DH

Avanço, Karla, Balula, Ana, Błaszczyńska, Marta, Buchner, Anna, Caliman, Lorena, Clivaz, Claire, Costa, Carlos, Franczak, Mateusz, Gatti, Rupert, Giglia, Elena, Gingold, Arnaud, Jarmelo, Susana, Padez, Maria João, Leão, Delfim, Maryl, Maciej, Melinščak Zlodi, Iva, Mojsak, Kajetan, Morka, Agata, Mosterd, Tom, … Wieneke, Lars. (2021). Future of Scholarly Communication. Forging an inclusive and innovative research infrastructure for scholarly communication in Social Sciences and Humanities. Zenodo. https://doi.org/10.5281/ZENODO.5017705

Shaw, P., Phillips, A., & Gutiérrez, M. B. (2023). The Future of the Monograph in the Arts, Humanities and Social Sciences: Publisher Perspectives on a Transitioning Format. Publishing Research Quarterly. https://doi.org/10.1007/s12109-023-09937-1

Tóth-Czifra, E., Błaszczyńska, M., Buchner, A., & Maryl, M. (2021). OPERAS-P Deliverable D6.6: Report on quality assessment of innovative research in SSH. https://doi.org/10.5281/zenodo.4922538