Assessing quality of European socio-economic research: Towards a new prototype

A new framework is emerging to help scholars, university administrators and funding bodies assess the quality of research publications covering social sciences and humanities (SSH). The preliminary framework, which was developed by the EERQI research project, should offer a more comprehensive and accurate way of judging the relevance of research publications in a multi-lingual world. The EERQI consortium is convinced that the framework has the capacity to raise the worldwide visibility of European research.

To appreciate the significance of EERQI’s assessment framework prototype, one must first be acquainted with the limitations of conventional assessment methods. Traditionally, assessing the quality of SSH research publications has meant measuring citation frequency and weighing journal impact factors. According to the EERQI project, the customary use of these methods is biased in three significant ways:

  • US publishing houses are over-represented.
  • English is hugely over-represented as a language for publication.
  • The approach to text production is rooted in the natural sciences and is not appropriate for SSH research.

These biases, which are imbedded in the traditional ranking method of quality assessment, are hugely unfavourable for European sciences and institutions, the consortium argues. The method produces an evaluation context in which individual European researchers and institutions are simply ignored. It also brackets out important subject domains and puts entire languages at a severe disadvantage. Motivated in part by “the need to remedy the inadequacies of this situation”, the initiators of the EERQI project set themselves the task of helping to adjust research quality indicators and methodologies to the European context.

As a starting point, the project decided to narrow its focus to research in educational science. The consortium operated on the assumption that educational science and research is “prototypical for vast areas of the whole field of Social Sciences and Humanities”. Education research was thus deemed to manifest characteristics of knowledge production that are also found in other SSH disciplines. As with other SSH subject areas, education research was considered by EERQI to be a field where existing methods of quality assessment “are not valid because they do not measure what they claim to measure”.

Illustrating the above mentioned language bias in the traditional ranking approach to quality assessment in scientific publications, the project noted that 89% of relevant education research journals included in the Social Sciences Citation Index (SSCI) are in English – see Table 1.

 

Table 1: Languages covered in SSCI-ranked educational research journals
 

 Source: Prof. Dr. Ingrid Gogolin (2010), EERQI State-of-the-Art Report1

 

The researchers observed that nearly the whole ‘Romanic-speaking’ research area – particularly the French-speaking world - is missing from the SSCI grid. Obviously, that omission limits the SSCI’s ability to capture the full range of education research publishing found in Europe.

Among some of the helpful tools developed by EERQI that do address this kind of limitation is a multi-lingual web-based search engine (http://www.eerqi.eu/page/eerqi-searcher). The methods utilised by this search engine are useful for educational research document retrieval and are also transferrable to other fields. Moreover, the search engine’s multilingualism could serve as a model for European search engines in general.

In addition to creating a searchable content base representing a more complete range of education research texts, EERQI developed sophisticated instruments to assist quality assessment efforts. These instruments were designed to support the process of detecting research quality in texts and to make the evaluation process more transparent. During the initial stages of the project, the researchers determined that there are two basic types of textual quality indicators: those external to the text, such as bibliometric (e.g. citation analysis and content analysis) and webometric features, and those that are internal (signals given within the words, graphs and metaphors, for example).

Thanks to EERQI, the task of identifying these indicators in education research texts can now be approached systematically. The EERQI toolbox offers two essential instruments - the Peer Review Questionnaire and Automated Semantic Analysis – for identifying quality in research texts. Learning to use these instruments properly may require some detailed instruction (exceeding the scope of this article), but the increased prospect of detecting quality more than justify the effort.

The EERQI researchers stress that their framework should be regarded as a prototype, and they admit that the project only “scratched the surface” of a monumental task. However, the project has taken us one step further in the ongoing quest to use correct tools to assess the quality of EU research in the socio-economic sciences and humanities.

 

1 See: http://www.eerqi.eu/sites/default/files/EERQI%20state-of-the-art%20report-december%202010.pdf

EERQI - European educational research quality indicators (duration: 1/4/2008 – 31/3/2011). FP7 Socio-economic Sciences and Humanities, Activity 6 “Socio-economic and scientific indicators”, Research area 6.4 “Use of indicators and related approaches for the evaluation of research policies and programmes”. Collaborative project (small and medium scale focused research project).

See: http://www.eerqi.eu/

Contact: Prof. Dr. Ingrid Gogolin, gogolin@uni-hamburg.de