EXECUTIVE SUMMARY
This research and policy paper summarises the key findings and recommendations from up-to-date performance analyses of European organisations and initiatives fighting fake news. These findings and recommendations represent the direct outcome of the piloting of an entirely novel methodology for defining key indicators for performance evaluation of fact-checking organisations in a number of organisations.
This research-based background for policy paper is the side-result of the activities taken in the COMPACT Project. The COMPACT Project represents a Cooperation and Support Action under the EC H2020 umbrella, which deals with broader issues of social media and convergence. However, considering the urgency of the issue of information disorder (of which fake news and hoaxes are an integral part) encapsulated and distributed especially via online platforms, a subgroup of COMPACT Project researchers carried out fast-track and first ever done performance analysis of effectiveness and efficiency of organisations and initiatives fighting fake news and hoaxes in Europe throughout November 2017 – January 2018. This initiative, supported by the Project Officer Alberto Rabbachin, also reflects an urgent need to engage science more deeply and expressly in developing fact- and research-based policy proposals and actions, thus helping to eliminate current and immediate threats of vital interests both at national and at EU level. One (relative) drawback of this research is that the data obtained and the responses are largely based on self-assessment by the respondents. To lessen this drawback of self-assessment, the study aimed at obtaining as much comparable feedback as possible. In that regard, some of the questions were complemented with additional queries requiring qualitative feedback to extract deeper contextual insights about the surveyed organizations.
Key findings
• The biggest challenges and issues encumbering the pursuit of goals stated by fact-checking and debunking organisations and initiatives (FC-DOIs) include lack of resources and insufficient stakeholders’ awareness on the issues related to information disorder (Figure 1).
Figure 1. Most challenging issues in fighting hoaxes and fake-news
• Although there is a clear general goal set for all of the FC-DOIs, there is a lack of clarity in the sub-goals and objectives that concern the internal processes of the organizations’ operation.
• The number of debunked news/hoaxes (last three-months average) varies highly across countries and is very much context dependent – influenced by both general political situation and ad hoc events such as election campaigns.
• Majority of FC-DOIs are ‘specialized’ in one type of content only (Figure 2). Specific visual content (photos, YouTube), although known to have far greater impact in the proliferation of fake news than text, is addressed to a lesser extent.
Figure 2. Type of content analyzed by the FC-DOIs
• The extent to which automated and semi-automated software are employed in these projects is very low, although there are already complete end-to-end computational fact-checking solutions available (Figure 3). Similarly, most of the FC-DOIs do pay attention to revision of tools, but there are still a significant number of them that have not yet considered this option.
Figure 3. Type of analysis for processing information
• The majority of FC-DOIs select their target sources and media by some predefined criteria, the most common of which is ‘public interest about the information’. Furthermore, most of the FC-DOIs employ mechanisms for information source evaluation (e.g. credibility, independence). However, even those that do pay attention to independence of sources rely only on human expertise and subjective evaluations.
• Almost all of the projects envisage political and human impact, and most of them are aware of the societal impact the work may have in general. Yet most of the FC-DOIs do not track, monitor and evaluate any impact.
• Many of the FC-DOIs also provided evidence of agenda-setting impact (e.g. legacy media referencing the results of their work) as part of their effectiveness assessment. This, in and of itself, speaks of the importance of fact-checking efforts in complementing the existing strategies for combating information disorder.
• Considering the distribution of users reached over the duration of a given project, it can be noticed that most of the projects have similar rate of expansion of their user base, with the oldest projects having significantly larger audience.
• Little considerations were reported with regard to either sustainability or business plans of FC-DOIs (Figure 4).
Figure 4. Considerations for long-term sustainability and business plans
• There appears to be strong collaboration among most of the European FC-DOIs reported by the respondents of the survey (Figure 5). However, there also appears a significant overlapping in tackling information disorder, when there are countries with two or even three separate FC-DOIs.
Figure 5. Collaboration with other similar (regional, EU, wider) initiatives
• Transparency of the majority of FC-DOIs (in terms of methodology, funding and operation) in Europe remains blurred. There is little willingness to be transparent about key information by the majority of FC-DOIs in Europe.
• The number of people engaged in the fact-checking process varies greatly among organizations (from 3 to 30). We noted that almost two thirds FC-DOIs report increase of their staff.
• There is a deep interest and potential for involvement into the regulatory issues related to the combating of information disorder online on both national and European level (Figure 6).
Figure 6. Alignment of the requirements for quality of online content with the national/regional media policie
Key Recommendations
• FC-DOIs should broaden the methodological means for approaching fact-checking issues. This includes relying on variety of experts from different fields, and being open to employ novel approaches including computational semantic analysis. The latter even appears to be urgent, considering the limited, imperfect, slow and costly human-based approaches to fact-checking and debunking, and, moreover, the emergence of artificial intelligence techniques for creation and distribution of fake news (so called digital factories).
• Fact-checking and debunking activities, although helpful, do not solve urgent and persistent problems with fake news and hoaxes – there is a need for a fundamental change in the communication policies (e.g. causal explanations), educational policies (e.g. in media literacy curricula) and in the regulatory policies and practices.
• FC-DOIs should increase and adjust their efforts in wider coverage of specific visual (photos) and audio-visual (YouTube) materials.
• It may be necessary to engage external efforts in this process of raising awareness in order to give more credibility to the FC-DOIs and raise the publicity of the issues related to tackling information disorder. At the same time, there is a need for a stronger and independent civil society informal control and of broader stakeholders’ involvement.
• There is a need for proper sustainability and business plans of the FC-DOIs to be in place.
• It would be useful to carry out performance assessment of European FC-DOIs based on additional set of criteria/indicators, such as economic indicators (for e.g. cost per output measurement can be at least indicative and potential revealing in this context).
• Carrying out fast-track, yet complex scientific performance assessments of all EU FC-DOIs based on alternative methodologies (possibly in comparison with other non-EU FC-DOIs) appears to be both desirable and efficient approach to dealing with emergent information disorder issues.
• A case can be made that efficient and effective efforts for combating information disorder may soon become part of some general set of cybersecurity measures.
Authors: Tanja Pavleska (SLO), Andrej Školkay (SK), Bissera Zankova (BG), Nelson Ribeiro (PT), Anja Bechmann (DK)
Contacts: atanja@e5.ijs.si; askolkay@hotmail.com; bzankova@gmail.com; nelson.ribeiro@fch.lisboa.ucp.pt; anjabechmann@aias.au.dk
Links:
Research findings and policy recommendations for organisations and initiatives tackling fake news
Performance analysis of fact-checking organizations and initiatives in Europe: a critical overview of online platforms fighting fake news
Research findings and policy recommendations for stakeholders and policy makers
