During the COVID-19 pandemic, member centers of See Check, a regional network of fact-checking organizations in Southeastern Europe, have witnessed how the dissemination of false and doctored information on social media platforms and online media can harm the public’s ability to form viable decisions about health issues and lower professional standards in journalism.

Social Media platforms have for the most part allowed the distribution of vast amounts of COVID-19 mis- and disinformation via their platforms, thereby increasing the frequency of online harassment in response to views that are unpopular with certain segments of users.

In such circumstances it is all the more vital that the public receives clear and factually correct information.

Razkrinkavanje.si, the Slovenian partner of See Check, has compiled an overview of lessons learned from fact-checking pandemic-related public statements and media reports with the aim of assisting journalists in the process of verifying information on the COVID-19 pandemic.

See Check verifies public statements and media reports in five countries: Razkrinkavanje.si in Slovenia, Faktograf.hr in Croatia, Raskrinkavanje.me in Montenegro, Raskrikavanje.rs and Fake News Tragač in Serbia, and Raskrinkavanje.ba in Bosnia and Herzegovina.

We hope that the lessons drawn from Razkrinkavanje.si’s experience will be useful not only to fellow journalists but also to the public at large. In the age of disinformation, the ability to form an opinion about important societal issues is increasingly becoming dependent on the ability to verify key information.

In the following manual we will present the main principles and some simple tools for verifying information on COVID-19. We will also present examples to complement the instructions. The manual also addresses key points with respect to digital security.

Overcoming the challenges of fact-checking

When making editorial decisions about which topic to fact-check, the public interest can be your guide. You may already pay special attention to statements by politicians and expert public officials, but often the sources of mis- or disinformation are citizens whose posts have gained a lot of traction on Social Media platforms.

Prioritize those claims that seem harmful to the public interest or the public debate about COVID-19, and that you also assess to be effectively fact-checked. A fact-check that needs to be fact-checked again does not serve its purpose. This is particularly important when a small fact-checking team tries to navigate situations in which disinformation is abundant.

When the team at Razkrinkavanje.si spots a suspicious claim, the assigned reporter first independently researches it and its source while paying attention to whether it has already been debunked.

There are several websites of acclaimed media organizations that are dedicated to high quality fact-checking. Look among the IFCN Code of Principles signatories on Poynter’s website or among fact-checking organizations from the EU that joined the European Digital Media Observatory’s community. These media outlets have all committed to observe professional and ethical standards while combating misinformation.

To understand the context of a claim, Razkrinkavanje.si’s reporters review relevant national and foreign public databases, posts on social media platforms, and expert reports and studies. For COVID19-related claims, they focus on sources that are dedicated to monitoring specific topical areas and/or gathering relevant official data on SARS-CoV-2 testing, infections, deaths, vaccination rates, hospitalizations, hospital capacity, and the pandemic’s effects on the economy and society.

In this sense, the first point of reference are centers for disease control and national statistical offices. It is also worthwhile to check trustworthy international websites and data aggregators. The WHO collects global data on health. Similarly, Our World in Data, a research project at the University of Oxford, tracks the impact of the pandemic in every country in the world. The European Union’s statistical office Eurostat holds data on the economy, society, agriculture, energy, and several other topics in the EU. The OECD regularly publishes data visualizations and explanatory texts on the COVID-19 crisis and the post-pandemic recovery along with compiling data, analysis, and recommendations. Additional open-access data resources are also available on the platform of the American governmental agency the National Institutes of Health.

Regardless of the variety and richness of information originating from such sources, their validity may be in question and may pose challenges to journalists. Even official data can be messy and imperfect. In large-scale events, such as the COVID-19 pandemic, comparing data from different countries will not necessarily give a true picture because of the differences in countries' reporting methods. Thus, it is incumbent on journalists to notify the readers about such inconsistencies.

”

Example: In late Summer 2021, the head of the COVID-19 advisory group in Slovenia and a prominent infectologist claimed that Slovenia’s hospital treatment system was under significantly more pressure than in other European countries because there were less available hospital beds, especially in intensive care units. Razkrinkavanje.si reviewed data from Eurostat, the OECD, and a study published in the Journal of Intensive Care Medicine and were able to confirm that Slovenia was indeed below the EU average with respect to available hospital beds for intensive therapy and care per citizen.

However, Razkrinkavanje.si found that the claim that the hospital treatment system was under more pressure than in other EU countries was unfounded because it couldn’t be supported by sufficient evidence. The system was also greatly affected by a lack of medical staff and low vaccination against COVID-19.

“

Online and digital tools that also turned out to be useful during the research:

Internet archive – for exploring old, deleted, and changed websites.
FotoForensics – for digital picture analysis, including metadata.
Google Reverse Image – for discovering visually similar images from around the web.
InVID – for verifying videos and images.
RevEye – for reverse image search.
TinEye – for image recognition and reverse image search.
Pimeyes – an online face search engine (not free).
Trendolizer – for scanning the internet for trending content.
Domain tools – for information about a domain, DNS, name servers, IPs (not free); domain owners are anonymized.
Simon Says – a transcription tool that converts audio and video media files directly to text using artificial intelligence.
Research Clinic – internet research links and study material compiled by BBC’s open source analyst Paul Myers.
Bellingcat's Online Investigation Toolkit – a vast selection of OSINT tools for verifying photos, videos, social media posts, and similar.

Another important “tool” is the reporter’s awareness of his or her own limitations. Fact-checkers need days to fact-check a lie that someone came up with in a few minutes, and sometimes it is not possible to disprove something with enough certainty. Tell your readers what you were able to prove, but be honest in your reporting – explain to them when you were unable to prove a part of a claim and why.

Journalists need to be conscious of their own biases and the possible biases of their sources and need to be mindful of not drawing rash conclusions, especially in the face of uncertainties and lack of hard scientific evidence, as was the case in the first months of the pandemic.

If you find mistakes after publication, correct the original story, clearly mark the correction, date it, and explain what you have corrected and why. The COVID-19 pandemic continues to be a developing situation, so journalists need to keep up with the changing circumstances.

Acknowledging your mistakes and limitations builds the public’s trust. Don’t miss an opportunity to be transparent about your work, just like you would expect a governmental minister to be transparent about his. It will pay off.

Read scientific papers with dedication

While research is most important for assessing the credibility of facts, a very good understanding of how to read and properly interpret academic studies is often key, especially for news on health.

However, it is important not to stop at the first paper that supports your assumption, but to first check other studies to see whether there is a general scientific consensus about the problem you are researching. Then start to review alternative theories: Are there many? How prominent are they? Were they published in peer-reviewed journals? Who represents these views and what is their academic background? Have the alternative findings been disproven by peers?

”

Example: In August 2021, an American family doctor and an opponent of COVID-19 vaccination claimed that the rate of miscarriages in pregnant women who were vaccinated in the first trimester increased eightfold. He was referring to preliminary findings of an ongoing study on the safety of mRNA vaccines for pregnant women which was published in the acclaimed peer-reviewed publication the New England Journal of Medicine. His claim began to spread from a US video platform known for sharing conspiracy theories, and in time it also reached Slovenian social media users.

But Razkrinkavanje.si found that the doctor’s claim was based on a misinterpretation of the data from the study. For example, its authors found that the side effects of the vaccines in pregnant women did not differ significantly from those reported by non-pregnant women. Overall, vaccination with mRNAs during pregnancy was found to be effective and safe according to the data. Contracting COVID-19 while pregnant and not vaccinated increases the risk of unwanted pregnancy outcomes, including miscarriage.

“

To assess the reliability and validity of scientific studies, one should first pay attention to where, by whom, and when a study was published to make sure that it was published by a reputable source and is not outdated.

The most reputable academic papers are published periodically by established publishing houses with high impact and citation factor; they are peer reviewed and have an international editorial board. If the paper reports on the treatment of an illness, information on the number of participants of clinical studies should be included. Clinical trials are usually conducted in three phases, and each phase should include a certain number of participating volunteers, from up to 100 in the first phase to several thousand in the third.

Scientific papers and journals on health and medicine are indexed in familiar online databases, such as PubMed, Scientific Research Publishing, MDPI, or Web of Science. It is also worthwhile to look for systematic reviews of primary research in human health care and health policy that are published in the Cochrane Library. It is always better to read the complete academic paper, but it might not always be freely available under open access principles. In such cases, get in touch with its authors and ask them for a copy of their work.

If the findings were reported as a preprint or in a non-peer-reviewed journal, it is important to clearly inform your readers that you are quoting from a study that has not yet been peer reviewed. It has not been scrutinized by the reviewing process which includes independent experts in the field, so it is better not to quote from it at all.

More often than not, journalists have to quote from studies that have been published in a foreign language. Even though this will largely be the English language that you may be well-versed in, you should make sure that your translation is accurate and credible. Consult a native language speaker when in doubt.

Build your expert sources

Since the pandemic is an issue that spans many areas, from health to the economy, seek the opinions of credible, independent experts with proven references and those from relevant national institutes.

This could significantly reduce the chance of error while increasing your article’s comprehensiveness. It will also help you to widen your professional network of expert sources that you can also rely on for sound opinions in the future. They can also help you to find other valuable experts, for example, when you need to verify information that concerns a specific academic subfield.

This may be specifically important when it comes to health experts commenting on COVID-19 issues. A governmental COVID-19 advisory group usually consists of medical experts with vast experience and knowledge, such as virologists and epidemiologists. They may be a good starting point, but try to also find other independent sources with relevant expertise and a proven track record.

It is important that you are precise in addressing relevant experts; anyone with an academic degree in medicine cannot and should not be asked to comment on health-related issues outside of their specific academic fields. If a dentist gives opinions about the performance of heart valves, you should probably verify their claims.

Understand and care for your data

It is essential to accurately interpret statistical data on new cases, tests, hospitalizations, available hospital beds, vaccination rates, and other COVID-19 related issues, but they should also be presented in a simple and understandable way without leaving room for misinterpretation. When possible, try to cross-reference the set with additional sources.

”

Example: In November 2020, the Slovenian Prime Minister claimed that there were at least 400 beds available for seriously ill COVID-19 patients, in contrast to only 120 at the beginning of his mandate in March 2020, to showcase the success of his government in providing sufficient hospital bed capacity.

With the aim of verifying this claim and obtaining more comprehensive information on the number of beds, Razkrinkavanje.si contacted every general hospital. The data showed this claim wasn’t true. What’s more, many hospitals stated that it wasn’t the lack of the beds that posed a significant problem for them, but rather the lack of hospital personnel for treating patients. This clarification gave an important context to the fact-checking article.

“

Always scrutinize the acquired datasets. Don't take a file at face value and trust it just because it contains numbers. If they were input automatically by a computer system, they may be more reliable than not, but if they were input by people, you should not forget that all people make mistakes. Make a copy of the original data and safeguard the original. Clean your data before any analysis.

Use only reliable, verified sources for obtaining data. The first point of reference should be national or international statistics bureaus and experts. However, when you are looking at and comparing datasets from various countries, keep in mind that countries have nationally specific methodologies for collecting and analyzing data, and that – especially during a pandemic – they often change. Let the readers know about any data discrepancies.

Provide at least two independent sources, or more if possible, to verify the claim and try to acquire primary documents and data to support it. An infographic published by another media outlet is a good secondary source, but you should acquire the original data, preferably in spreadsheet format. If it’s in a pdf, you can scrape it and extract the relevant spreadsheet in a suitable file type.

Always create a copy of a dataset to use for your analysis and keep the original file intact. It is there as your parachute in case you need to start from scratch or analyze a calculation mistake.

First inspect the data for clarity. If you see typos, missing values, and similar issues, your data is “dirty” and you should clean it before you start working with it. You can use filtering, sorting, formulas, and other techniques available in your chosen spreadsheet program to clean your data. For large or very dirty data, you can use programs such as Open Refine for more efficient cleaning and analysis.

When your dataset is clean, you can get down to business. By then, you will have a much better understanding of your dataset and your analysis will be sounder.

Always carefully read the methodology and any disclaimers added by data sources to avoid misinterpreting the data. You should briefly explain any important methodological caveats to your readers and, where relevant, include a reference to the representative sample of a study in your story. Name the data source in the story and link to it when possible.

Don’t cherry-pick the numbers or paragraphs from an analysis to confirm your personal opinion based on diagonally reading a study. A fact-check’s accuracy is more important than you being right.

When in doubt about your interpretation of statistical data, consult with an expert from a national statistical office or a university professor from a relevant field.

File precise inquiries

Public officials tend to send generalized answers to journalists who send them general questions. This usually happens when a reporter has not taken the time to read up on a topic but sent out questions on-the-go “to speed up the process”.

More often than not, experts get such general questions, too. Some take their time and provide reporters with detailed explanations because their inquiries have revealed their poor understanding.

This is time-consuming, ineffective, and does not serve the aims of reporting. It is also unprofessional. Journalists should not expect others to do their work for them.

Reporters should familiarize themselves with a topic as much as possible in a given context. Even if it’s breaking news, one can still take a little time to read up and then ask the right questions. This also demonstrates respect for your sources, who have better things to do than to explain the basics to an unprepared reporter.

Nurture your digital security

Threats are constantly evolving and so are the tactics of threat actors. Hackers have taken advantage of the COVID-19 crisis to spread malware, disrupt work, sow doubt, and earn easy money. According to the EU agency for cybersecurity ENISA, the number of phishing attempts increased 600% immediately after the start of the pandemic.

As the threats increase so does the need for journalists to include security considerations in their workflows. The COVID-19 pandemic has made them even easier targets of smear campaigns and doxing, so it is important that they secure their work, sources, communication, and privacy in order to be ready for possible malicious actions.

Today, digital security awareness is important for any journalist, including fact-checkers, and should become an integral part of their work routine.

The following tools are useful for securing your work, colleagues, and sources:

Protect your Google, Social Media, banking, and other accounts with two-factor authentication for an additional security layer. You will need a mobile application such as Authenticator or, for stronger security, a USB key such as a YubiKey.

Use long passphrases instead of shorter passwords on any type of account and save them in a password manager, not in your browser or notebook. There are several password manager options, such as KeePass XC (free), Bitwarden (free and paid options), or 1Pass (paid). Always save backup codes for your Google and other accounts.

Be aware of phishing attempts and don’t open email attachments with uncommon extensions, such as .jar or .hta; be mindful of .zip and .rar extensions as they can be a vehicle for viruses and malware. If you have to open such files, first check them with an anti-malware program.

In your browser's settings, set up automatic deletion of cookies and search history after quitting the browser.

Remove recently downloaded files from your download folder daily. Store all work files, or at least sensitive ones, in encrypted folders. You can do this with VeraCrypt, an excellent open-source utility program.

It is safer to browse in a private window, but that only prevents your computer from saving your browsing history. To browse while being protected from others, be sure to always use a VPN to mask your IP address.

Regularly back-up your drive, which should be fully encrypted. Store the password for the encryption in your password manager and on paper in a safe location.

Diligently update your operating system and program software as updates become available.

Always sign out from applications, and shut down your computer when you are done working. It’s not only good for your psychological health but also more secure. If you suspect that your phone communications may be at risk, turn your mobile phone off completely once a day then turn it back on. This may help with disarming a piece of malware installed covertly.

Being aware of possible threats, mindful of solutions, and using a variety of trustful digital tools can make a journalist’s work and routine much more secure. While no system or tool is 100% secure, we can recommend the following ones:

Signal: a free mobile and desktop application for secure, fully encrypted communication, including file sharing.

VPN Mullvad: for masking your IP address to protect your internet activity.
VeraCrypt: for securely storing your files.
PGP key: for encrypting and thus securing your email communication with OpenPGP (on Mozilla Thunderbird) or with GPG Tools (on MacOS Mail).
Google Authenticator: for two-factor authentication.
1password or KeePassXC: password managers.
Tresorit: for sharing large files via an encrypted link; think of it as encrypted WeTransfer.
HTTPS everywhere: a browser extension that forces encryption on many websites that are still using the old and insecure http protocol.

To fact-check COVID19-related claims, the journalists of the See Check network turn to dozens of official databases, infectious disease experts, doctors, professors of medicine, and other reliable sources in their respective countries. Some are listed below but you can always contact a See Check network partner directly for any specific needs.