How can I trust this evidence? The Development of a Tool to Appraise Evidence Resources

As the COVID-19 pandemic spread I – like many others – wanted evidence about what it was, how it spreads, and how to best protect myself and my friends and family.  However, whilst people have easy access to information about pandemics due to the availability of internet connectivity and electronic media, information overload presents a new set of challenges, such as false news, conspiracy theories, magical cures, and racist news spreading at an alarming rate, with the potential to increase anxiety and stress and even lead to death.

So, the emergence of COVID-19 left us with both too much and too little evidence for prevention, control of spread, and management. Too much because of the overload, but too little as we don’t know which evidence to trust. This is why we need to appraise evidence resources critically. This is an intriguing issue with many possible approaches. Here I’d like to draw your attention to what we’ve experienced and managed to learn from looking at different evidence resources for the COVID-19 pandemic.

During the pandemic, vast quantities of evidence are being generated from various disciplines for use by academicians, researchers, social workers, governments, and individuals in providing coordinated efforts to combat the pandemic and its consequences. People in such situations are desperate for any kind of information from any source as long as it gives them hope, and they consume it without questioning its source, authenticity, or validity. To help mitigate this confusion and disarray from such a wide range of information, evidence resources on a specific subject are typically created to serve as a central source of information.

Evidence resources include

• Evidence collections, which are collections of studies or other evidence resources provided by an organization,
• Evidence platforms, which are online resources that contain links to relevant evidence resources, databases, which are searchable sets of studies,
• Evidence maps, which are maps that present the evidence usually in an online pictorial framework, and
• Guidance and toolkits, which recommend policy and practice.

All these resources can be created by any organization and/or by individuals online and are usually accessible by all.

The intention in generating evidence resources is a good one. They seek to make knowledge and information available. But their quality must be vetted to ensure that consumers of evidence provided by these resources are protected from malice, misinformation, outdated information, and extortion. We call this having confidence in the evidence resource.

To protect the cardinal purpose of knowledge sharing and advance the good intentions of providing evidence-based information and guidance, it is necessary to separate high-quality evidence resources from low-quality evidence resources. As a result, there is a need to create a critical appraisal tool that can be used to evaluate these sources of evidence.

One might argue that the evidence obtained from these sources can be evaluated for quality; while this is true for primary studies and systematic reviews, it may not be as easy with toolkits and guidelines because they rely on already synthesized evidence and are like directives on what should and should not be done; therefore, it is critical to ensure that they come from a trusted source and that their development was based on high quality systematic reviews.

The development of a critical appraisal tool to appraise evidence resources is thus based on the aforementioned reasons, and we intend to create a tool that is both reliable and easy to use. This is to allow the consumer to assess the strength and weakness of various evidences resources accessed in search of new information and knowledge, whether for research, academics, or policymaking.

Our tool assesses the evidence resources against the following domains;

1. Clarity of purpose: – Does the evidence resources have a clear description of the purpose?
2. Achievement of purpose: – Are available resources clearly and specially aligned with the purpose provided in the description?
3. Ease of use:- Can the consumer use the available evidence with ease and without difficulty, for example can the materials be downloaded for free?
4. Updating New information: – Are the evidence resources frequently and regularly being updated?
5. The sources of available content: – Is there a description of how the content available on the evidence resource was identified/ obtained?
6. Overall (based on the lowest rating on any of the critical domains except source of information)