Extract from the second joint meeting of WHO GACVS and WHO ACSoMP, published in the WHO Weekly Epidemiological Record of 3 March 2023

WHO publishes vaccine reaction rate sheets for non-COVID-19 vaccines.[1] These rate sheets give a summary of adverse reactions observed for each vaccine, which can be used to investigate safety signals as well as for communication and discussion. The PVG team contracted the Australian National Centre for Immunisation Research and Surveillance (NCIRS) to develop a rate sheet for COVID-19 vaccines. The initial plan will focus on five known serious reactions, myocarditis/pericarditis, thrombosis with thrombocytopenia syndrome (TTS), Guillain Barré Syndrome (GBS), immune thrombocytopenia (ITP) and anaphylaxis. These events will be assessed for each WHO Emergency Use Listed (EUL) COVID-19 vaccine platform. In addition, adverse event rates of common local and systemic reactions will be summarized, using evidence from clinical trials.

Systematic literature searches will be performed by event and vaccine platform using several databases from the start of COVID-19 vaccination programmesto the present. The inclusion and exclusion criteria will be specified using Population, Intervention, Comparator and Outcomes (PICO) approaches for each research question. If good quality systematic reviews are identified, they will be used. Each event will be analyzed successively. Formal endorsement of the COVID-19 rate sheet is expected in September 2023, although some sections of the rate sheet could be published earlier.

Another innovative approach for AEFI monitoring is the development of new dashboards based on data from the EWS, which collects data from any accessible open source, including social media, Twitter, Facebook and Instagram, with restrictions, content from news channels, blogs and web sources using Peek, which is an open-source analysis platform. These data are used to create dedicated data collections on specific topics, such as COVID-19 vaccines, which was the first vaccines that the EWS was used for. Data for other vaccines or other topics can be obtained using the process. While the data are being analyzed, it is enriched with taxonomic information. Artificial intelligence (AI) is used to filter out irrelevant data and to detect relevant data and new types of information. The information is complex, and the new targeted dashboards are useful for making the information readily accessible. The initial project with COVID-19 vaccines in November 2020 involved screening about 40 million documents. In May 2022, COVID-19 medicines and mpox were added. In 2023, the other WHO prequalified vaccines will be added. It is also planned to extend coverage to medicines, which is more challenging as there many more medicines than vaccines.

Recently weekly summary updates were produced showing the total number of mentions of a specific topic, for example, drugs used to treat COVID-19. Rapidly changing mentions of drugs can be identified, either new drugs or surge of mentions compared with previous summary. More detailed information can be accessed through links in the summary report. AI does not make an expert medical assessment of the data, that is where humans come in, but it can minimize the number of non-relevant documents and maximize the number of relevant documents.

Finally, the concept of a project to produce a business intelligence dashboard (BID) for monitoring serious AEFI reporting rates using data from WHO UNICEF Joint Reporting Forms (JRFs) and UMC’s VigiBase data was demonstrated. A business intelligence platform is technology that helps organisations gather, understand, and visualize their data so that they can use the information to make better decisions. By monitoring the performance status, the platform can identify what is needed to improve AEFI surveillance status and initiate appropriate actions before target deadlines. This enhances efficiency by identifying potential issues before they become large problems, and thus, enabling rapid and appropriate decisions to be made in near real-time, based exclusively on in-country serious AEFI data in VigiBase and JRFs. The expected benefits from using this approach include an open and transparent resource that will be available on the WHO-HQ PVG website. It will allow near real-time monitoring of the Immunization Agenda IA2030 framework indicator for serious AEFI reporting rates per million population.[2] Such a dashboard will encourage countries to align with the ‘one country – one AEFI data’ concept. GACVS cautioned that such a dashboard in the public domain could be misinterpreted and misused and recommended that it be available to relevant stakeholders via restricted access.

Extract from GACVS meeting 8-9 June 2021, published in the WHO Weekly Epidemiological Record of 23 July 2021

GACVS members were introduced to the concept of an early warning system (EWS) for identifying concerns about COVID-19 vaccines and potentially about vaccines in general. The speed and scale of development and introduction of COVID-19 vaccines has been unprecedented, with the result that a growing number of vaccines, some based on new technology platforms, are now being deployed across the globe following the review of encouraging – but incomplete – data on safety and efficacy. Clinical trials and post-authorization studies are underway to address knowledge gaps. At the same time, intense media and public attention on the safety of “fast-tracked” COVID-19 vaccines has led to an “infodemic” of misinformation, disinformation and incomplete or contradictory information, which is spreading rapidly through social media and other channels.

Functional vaccine safety surveillance systems, mainly in high-income countries, have monitored the safety and efficacy of the different COVID-19 vaccines being administered in their populations. International mechanisms are in place for sharing information as it becomes available to enhance knowledge about COVID-19 vaccines. However, various challenges persist, such as delayed reporting of adverse events following immunization (AEFI) or no reporting at all from some parts of the world and lack of capacity to identify, in real time, the spreading of rumours and misinformation. The EWS for vaccine-related events could complement functional vaccine safety surveillance systems by identifying vaccine safety issues (real or perceived) through social media listening.

The GACVS agreed that, in order to support the kind of proactive response needed at a time of heightened interest and news about the safety of vaccines, there is a need to identify, as early as possible, information or concerns about potential side-effects following COVID-19 vaccination.

As the rollout of COVID-19 vaccines started, WHO’s PVG team began setting up and hosting a real-time online listening function that can facilitate early detection and analysis of, and response to, COVID-19 vaccine-related events and concerns.

GACVS members were informed that the EWS is an event-based surveillance system that aims to complement information from routine surveillance systems, recognizing that both contribute to the early warning function that is critical to a prompt and proportionate response to ensure human safety worldwide. More specifically, the main objectives of the EWS are:
to detect COVID-19 vaccine-related events rapidly; to ensure immediate communication to WHO for review, investigation, characterization and risk assessment of the information in order to determine the level of risk posed by the detected event; and, when requested, to verify the initial information for further review by WHO and the GACVS.

The need to specifically address “conversational signals” on the safety of COVID-19 vaccines, as found in existing social listening projects, prompted the need to invest in a platform³ that can leverage machine learning and artificial intelligence to enable wide-scale tracking and analysis of online public content on COVID-19 vaccine-related events. Currently, the platform is retrieving an average of 33 000 items of data on COVID-19 vaccine-related events every day. The platform’s classification algorithms are trained to detect news, official announcements and signs of unusual activities/information/conversation (in relation to COVID-19 vaccine-related events); to extract statistical evidence to reduce bias and noise, categorizing sources of information in order to eliminate noise and to gain time for response when, for example, misinformation or harmful information is being shared; and to filter the popularity of interactions on social media.

When dealing with document retrieval of unstructured data, there is a fundamental trade-off between recall⁴  and precision,⁵ such that greater recall normally means reduced precision and greater precision means reduced recall. However, by using smart algorithms it is possible to improve both without detriment to the other. This increases the chances of detecting potential vaccine-associated events,⁶ including AEFI and adverse events of special interest (AESI) relating to COVID-19 vaccines.

Examples of AEFI relating to COVID-19 vaccines, as identified on the platform, were presented to the GACVS. Samples ranged from product-related or quality defect-related reactions to anxiety-related reactions, including errors in administration, in different countries and across multiple vaccines.

The next steps in the pilot project will focus on further developing the analysis algorithms, so that the platform can be applied to low- and middle-income country settings to aid early detection of emerging issues and to supplement official reporting mechanisms, including in areas where delays and incompleteness of reporting are anticipated.

WHO regional representatives found the tool useful for early identification of issues that, if not promptly addressed, could affect vaccine uptake. GACVS members noted that the early identification of potential safety concerns would give more time to prepare a scientific and communication response. Three particular use cases were highlighted, namely that the EWS’s early detection capabilities through social listening would help in promptly managing signal detection, immunization errors and vaccine hesitancy. It was strongly suggested that such a system could be deployed across WHO regions in a sustainable manner in order to systematically document/monitor new fields such as adult immunization.

GACVS felt the EWS could not only help to detect information on potential safety signals but also to spot pockets of misinformation that need to be addressed promptly. GACVS members recommended an evaluation, comparing this kind of social media listening with the manual monitoring of online media. Members stressed that social media sources could highlight not just scientific problems but also communication problems. 

In conclusion, the Advisory Committee recognized the usefulness of such an approach and recommended further development of the concept, noting that in moving forward there should be a clearer continuum between the findings, action taken and specific recommendations to countries. The GACVS also suggested that this approach to identifying early safety signals could be shared with regions and countries.

GACVS members stated that social media monitoring was important and the EWS appeared to be a significantly useful tool. As the pandemic was likely to continue for several more years, there was considerable added value not only in identifying possible safety signals for COVID-19 vaccines and other vaccines, but also in monitoring public discussion of vaccines and vaccination to complement other approaches to better understand opinions and attitudes about vaccination.

¹ All GACVS statements (accessed 21 July 2021)

⁵ Precision: the total number of relevant documents can be improved by using natural language processing to train the artificial intelligence to understand the meaning of a given text in order to classify the text accurately for analysis.

⁶ Vaccine-related events: vaccine reaction, media report or rumour, new study, temporary suspension, recall of vaccine and vaccine replacement.