Search tips
Search criteria 


Logo of bmcphBioMed Centralsearchsubmit a manuscriptregisterthis articleBMC Public Health
BMC Public Health. 2013; 13: 83.
Published online 2013 January 29. doi:  10.1186/1471-2458-13-83
PMCID: PMC3608255

How has the flu virus infected the Web? 2010 influenza and vaccine information available on the Internet



The 2009–10 influenza pandemic was a major public health concern. Vaccination was recommended by the health authorities, but compliance was not optimal and perception of the presumed associated risks was high among the public. The Internet is increasingly being used as a source of health information and advice. The aim of the study was to investigate the characteristics of websites providing information about flu vaccine and the quality of the information provided.


Website selection was performed in autumn 2010 by entering eight keywords in two of the most commonly used search engines ( and The first three result pages were analysed for each search, giving a total of 480 occurrences. Page rank was evaluated to assess visibility. Websites based on Web 2.0 philosophy, websites merely displaying popular news/articles and single files were excluded from the subsequent analysis. We analysed the selected websites (using WHO criteria) as well as the information provided, using a codebook for pro/neutral websites and a qualitative approach for the adverse ones.


Of the 89 websites selected, 54 dealt with seasonal vaccination, three with anti-H1N1 vaccination and 32 with both. Rank analysis showed that only classic websites (ones not falling in any other category) and one social network were provided on the first pages by Yahoo; 21 classic websites, six displaying popular news/articles and one blog by Google. Analysis of the selected websites revealed that the majority of them (88.8%) had a positive/neutral attitude to flu vaccination. Pro/neutral websites distinguished themselves from the adverse ones by some revealing features like greater transparency, credibility and privacy protection.


We found that the majority of the websites providing information on flu vaccination were pro/neutral and gave sufficient information. We suggest that antivaccinationist information may have been spread by a different route, such as via Web 2.0 tools, which may be more prone to the dissemination of “viral” information. The page ranking analysis revealed the crucial role of search engines regarding access to information on the Internet.

Keywords: Internet, Influenza vaccine, Public health, Information


During the 2009–10 flu pandemic, compliance with preventive measures proposed by health authorities was not optimal. The presumed risks associated with vaccination attracted mass media attention, and risk perception among the public was high. Concerns regarding the population-based public health response to this pandemic focused on the need for more effective communication; a BMJ editorial cited an urgent need to restore public trust before the next pandemic comes along [1].

The Internet is increasingly used as a “trusted” source of health-related information, although our knowledge regarding the role of this source in shaping behaviour during a pandemic event is largely unknown. Of particular interest is the role of search engine algorithms, which mainly determine content visibility and user accessibility. Previous studies have investigated the role of different search terms in determining the search engine results. For example, keyword “vaccination” was found to lead to a greater amount of content with an adverse attitude towards vaccination compared to “immunization”, since the antivaccinationists have a strong philosophical preference for the former [2,3].

Recent contributions have pointed out that research on online vaccine information is quite rare and not up to date [4], although there is evidence that exposure to critical websites is associated with changes in risk perception [5]. Previous studies have explored the contents of antivaccinationist websites, analysing their statements and arguments [6-8], and the prevalence of quality markers in websites dealing with vaccination [9]. Other studies have analysed YouTube contents regarding immunization [10], including H1N1 influenza [11] and focused on the Internet’s influence on risk perception from a psychological perspective [12]. Regarding the specific issue of information about flu vaccination on the Internet, one study performed a content analysis of the information provided by the social network Twitter during the H1N1 outbreak [13]; another dealt with the probability of finding WHO recommendations on swine flu prevention [14]. A recent study has even investigated psychological and demographic factors associated with H1N1 vaccination, highlighting the critical role of communication strategies in shaping behaviour [15].

On the basis of the above considerations, this study aimed to evaluate online content regarding vaccination right after the 2009–10 pandemic in order to understand the role that Internet may have had in shaping people’s risk perception. To achieve this objective, we conducted an overall assessment of the websites providing flu vaccine information by evaluating their visibility (by rank analysis), and integrating this with a content analysis. Possible differences in the results provided by different search engines were also assessed.


Website selection

Website selection was performed, from an Italian IP address, in autumn 2010, via the two most commonly used search engines: and [16]. Eight key phrases (“flu vaccine”, “flu vaccination”, “flu immunization”, “flu shot”, “influenza vaccine”, “influenza vaccination”, “influenza immunization” and “influenza shot”) were entered and the first three pages of results (each showing ten occurrences) were downloaded for each of them, giving a total of 480 occurrences. The selected key phrases were chosen by the authors assuming that an average Internet user would likely use them to make simple searches on the Web regarding influenza immunization.

The preliminary exclusion criteria were:

websites not available in English;

websites requiring registration for access;

Web pages merely containing links to other pages (portals);

websites which did not lead us to the research content within three clicks of the occurrence;

websites dealing with non-human-related matters (e.g. veterinary issues);

websites targeting health professionals only.

The advanced exclusion criteria were:

websites based on web 2.0 philosophy (e.g. blogs, social networks, communities, forums);

websites displaying popular news/articles;

single files (e.g. pdf, ppt).

We considered as “classic websites” those not falling in any of the above categories.

The search engine results were immediately saved in order to avoid variation in ranking. Web pages occurring more than once were only taken into account the first time.

The websites selected when applying both preliminary and advanced exclusion criteria were analysed. To allow a comprehensive rank assessment, only preliminary exclusion criteria were applied to the search engine results.

Rank analysis

Page rank was assessed for all the occurrences on the first page only (corresponding to the first ten results) for Google and Yahoo which fitted the preliminary exclusion criteria. The results provided by the two search engines were then compared.

Website coding

The websites selected when applying both preliminary and advanced exclusion criteria underwent a two-step analysis, of the website itself and of the information provided.

All the websites were analysed by one author. Critical points were discussed with the other authors and differences were resolved by consensus.

1. Analysis of the website

1a. Classification of adverse vs. pro/neutral websites

Websites were considered adverse if they explicitly recommended that users should not comply with public health recommendations, suggested users try alternative preventive methods, declared that evidence about vaccine effectiveness/safety is inadequate, or overemphasized the risks associated with vaccination. Web pages not fitting such criteria were considered “pro/neutral” [3,17].

1b. Comparison of adverse vs. pro/neutral websites with regard to the WHO’s “Good Information Practice Essential Criteria for Vaccine Safety Web Sites”

Website analysis was performed on the basis of the WHO’s “Good Information Practice Essential Criteria for Vaccine Safety Web Sites” [18], partially simplified in respect of the original version but keeping the essential criteria. It specifically regarded the following:

general information: extension and type of website;

credibility: mission of site, disclosure of ownership or source, transparency of sponsorship, accountability to users, data protection and responsible partnering;


design: logical organisation, ease of navigation, consistent plan and professional presentation;

content: authority of sources, accuracy, currency and review process.

For each of the above we assessed whether the selected websites satisfied the specific requirements proposed by the WHO (cited in full in Table 1) or they did not and compared websites with opposing attitudes towards flu vaccination.

Table 1
Website analysis according to the WHO’s “Good Information Practice Essential Criteria for Vaccine Safety Web Sites”

2. Analysis of the information provided

2a. Analysis of the information provided by pro/neutral websites

The information provided by pro/neutral websites was analysed according to the content analysis method [19], using a codebook that enabled the following information to be collected: type of flu vaccine (seasonal flu vaccine, H1N1 flu vaccine, or both: if not specified, seasonal flu vaccine was considered), general information, administration route, doses, administration period, indications, effectiveness, contraindications, benefits and risks.

For each of the above we evaluated the presence of information, completeness and adherence to the gold standard - represented by the “WHO Position Paper on Influenza” [20] and “Vaccine for pandemic H1N1 2009” [21] - and whether scientific bibliographical references were provided.

Websites providing information on both types of vaccine were analysed twice: with regard to information about the H1N1 vaccine and to information about the seasonal vaccine.

2b. Analysis of the information provided by adverse websites

A qualitative approach was used for analysing the information provided by adverse websites, to assess the presence of arguments used by the authors to support their attitude.

Statistical analysis

Data on the comparison of adverse vs. pro/neutral websites with regard to the WHO criteria and on the comparison of information about seasonal and H1N1 flu vaccine were analysed using STATA (Stata Statistical Software Release 8.0, 2003; Stata Corporation, College Station, TX). We adopted a descriptive approach and calculated statistical significance using the X2 test, rejecting the null hypothesis below a p-value of 0.05.


An overall flow-chart of the analysis process is shown in Figure 1.

Figure 1
Flow-chart of the analyses performed.

Website selection

Using the above preliminary and advanced exclusion criteria, we selected 89 websites providing information about flu vaccination.

Rank analysis

A total of 49 different occurrences fitting the preliminary inclusion criteria were shown on the first search engine page for each key phrase.

Differences were revealed with regard to the kind of website shown by the two search engines: whereas Yahoo only showed classic websites and one based on Web 2.0 philosophy, Google ranked on the first result page 21 classic websites, six displaying popular news or articles and one based on Web 2.0 philosophy. In addition, whereas Google always ranked Wikipedia first, Yahoo ranked four websites: three with a .gov extension (,, and Wikipedia. Two adverse websites appeared on the first page of Google and only when entering two key phrases (“flu shot” and “flu vaccine”), while no adverse website appeared on the first page of Yahoo.

Figure 2 shows the positions of occurrences on the first page of both the search engines when the eight key phrases were entered. Blank spaces correspond to non-eligible websites.

Figure 2
Page Rank Analysis. Figure Figure22 shows the positions of occurrences obtained on the first pages of both search engines when entering the eight key phrases. Blank spaces correspond to non-eligible websites.

Website coding

1. Analysis of the website

1a. Classification of adverse vs. pro/neutral websites

Analysis of the websites revealed that the majority (88.76%; n 79) had a positive/neutral attitude to flu vaccination, and only ten of them (11.23%) were adverse.

1b. Comparison of adverse vs. pro/neutral websites with regard to the WHO’s “Good Information Practice Essential Criteria for Vaccine Safety Web Sites”

The most frequent extension was .com (60% of adverse and 35.4% of pro/neutral websites, p>0.05), followed by .org (10% of adverse and 16.5% of pro/neutral websites, p>0.05). The extensions .gov and .net characterized pro/neutral websites only (16.5% and 2.5% respectively, p>0.05; data not shown).

Remarkable differences were noted between websites with opposing attitudes. The results of this comparison are shown in Table 1.

2. Analysis of the information provided

2a. Analysis of the information provided by pro/neutral websites

Of the pro/neutral websites, 51 dealt with seasonal flu vaccine only, three with anti-H1N1 flu vaccine only, and 25 both of them.

Table 2 shows the percentages of websites dealing with seasonal and H1N1 flu vaccine which offer general information about the vaccine and notions about administration routes, doses, vaccination period, indications, effectiveness, contraindications, benefits and risks. The percentages of websites providing comprehensive and coherent information in relation to gold standard and scientific bibliographic references are reported for each of these topics.

Table 2
Analysis of the information provided by the websites. Comparison of information about seasonal and H1N1 flu vaccine

2b. Analysis of the information provided by adverse websites

The analysis was performed on nine of the ten selected classic websites with an adverse attitude to flu vaccination, since one of them disappeared from the Web after selection.

The websites dealt with several arguments, as shown in Table 3. Danger to human health related to vaccination was the most widely recurring one: 100% of the websites analysed provided descriptions of possible adverse reactions, often (55.5%) recounting dramatic personal experiences (“After getting it, I became ill, not really bad but sort of nagging lingering feeling of not being well […] I woke up next morning with my left arm paralyzed…”).

Table 3
Recurring arguments on websites with an adverse attitude to seasonal/H1N1 flu vaccine

Six of them (66.7%) provided a list of the vaccine’s constituents and their respective risks for human health (“Do you want any of these vaccine constituents in YOUR blood stream?”).

Nearly all the websites (88.9%) argued that flu vaccination was useless, mainly because of poor matches between the viruses in the vaccines and those actually infecting people (“…vaccines are being used as an ideological weapon. What you see every year as the flu is caused by 200 or 300 different agents with a vaccine against two of them. That is simply nonsense”).

Seven of them (77.8%) stated that vaccination policy is driven by the enormous economic interests of pharmaceutical manufacturers, whose profits are boosted by exaggerated government claims.

According to six websites (66.7%), control over mass media leads to a substantial ignorance on the part of the public and influences personal choices, whereas everyone should be able to make informed decisions with regard to his or her own health.

Five websites (55.5%) dealt with doctors’ unreliability and the inefficiency of traditional medicine and/or invited readers to consider alternative medicine and homeopathic or natural remedies.

According to four websites (44.4%), the mass media and government are responsible for unjustified panic about flu, the severity of which is greatly overestimated (“Aspirin kills 400% more people than H1N1 swine flu”) and the same number of websites referred to “enlightened” physicians denouncing the system, some of them (22.2%) being authors themselves.

Two websites put forward ethical arguments, based on the use of pork products in vaccine manufacture (“…to separate cells we have to use a pork product - pork trypsin - and a lot of people who would not ingest pork products for religious reasons are seduced without knowing it into violating their convictions…”) and referred to alleged tests on homeless people in Poland.

Unusual theories were discussed by two authors: one claimed that seasonal flu vaccination greatly increases the risk of contracting H1N1 flu by weakening the immune system; the other described the 1918 “Spanish Influenza” as the “Frankenstein monster” created by American doctors with their vaccines.

In addition to recurring arguments, adverse websites showed other common features, such as the above-mentioned predilection for narrating personal experiences and a significant propagandistic attitude. On six websites (66.6%) we found an invitation to promote the anti-vaccination movement by involving new people, joining activist associations, writing articles for local newspapers, and selling alternative products and information packs online. One website even contained a link to an anti-vaccination song entitled “Don’t inject me”.

Only one article invited readers to discuss with their doctors whether they should get vaccinated.


According to the above findings, the majority of the selected websites had a positive/neutral attitude to flu vaccination. They distinguished themselves from the adverse ones by some revealing features: firstly, greater transparency, in relation to more frequent declaration of the sources of funding; secondly, greater credibility, since in a statistically higher percentage of cases explicitly declared the website mission and the public to whom the information is addressed. In addition, their content always adhered to their stated mission (not the same for adverse websites) and they more frequently declared the owner’s name and credentials. Furthermore, they were characterized by higher data and privacy protection and by more professional appearance and consistent design. Lastly, they showed a significantly higher amount of copyrighted information, which can be considered as an expression of responsibility for content.

The quality of the information provided by pro/neutral websites was generally satisfactory. In particular, adherence to the gold standard was optimal, whereas data about completeness were less satisfactory, especially on websites dealing with seasonal flu vaccine.

The presence of information about vaccination period, risks and contraindications was statistically higher for the seasonal flu vaccine compared to the H1N1 vaccine, so it is possible to assume that the set of information available on the Web about the former is more exhaustive and scientifically structured. Completeness of the information about contraindications was also better for the seasonal vaccine, while information about doses was more complete for the H1N1 vaccine.

Previous studies have pointed out that the influence of Internet information on risk perception, decision process and hence vaccination behaviour does not always act in a conscious way; the fact that individuals do not admit to considering the Internet a reliable source does not necessarily mean that it does not influence their choices [22,23]. Based on the above results, the way adverse websites were formulated seems to be consistent with the purpose of influencing the readers’ attitude towards vaccination. Some arguments were often repeated: danger of vaccine due to adverse affects and toxic constituents, economic interests driving vaccination policy (the “plot theory”), unreliability of institutional medicine. Furthermore, a narrative form widely prevailed. Such personal and emotional narration, frequently regarding children and performed between parents, has been proved to be a critical factor of the effects of antivaccinationist information [5,23], since it works on readers’ emotions and fears.

Search engine ranking refers to the position at which a particular occurrence appears in the results of a search engine query. As a consequence, it is closely connected with visibility to users.

Differences were highlighted in the kind of results provided by the search engines. In particular adverse websites were shown on the first page by Google only, and Yahoo results were characterized by a higher presence of institution (.gov extension) in the first positions, which we expect to contain a more reliable information [24].

Page rank is a crucial factor based on what can be called “the internet paradox”: although the Web contains virtually unlimited information, it has been observed that users generally do not go beyond the first page of search engine results and have a low tolerance of going in depth through what is retrieved [25].

One particular strength of this study is that we performed a broad-spectrum analysis on different levels: visibility according to page ranking, analysis of the websites and analysis of the information provided, choosing for websites with opposing attitudes to flu vaccination the methods which most suited their features.

One limitation of the study is intrinsic to the Internet: information on the Web is constantly changing, whereas we analysed the Web pages which were available at a particular time. Since we carried out website selection in autumn 2010, some months after the flu pandemic peaked, we realise that page rank may well have changed in the meantime, as the topic was becoming of less pressing actuality.

Secondly, our assessment of the information provided using different methods of analysis, based on the websites’ attitude towards flu vaccination, was due to the very nature of the information. Actually, there was an extreme difference in the way pro/neutral and adverse websites were structured. Adverse websites, far from providing a structured set of information, as the pro/neutral ones did, showed a prevalence of narrative form, rich in dramatic personal experiences which were generally related to adverse reactions to the vaccine. The use of a fixed analysis scheme would have prevented us from assessing the highly intriguing topics and theories used by antivaccinationist authors to support their attitude and spread it among the people.

Moreover, Web 2.0 was excluded from the analysis. This was due to its extremely dynamic structure, which requires specific analysis strategies and makes it impossible to univocally assign an attitude to several comments, made by as many authors, on the same Web page.

Web 2.0 is continuously expanding and it is becoming a common way to obtain health information. Even important health institutions such as the CDC use the new communication tools, like mobile phone applications, in order to spread information among the population. We therefore suggest that a possible future development of the research could interestingly extend it to this area of the Internet, also in view of strategically exploiting its potential by the public health authorities.

Furthermore, we decided to limit the research to the two most commonly used search engines ( and, although others, such as and, are also very popular [16], and to a limited set of key phrases.


The vast majority of the websites analysed had a positive/neutral attitude towards flu vaccination and overall they provided satisfactory information. They also were characterized by a certain presence of reliable organisations and institutions, for example the CDC (Centers for Disease Control and Prevention) official website and (a government website managed by the U.S. Department of Health & Human Services). This naturally raises the question as to whether antivaccinationist information on the Web could take different routes, such as Web 2.0. If confirmed, this would entail a new challenge for the public health authorities. Promotion of prevention on the Internet by providing traditionally structured information may no longer be sufficient and the use of new strategies, such as Web 2.0 tools, may be more effective as they may be more likely to spread information via “viral” dissemination.

Competing interests

The authors declare that they have no competing interests.

Authors’ contribution

UG, LC, LC and GO conceived and designed the study. AC and SM acquired the data. UG, LC, AC and SM analysed and interpreted the data. All the authors were involved in drafting the manuscript and revising it critically and have given their final approval of the version for publication.

Pre-publication history

The pre-publication history for this paper can be accessed here:


  • Godlee F. Conflicts of interest and pandemic flu. BMJ. 2010;340:C2947. doi: 10.1136/bmj.c2947. [PubMed] [Cross Ref]
  • Davies P, Chapman S, Leask J. Antivaccination activists on the world wide web. Arch Dis Child. 2002;87:22–25. doi: 10.1136/adc.87.1.22. [PMC free article] [PubMed] [Cross Ref]
  • Wolfe R, Sharp L. Vaccination or immunization? The impact of search terms on the internet. J Health Commun. 2005;10(6):537–551. doi: 10.1080/10810730500228847. [PubMed] [Cross Ref]
  • KARE Knowledge Acquisition, Reuse and Evaluation. Ontology and Semantic Web Services for Knowledge Management. 2010. Third International Workshop on Knowledge Acquisition, Reuse and Evaluation. Available: Accessed Sept 2010.
  • Betsch C, Renkewitz F, Betsch T, Ulshöfer C. The influence of vaccine-critical websites on perceiving vaccination risks. J Health Psychol. 2010;15(3):446–455. doi: 10.1177/1359105309353647. [PubMed] [Cross Ref]
  • Wolfe R, Sharp L, Lipsky M. Content and design attributes of antivaccination web sites. JAMA. 2002;287(24):3245–3248. doi: 10.1001/jama.287.24.3245. [PubMed] [Cross Ref]
  • Zimmerman RK, Wolfe RM, Fox DE, Fox JR, Nowalk MP. Vaccine criticism on the World Wide Web. J Med Internet Res. 2005;7(2):e17. doi: 10.2196/jmir.7.2.e17. [PMC free article] [PubMed] [Cross Ref]
  • Kata A. A postmodern Pandora's box: Anti-vaccination misinformation on the Internet. Vaccine. 2010;28(7):1709–1716. doi: 10.1016/j.vaccine.2009.12.022. [PubMed] [Cross Ref]
  • Mayer MA, Leis A, Sanz F. Health information on the Internet and trust marks as quality indicators: vaccines case study. Aten Primaria. 2009;41(10):534–542. doi: 10.1016/j.aprim.2009.02.003. [PubMed] [Cross Ref]
  • Keelan J, Pavri-Garcia V, Tomlinson G, Wilson K. YouTube as a Source of Information on immunization: a content analysis. JAMA. 2007;298(21):2482–2484. [PubMed]
  • Pandey A, Patni N, Singh M, Sood A, Singh G. Youtube as a source of information on the H1N1 influenza pandemic. Am J Prev Med. 2010;38(3):e1–e3. doi: 10.1016/j.amepre.2009.11.007. [PubMed] [Cross Ref]
  • Betsch C. Innovations in communication: the internet and the psychology of vaccination decisions. Euro Surveill. 2011;16(17):pii: 19849. [PubMed]
  • Chew C, Eysenbach G. Pandemics in the age of Twitter: Content analysis of tweets during the 2009 H1N1 outbreak. PLoS One. 2009;5(11):e14118. [PMC free article] [PubMed]
  • Gesualdo F, Romano M, Pandolfi E, Rizzo C, Ravà L. Surfing the web during pandemic flu: availability of World Health Organization recommendations on prevention. BMC. 2010;10:561. [PMC free article] [PubMed]
  • Bish A, Yardley L, Nicoll A, Michie S. Factors associated with uptake of vaccination against pandemic influenza: a systematic review. Vaccine. 2011;29(38):6472–6484. doi: 10.1016/j.vaccine.2011.06.107. [PubMed] [Cross Ref]
  • Top Ten search Engines. Available: Accessed Jun 2010.
  • Tozzi AE, Buonuomo PS, Atti ML Cd, Carloni E, Meloni M. Comparison of quality of Internet pages on Human Papillomavirus immunization in Italian and in English. Adolesc Health. 2010;46(1):83–89. doi: 10.1016/j.jadohealth.2009.05.006. [PubMed] [Cross Ref]
  • World Health Organization. Good information practice essential criteria for vaccine safety websites. Available: Accessed Jun 2010.
  • Riffle D, Lacy S, Fico FG. Analyzing media messages. Using quantitative content analysis in research. Mahwah, New Jersey: Lawrence Erlbaum; 2005.
  • World Health Organization. Position Paper on Influenza. Wkly Epidemiol Record. 2005;33(19 Aug):279–287.
  • World Health Organization. Vaccine for pandemic H1N1 2009. Available: Accessed Jun 2010.
  • Fitzsimons GJ, Hutchinson JW, Williams P. Non-conscious Influence on Consumer Choice. Marketing Letters. 2002;13(3):267–7.
  • Betsch C, Ulshöfer C, Renkewitz F, Betsch T. The influence of narrative v. statistical information on perceiving vaccination risks. Med Decis Making. 2011;31(5):742–753. doi: 10.1177/0272989X11400419. [PubMed] [Cross Ref]
  • NIH. Evaluating Health Information. Available: Accessed Nov 2012.
  • Spink A, Wolfram D, Jansen MBJ, Saracevi T. Searching the web: The public and their queries. Journal of The American Society For Information Science And Technology. 2001;52(3):226–234. doi: 10.1002/1097-4571(2000)9999:9999<::AID-ASI1591>3.0.CO;2-R. [Cross Ref]

Articles from BMC Public Health are provided here courtesy of BioMed Central