Search tips
Search criteria 


Logo of jurbhealthspringer.comThis journalToc AlertsSubmit OnlineOpen ChoiceThis journal
J Urban Health. 2008 March; 85(2): 217–227.
Published online 2008 January 30. doi:  10.1007/s11524-007-9247-6
PMCID: PMC2430112

Assessing Disparities in Adult Vaccination Using Multimodal Approaches in Primary Care Offices: Methodology


Racial disparities in invasive pneumococcal disease and pneumococcal polysaccharide vaccination (PPV) persist despite significant progress. One reason may be that minority patients receive primary care at practices with fewer resources, less efficient office systems, and different priorities. The purposes of this paper are: (1) to describe the recruitment of a diverse array of primary care practices in Pittsburgh, Pennsylvania serving white and minority patient populations, and the multimodal data collection process that included surveys of key office personnel, observations of practice operations and medical record reviews for determining PPV vaccination rates; and (2) to report the results of the sampling strategy. During 2005, 18 practices participated in the study, six with a predominantly minority patient population, nine with a predominantly white patient population, and three with a racial distribution similar to that of this locality. Eight were solo practices and 10 were multiprovider practices; they included federally qualified health centers, privately owned practices and faculty and University of Pittsburgh Medical Center community practices. Providers represented several racial and ethnic groups, as did office staffs. PPV rates determined from 2,314 patients’ medical records averaged 60.3 ± 22.6% and ranged from 11% to 97%. Recruitment of practices with attention to location, patient demographics, and provider types results in a diverse sample of practices and patients. Multimodal data collection from these practices should provide a rich data source for examining the complex interplay of factors affecting immunization disparities among older adults.

Keywords: Pneumococcal polysaccharide vaccine, Adults, Immunization, Primary care.


Racial disparities have been noted for years in pneumococcal polysaccharide vaccination (PPV) rates, despite coverage by Medicare and national recommendations to support PPV use. Nationally, 57.1% of all adults over age 65 years have ever received a pneumococcal polysaccharide vaccine (PPV), with 62% of non-Hispanic whites, 35.6% of non-Hispanic blacks, and 33.2% of Hispanics reporting vaccination.1,2 Furthermore, incidence of invasive pneumococcal disease is significantly higher among black adults over age 65 years than white adults, whether healthy or with underlying chronic conditions.3 Since the introduction of pneumococcal conjugate vaccine for children, the incidence of invasive pneumococcal disease, especially among older adults, has declined,4 as has the disparity in incidence between blacks and whites.5 However, blacks and those with comorbid conditions continue to have a higher risk of invasive pneumococcal disease.4,5

Recent research indicates that blacks and whites are largely treated by different medical providers and that these providers differ in training and in resources available to them.6 Thus, we designed a study to investigate the extent to which practice-level differences in the handling of adult immunizations and the attributes of the medical practice contribute to racial disparities at the population level. Through multimodal data collection that included surveys of providers, medical record review, and observation of office organization and culture, we examined differences among practices serving primarily black patients, those serving primarily white patients, and those with racially mixed populations. Specific aims were to: (1) identify strategies used by the medical practices to facilitate immunizations, including standing orders, patient reminders, express vaccination clinics, flow sheets or registries for tracking immunizations, and provider prompts; (2) characterize the practices to understand current organization and patterns of patient care and critical pathways for potential change; (3) calculate PPV vaccination rates for the various practices; and, (4) using hierarchical regression analyses, calculate the determinants of these rates from the above strategies and characteristics of the practices. The purpose of this paper is to describe the complex data collection methods, types of results anticipated for this study, and preliminary results of the practice recruitment strategy.


General Information

One of the requirements of the request for applications that funded the project was that the investigators compare practices within a geographically defined area. The setting is the Pittsburgh region, which includes the second most elderly metropolitan community in the United States and, among the elderly, a higher proportion of blacks than is their proportion nationally. Several factors determined the selection of practices, including to the extent possible, practices that: (1) served a large percentage of elderly patients; (2) were of diverse types, e.g., internal medicine, family medicine, solo practitioner, multiphysician; (3) had white and minority providers; (4) would be likely to participate; and (5) used electronic medical records. Thus, we intentionally sampled practices based on these factors and the desired racial and socioeconomic characteristics of persons in the census tracts surrounding the practice location.

In designing the practice recruitment strategy and other aspects of the study, we also sought the advice of the Community Research Advisory Board (CRAB) established by the Center for Minority Health at the University of Pittsburgh. The CRAB consists of representatives of many sectors of the minority community including religious leaders, public safety workers, community activists, health care workers, and other opinion leaders. Before submission, investigators presented the proposed study design to the CRAB who provided input into the design and made suggestions for recruiting medical practices serving primarily black patient populations.

Practice Sampling/Recruitment

Allegheny County and its surrounding counties were mapped using 2000 Census data for income and for percent minority population. Practice sites were identified on these maps. In selecting practices, we attempted to match a (solo or multiprovider) practice serving primarily minority patients with a similar practice serving primarily white patients in socioeconomically comparable neighborhoods. We started recruitment of practices with the University of Pittsburgh Medical Center (UPMC) Community Medicine Inc., (CMI) a large, multispecialty network composed of diverse, previously independent, nonacademic practices that had been purchased by UPMC. Approximately 50 practices/offices offer primary care and all use a common electronic billing system. A few of the practices use an electronic medical record, which includes software for tracking preventive services such as immunizations.

The 12 UPMC network practices were supplemented with four community practices, two of which are federally qualified health centers (FQHCs). UPMC practices received a letter from the Medical Director of CMI requesting their participation. All practices were recruited by direct contact from the investigators by telephone, letter, and email to the practice managers, office managers, and/or lead physician. Five practices (three UPMC and two private, solo practitioners) failed to respond to requests to consider participation or refused requests to participate.

Once a practice had agreed to participate, the project coordinator scheduled a visit with the office or practice manager. The purpose of this visit was to introduce the study team, explain all the parts of the study, answer questions, and leave consent forms, surveys, and a brochure about the study.

Clinician and Office Manager Selection

General internists, family physicians, physician assistants, and nurse practitioners with their own patient case loads, seeing elderly patients from the participating sites were eligible for inclusion. Exclusion criteria for clinicians include seeing <50% primary care patients. The office or practice manager and the lead nurse or patient care assistant from each practice were also asked to complete questionnaires. As described below, these surveys were designed to triangulate data collection.

Survey Development

Separate questionnaires were developed for the physicians, nurses, and the office managers. The questionnaires were designed to describe current medical practices and determine barriers and facilitators to organization change that could lead to future adoption of immunization improvement strategies. The PRECEDE-PROCEED framework (Figure 1) was used to develop the surveys. Constructs from this framework included predisposing factors (physician training, Awareness-to-Adherence model); reinforcing factors (incentives, office culture); enabling factors (express vaccine clinics, provider prompts); physician behavior (recommendation to patients, own vaccination status); environmental factors (location of refrigerator relative to exam rooms, staffing ratios, qualifications). A variety of immunization strategies derived from evidence-based reviews by the Task Force on Community Preventive Services in The Guide to Community Preventive Services and by Gyorkos et al.,7,8 were assessed, including use of patient reminders, provider prompts, standing orders, and walk-in vaccination. Mission, office organization, and staffing were determined in both the clinician and office manager surveys. Office organization included leadership, financing, length of visits, medical record management, and quality improvement processes. A published questionnaire9 was used to assess practice culture, organization, and management style using the Competing Values Framework.10,11


The questionnaires differed among physicians, nurses, and office managers, but some questions were asked of more than one type of respondent to allow comparisons among them. The surveys were developed and revised through an iterative process by a multidisciplinary team who examined them for face and content validity.12 They were pilot tested before use and revised accordingly.

Participants were offered $50.00 payment in the form of a check or gift certificate. Survey data were entered twice into an electronic data base, results were compared electronically, and discrepancies were reconciled to reduce keystroke entry error.

Patient Sampling

Two-stage stratified sampling was used to determine the impact of practice and clinician factors on immunization status. The first stage was an intentional sample of diverse practices, as discussed above, that was stratified by race. In the second stage, random sampling of patient records within the practices was conducted, leading to a clustered, random sample of patients. Because of this hierarchical structure, we used a hierarchical linear modeling sample size estimation based on Byrk and Raudenbush.13 Based on an alpha of 0.05 and power of 0.80, to detect at least a 10% difference between practices and between races, for 10 practices with three clinicians per practice, a total of 165 patient records were needed per practice. As a confirmation of the estimated sample size from the hierarchical linear modeling and for analysis of the subgroup analyses, an additional method to calculate sample size was cluster randomization using a t test, as calculated by NCSS/PASS software based on an alpha of 0.05 and power 0.80. For five practices in each stratum, with vaccination rates ranging from 39% to 75%, the number of patients per practice required would be 88–121. Thus, a total of 10 practices with 880 to 1210 patient records would be sufficient based on cluster randomization. To be conservative, the number of practices was increased to 18, using 165 patients per practice, with a goal of 2310 records for review.

For each office, the central billing system, billing computer, or electronic medical record was used to create a list of eligible patients for sampling. Because of HIPAA regulations, record retrieval was performed by a certified honest broker.14 This individual then selected patients who were born before 1/1/1940, who were living and who had an office visit in the last 12 months, indicating that they were active patients of the practice. This list was randomized following an algorithm designed to sample at least 15 patients per physician in a practice and to proportionally distribute the patients by their most frequent primary care provider. (See Appendix.)

Medical Record Review

The certified honest broker scheduled visits to review medical records and signed a confidentiality agreement with those practices requesting one. Using the randomized list generated by the sampling scheme above, the honest broker first reviewed charts to determine eligibility, that is, the patient was born before 1/1/1940 and had at least one visit in 2001 and in 2005. This step was to ensure continuity of care at that practice. If eligible, the patient’s PPV vaccination status and demographic information including race and address (to determine census tract) were extracted from the complete medical record including flow sheets, paper charts, and electronic medical records (EMR). Medical records were reviewed as far back as available for documentation of PPV receipt. The data were entered into an electronic spreadsheet on a laptop computer. The honest broker continued to review medical records from the sampling list at a given practice until a sufficient number (150–175) were collected or until all eligible charts in the practice (for smaller practices) were reviewed.

Following manual record review, for UPMC practices only, an additional search for the same time period was conducted using UPMC’s electronic depository of medical records called the Medical Archival Record System (MARS) and the billing database; however, relatively few vaccines were found that were not in the office record. These two data sets were combined and the records were deidentified. A research assistant identified census tracts for each patient’s address, and the corresponding per capita income for that tract was added to the data.

Medical Practice Observations

A field observation guide used in a previous study15,16 was adapted as a tool to collect data about medical practice characteristics. This guide outlined a variety of features and aspects of medical practices that observers were to pay special attention to, such as physical condition and layout of the office, staff clothing, and quality of interpersonal interactions. It included open- and closed-ended questions and prompts for field notes to collect specific information about procedures and routines of medical practices that facilitate vaccinations, such as use of standing orders, telephone reminders, and posted information regarding vaccinations, and literacy level and cultural appropriateness of educational and reading materials in the office. They also noted practice features that might inhibit vaccination, and aspects of the physical environment and setting of each practice, including transportation routes.

The observation team consisted of four trained observers, any two of whom conducted observations at every office. Observers individually prepared and coded their field notes. These notes were then shared with the lead member of the team for verification of correct and consistent use of the codebook, with differences reconciled by consensus. All three members systematically reviewed the data to develop models of practice culture. Written descriptions of each practice were developed, using data from both the collection sheets and from field notes.

Collection of Timing Data

A timing data collection sheet was developed to facilitate collection of data related to the time patients spent in various segments of the visit from registration to check out.15,16 To collect timing data given privacy restrictions, one observer was stationed in the waiting room and another in a location where patients entering the exam room area could be observed. The waiting room observer noted the time that each person entered and left the waiting room and the description of the clothing he or she was wearing to assist with collation of time observed by the back office observer. The back office observer recorded the time that each person entered the exam room area, entered and left the exam room, and when medical staff, including the PCP, entered and left the exam room. This strategy was selected because it was compliant with HIPAA as the observers do not interact directly with patients while they are receiving medical care. Five visit times will be analyzed: length of time in the waiting room, in the back office area, (including time at a nursing station for vital signs if a separate station is used), in the examination room, time the clinician is in the exam room and the total visit.

Quantitative Data Analysis

As an initial step, descriptive statistics were generated for each of the variables and the necessary assumptions for the planned statistical tests investigated. Statistical significance was set at P  0.05.

Medical record review data were used to calculate vaccination rates for PPV. A practice’s PPV vaccination rate was calculated as the number vaccinated divided by the number in the sample.

Analysis of Predictors of Patient Immunization Status

Hierarchical linear modeling (HLM), which accounts for the concomitant effects of the nested structure, will be used to model multivariable effects of variables in predicting binary outcomes (i.e., received or did not receive PPV vaccination). First, possible associations between vaccination status and selected key variables of interest will be tested; these variables include selected items from the surveys and practice description variables. Second, variables found to be related to vaccination status or to reduced variance will be entered as potential variables in these multivariable models. Race and income based on census tract will be forced into the models as level-one variables to investigate race while accounting for socioeconomic status (SES).


Eighteen practices were recruited into the study with 37 providers, 18 office managers, and 18 head nursing staff completing surveys. The sampling scheme was successful in producing a range of practice types (Table 1). There were eight solo practitioner offices, three with predominantly minority elderly populations (minority), four with white elderly populations (white), and one with a racial mix close to the elderly population in this geographic area (mixed). The ten multi-provider practices were similarly distributed. This resulted in 20% of patients being from minority practices, 22% of patients being from mixed practices, and 58% of patients being from white practices. Per capita income of the census tract in which the practices were located ranged from $7,453 to $24,486/year. The racial representation of physicians and office staff varied as well, with different combinations of race among physicians, staff, and patients. Office structure of the practices included FQHCs, privately owned non-FQHCs, faculty, and UPMC community practices.

Characteristics of participating practices and PPV vaccination rates

The goal of reviewing 2310 medical records was reached as 2314 (100.2%) were actually reviewed. PPV vaccination rate varied from a low of 11% to a high of 97%, with a mean of 60.3 ± 22.6%, and only one practice exceeding national goals of 90% set by Healthy People 2010.17

Comparison of Manual and Electronic Data Collection

Electronic medical records were available to confirm manual record reviews for 14 practices. The difference between PPV immunization rates determined from manual record review only and those determined from a combination of the two methods ranged from 0% to 14.8%, with a mean difference of 3.4 ± 3.8%. Three practices had 0% difference, seven practices had 1.1–3.0% difference, and three practices had 4.6–5.2% difference. The one practice with a 14.8% difference was unable to locate older charts for 17 of its patients, and, therefore, manual review was less complete than at other sites and accounts for the large difference between manual review and combined results for that site.

As this is a methods paper, statistical testing has not been completed. Nevertheless, as Table 1 shows, practices serving high proportions of minority elderly patients tend to be located in neighborhoods with lower per capita incomes. Considerable variation in vaccination rates by practice type was found. Both the lowest and highest vaccination rates were found in solo practices; vaccination rates did not differ between solo and multiprovider sites (P = .138). Although PPV vaccination rate did not differ across practice groups divided into white practices (0–5.8% minority elderly patients), mixed practices (13.7–20.3% minority elderly patients), and minority practices (36.4–97.0% minority elderly patients), there was a significant inverse correlation between percent minority elderly patients and PPV vaccination rate (r = −.619; P = .006).


The interaction between patient race, income, and clinical factors is an important determinant of immunization status. Our data are complex and have yet to be formally analyzed, but already they suggest that patient race, income, and practice type may be correlated. Others have noted the correlation between race and income.18 Given the documented racial disparities in immunization, the logical next step is to clarify the determinants of racial disparities. Furthermore, it is essential that we identify ways to increase immunizations in minority patients in light of the higher attack rates of Streptococcus pneumoniae in certain minority groups. We plan further analyses including hierarchical linear modeling to determine the best practices to maximize adult immunization rates and eliminate racial disparities in this important health promotion measure.

Strengths and Limitations

Among strengths, we consider the benefits of utilizing a variety of underlying models to understand clinician behavior, office culture, and practices that enhance or inhibit vaccination. Second, our questionnaires and observations are second generation, building on our previous work. Third, we have added components not found in our previous work, including the competing values framework.

A possible limitation is conducting this study in one region of the country; however, this region has the second oldest population of any metropolitan area in the country with a high proportion of elderly blacks. Unfortunately, the low number of elderly Hispanic patients prohibits the examination of factors related to their historically low rates of PPV receipt.

Secondly, in some practices it was not possible to review 165 medical records for a variety of reasons including: (1) some physicians had been with the practice for fewer than 5 years; (2) some patients did not have a history with the practice for 5 years; (3) in some practices EMR was implemented within the past 5 years without digitizing earlier paper charts; and (4) some older paper charts were in storage and not accessible. A third limitation is that vaccines can be given in nontraditional settings such as pharmacies, and may not be recorded in the medical record. Some immunizations given elsewhere (specialist offices, the hospital, and the emergency department) were captured with UPMC’s electronic data bases. However, the number of vaccines found that were not in the office record was small (3.4% overall). Although we were unable to similarly capture out-of-office immunizations in non-UPMC practices, a low level of differential review occurred, but we believe this difference to have a minimal effect.

Practices were carefully selected to ensure that they reflect diverse populations, namely, practices that serve high proportions of elderly minority patients, practices that serve high proportions of elderly whites, and some with mixed populations. However, we cannot assume that they represent all types of practices across the country, especially those that serve large numbers of elderly Hispanics and other racial and ethnic groups.


Recruitment of practices with attention to location, patient demographics, and provider types results in a diverse sample of practices and patients. Multimodal data collection from these practices should provide a rich data source for examining the complex interplay of factors affecting immunization disparities among older adults.


The authors wish to acknowledge Pascale M. Wortley, PhD from the Centers for Disease Control and Prevention, for her invaluable advice in the preparation of this manuscript.

This study was supported by the Centers for Disease Control and Prevention Grant No. 5 U01 IP000054-02 and the National Institutes of Health and the EXPORT Health Project at the Center for Minority Health, University of Pittsburgh Graduate School of Public Health, NIH/NCMHD Grant No. P60 MD-000-207. Its contents are the responsibility of the authors and do not necessarily reflect the official views of the CDC, ATPM, or the NIH.


Sampling Scheme

A proportional random cluster sample will be conducted to achieve about 165 patients per practice, nested by clinician.

  1. Site or UPMC will develop list of names of persons aged 65 or older seen in the last year as outpatients. We will assume N patients appear on the list. Inpatient visits are not to be included.
  2. Honest broker will take list and assign each patient to a clinician based on the clinician seen most frequently (over last 3 years).
  3. The list will be stratified by clinician.
  4. If a clinician has <15 patients, then delete clinician and clinician’s patients (revise N to be the number of patients remaining).
  5. If, and only if, N <175, then take all patients for the practice (inclusive sample randomization not necessary).
  6. If N  175, each clinician specific list of patients will be randomized by honest broker as follows below.
  7. Calculate the proportion p of patients to be sampled; p = 175/N.
  8. The number of patients on a particular clinician’s list is n. For each clinician in a practice, calculate n × p. If n × p > 15 for each clinician, then randomly sample n × p patients and sampling is complete. If n × p  15, then go to Step 9.
  9. If n × p  15 for a particular clinician, then randomly sample 15 patients for that clinician and sample n × p patients for the other clinicians.


Zimmerman, Nowalk, Raymund, Tabbarah, and Fox are with the Department of Family Medicine and Clinical Epidemiology, University of Pittsburgh School of Medicine, Pittsburgh, PA, USA; Zimmerman and Terry are with the Department of Behavioral and Community Health Sciences, University of Pittsburgh Graduate School of Public Health, Pittsburgh, PA, USA; Wilson is with the UPMC St. Margaret Family Medicine Residency, Pittsburgh, PA, USA.


1. Centers for Disease Control and Prevention. Early release of selected estimates based on data from the 2006 National Health Interview Survey. Available at 2007. Accessed on August 20, 2007.
2. Centers for Disease Control and Prevention—National Center for Health Statistics. Early release of selected estimates based on data from the January–September 2005 National Health Interview Survey. Available at 2006.
3. Kyaw MH, Rose CE Jr, Fry AM, et al. The influence of chronic illness on the incidence of invasive pneumococcal disease in adults. J Infect Dis. 2005;192:377–386. [PubMed]
4. Lexau CA, Lynfield R, Danila R, et al. Changing epidemiology of invasive pneumococcal disease among older adults in the era of pediatric pneumococcal conjugate vaccine. J Am Med Assoc. 2005;294:2043–2051.
5. Flannery B, Schrag S, Bennett NM, et al. Impact of childhood vaccination on racial disparities in invasive Streptococcus pneumoniae infections. J Am Med Assoc. 2004;291:2197–2203.
6. Bach PB, Pham HH, Schrag D, Tate RC, Hargraves JL. Primary care physicians who treat blacks and whites. N Engl J Med. 2004;351:575–584. [PubMed]
7. Gyorkos TW, Tannenbaum TN, Abrahamowicz M, et al. Evaluation of the effectiveness of immunization delivery methods. Can J Public Health. 1994;85:S14–S30. [PubMed]
8. Task Force on Community Preventive Services. Recommendations regarding interventions to improve vaccination coverage in children, adolescents, and adults. Am J Prev Med. 2000;18:92–96. [PubMed]
9. Lin MK, Marsteller JA, Shortell SM, et al. Motivation to change chronic illness care: results from a national evaluation of quality improvement collaboratives. Health Care Manage Rev. 2005;30:139–156. [PubMed]
10. Zammuto RF, Krakower JY. Quantitative and qualitative studies of organizational culture. Res Organ Change Dev. 1991;5:83–114.
11. Shortell SM, Marsteller JA, Lin M, et al. The role of perceived team effectiveness in improving chronic illness care. Med Care. 2004;42:1040–1048. [PubMed]
12. Aday LA. Designing and Conducting Health Surveys. San Francisco: Jossey-Bass Inc.; 1989.
13. Bryk A, Raudenbush S. Hierarchical Linear Models. Newbury Park, CA: Sage Publications; 1992.
14. Boyd AD, Hosner C, Hunscher DA, Athey BD, Clauw DJ, Green LA. An ‘Honest Broker’ mechanism to maintain privacy for patient care and academic medical research. Int J Med Inform. 2007;76:407–411. [PubMed]
15. Silverman M, Terry MA, Zimmerman RK, Nutini JF, Ricci EM. The role of qualitative methods for investigating barriers to adult immunization. Qual Health Res. 2002;12:1058–1075. [PubMed]
16. Silverman M, Terry MA, Zimmerman RK, Nutini JF, Ricci EM. Tailoring interventions: understanding medical practice culture. J Cross-Cult Gerontol. 2004;19:47–76. [PubMed]
17. U.S. Department of Health and Human Services. Healthy People 2010. With Understanding and Improving Health and Objectives for Improving Health. 2nd ed. 2 vols. Washington, DC: U.S. Government Printing Office; 2000.
18. Krieger N, Rowley DL, Herman AA, Avery B, Phillips MT. Racism, sexism and social class: implications for studies of health, disease, and well-being. Am J Prev Med. 1993;9:82–122. [PubMed]

Articles from Journal of Urban Health : Bulletin of the New York Academy of Medicine are provided here courtesy of New York Academy of Medicine