|Home | About | Journals | Submit | Contact Us | Français|
To identify, prioritize, and organize components of a cultural competence curriculum to address disparities in cardiovascular disease.
In 2006, four separate nominal group technique sessions were conducted with medical students, residents, community physicians, and academic physicians to generate and prioritize a list of concepts (i.e., ideas) to include in a curriculum. Afterward, 45 educators and researchers organized and prioritized the concepts using a card-sorting exercise. Multidimensional scaling (MDS) and hierarchical cluster analysis produced homogeneous groupings of related concepts and generated a cognitive map. The main outcome measures were the number of cultural competence concepts, their relative ranks, and the cognitive map.
Thirty participants generated 61 concepts, 29 were identified by at least 2 participants. The cognitive map organized concepts into four clusters, interpreted as: (1) patient’s cultural background (e.g.,, information on cultures, habits, values); (2) provider and health care (e.g., clinical skills, awareness of one’s bias, patient-centeredness, and professionalism), communication skills (e.g., history, stereotype avoidance, and health disparities epidemiology); (3) cross-culture (e.g., idiomatic expressions, examples of effective communication); and (4) resources to manage cultural diversity (e.g., translator guides, instructions and community resources). The MDS two-dimensional solution demonstrated a good fit (stress=0.07; R2=0.97).
A novel, combined approach allowed stakeholders’ inputs to identify and cognitively organize critical domains used to guide development of a cultural competence curriculum. Educators may use this approach to develop and organize educational content for their target audiences, especially in ill-defined areas like cultural competence.
The Institute of Medicine has suggested that a culturally-sensitive approach to care that avoids bias, prejudice, and stereotyping may help reduce health care disparities.1 Multiple frameworks have been proposed to improve cultural competence among health care providers.2–10 Cultural competence training has covered many different topics, has taken any number of different shapes, and has been shown to improve provider knowledge, attitudes, and skills, although patient outcomes are not frequently assessed.11,12 However, no clear consensus is available for the organization of content that ought to be included in a cultural competence curriculum for physicians or physicians-in-training.
Processes more frequently employed in the social sciences have been useful in gathering and organizing data that can shape educational products. For instance, nominal group technique (NGT) 13,14 offers advantages over traditional focus group discussions by allowing equal participation, avoiding participant’s inhibition in informal group discussion, avoiding focusing on one particular area, reducing apprehension from public speaking, generating a greater number of ideas, and minimizing inefficiencies and process loss.15–17 NGT has been used in medical education to assess trainees’ perceptions of successful teaching rounds,18 to evaluate curricula,19 and to identify key attributes of an ideal continuing medical education (CME) institution to create a roadmap for excellence,20 among others. In medical practice, NGT has been used to identify intervention strategies to improve osteoporosis treatment in nursing homes, 21 identify reasons for lack of medication adjustment in patients with seemingly uncontrolled hypertension, 22 to identify barriers for diabetes self-management, 17 and to achieve consensus among stakeholders in guidelines development.23 Importantly, NGT does not limit participants to a pre-determined framework; rather, it forces participants to consider any and all options.
In addition, grouping methods allow the visual representation of mental processes, such as the data gathered during NGT, for instruction. The goal is to reveal the underlying cognitive dimensions, or “cognitive map,” that describes the spatial representation of how the items generated during NGT are related to each other and how ideas are prioritized within each cluster.24–27 Card-sorting, hierarchical cluster analysis, and multidimensional scaling have been used to organize information and understand group similarities,28–32 to understand consumer behavior,33 to design the contextual navigation of Web sites,28,34,35 to provide insight on how journals are cognitively organized,26 and to understand motivation in schools.36 However, to our knowledge, such methods have not been used in medical education research.18,37
We set out to inform the development and organization of a cultural competence curriculum that would increase health care professionals’ knowledge of health disparities and fundamental issues of cultural competence as a step in enhancing cardiovascular care for minority groups. We did so by engaging groups of relevant stakeholders (including cultural competence experts, physicians, and physicians-in-training) in a robust, formative assessment to identify, prioritize, and organize the critical components of cultural competence.22 The combined approach we describe is a novel application in medical education research; the qualitative and quantitative approach is specifically designed to guide the process of content development and organization of information.
We report below the development of the Cultural Competence Curriculum program, an Internet-based educational intervention for physicians and physicians in training aimed at reducing health disparities in cardiovascular health.
First, we used NGT to generate and prioritize a list of concepts deemed important to include in the curriculum. As explained earlier, NGT is a highly structured, small-group-meeting approach used to elicit and organize answers to a specific question.18 NGT involves the following steps: (1) silent, written generation of responses to a specific question; (2) round-robin recording of ideas; (3) discussion for clarification only; and (4) voting on the rank order of the top three ideas, usually with respect to importance, feasibility, or other characteristics. (These steps are described in much greater detail elsewhere 18). Figure 1 is a graphical description of the question development, the NGT groups, and how the responses were processed to the question that was developed.
To formulate the central question for our NGT sessions, we convened a panel of clinician-educators, nurses, minority community physicians, minority health researchers, health services researchers, social scientists, and distance-learning experts. The panel reviewed the literature and composed and pilot-tested the question among medical students, faculty members, and a community physician.
Between February and March 2006, we conducted four separate NGT sessions with medical students (n = 9), internal medicine residents (n = 7), primary care community physicians (n = 7), and disparities or educational researchers (n = 7). The small sample size was by design and is justified, given NGT methodology where the process allows for idea saturation quickly with smaller numbers of participants, provided the demographic composition of the groups broadly represents the target end-user. We tried to have participants who represented a mixture of age, gender, and ethnicity; however, by design, we did not ask or collect information on such characteristics, since such inquiries could have influenced the responses during the NGT sessions.
We recruited two convenience samples of fourth-year medical students and internal medicine house officers from The University of Alabama at Birmingham, Birmingham, Alabama. We recruited community physicians from the Alabama Practice-Based CME Network, a group in regular communication with the academic medical center, for purposes of exploring educational need in the community and easy delivery of educational materials and activities addressing those needs. Participants were recruited via fax and awarded a $100 honorarium. Finally, we recruited investigators funded by the National Heart, Lung, and Blood Institute.38 NGT participants were informed that the purpose of the session was to generate their ideas concerning curricular content, not format, for a cultural competence program focusing on cardiovascular disease risk and treatment; the program would be available to medical students, residents in training, and practicing physicians.
The final question formulated by the panel and posed to NGT participants was “What sorts of things could be included in a curriculum that focuses on cultural competence training for physicians?” Once all responses to the question had been given, each group was asked to identify the three most important ideas and the three most feasible ideas that the responses contained. Experienced facilitators conducted the sessions.
The ideas from the NGT sessions that received at least 2 votes as “most important” or “most feasible” were then organized by a group consisting of a convenience sample of 45 educators and disparities researchers. These participants physically grouped and ranked the ideas, which were written on index cards. These educators and researchers hailed from 19 cities and 14 states in the United States. There were 29 general internal medicine physicians, one pediatrician, three specialists, four nurses, four PhDs, and four from other professions. Of these participants, 26 were women, 19 were men; there were 24 Caucasians, 8 African Americans, 6 Hispanics, 3 Asians, 3 unknown, and 1 “other.”
Each participant worked independently for 10–20 minutes to
Last, within each pile, participants ranked the ideas, again based on their own criteria, by placing the most important idea at the top of the pile and the least important idea at the bottom of the pile. We did not provide an incentive or honoraria.
To analyze the results of the card sort, we used multidimensional scaling (MDS) and hierarchical cluster analysis (HCA).22 The goal of MDS is to reveal the psychological dimensions in the data that can meaningfully describe the underlying cognitive constructs. The result, a “cognitive map,” is a spatial representation of how the ideas are considered to be similar to or different from each other.26,39 The coordinates defining each item’s position from the MDS analysis are used as input data in the cluster analysis. The goal of HCA is to facilitate the interpretation of the MDS map by dividing a set of objects into a smaller number of homogeneous groups on the basis of their similarity.40 Enclosed within lines, the smaller homogeneous groups represent “families,” and clusters of groups represent “neighborhoods.” 41 The map in our study reflected how the card sorters cognitively organized the areas to include in the cultural competency training curriculum for physicians.
The data from the card-sorting task were first used to create a matrix for each participant. This co-occurrence matrix indicated the number of times each item was sorted with all other items (29×29 matrix). The individual co-occurrence matrices were then aggregated across all participants to form a group co-occurrence matrix. This matrix indicated the number of times all participants sorted each reason into similar or dissimilar piles. The group co-occurrence matrix was then analyzed with MDS (using the ALSCAL algorithm) and with cluster analysis to derive the cognitive map. This iterative process derived an optimal geometric solution that reflects the proximity of the data and the relative ordering from the distances observed in the matrix.24
The number of dimensions in an MDS solution is determined by its interpretability and analysis. Perhaps the most important is the interpretability. A two-dimension solution is usually recommended because additional dimensions make interpretation more difficult.41 MDS analysis results in the R-square (RSQ) and stress statistics. The RSQ statistic reflects the extent to which the map corresponds to the actual proximity data, and values of RSQ >.90 suggest high correspondence. The stress statistic reflects the “badness of fit,” with acceptable stress values being >.01 and <.20 (<.01 indicates a degenerate solution, >.20 indicates poor fit). 40,41
Although MDS and cluster analysis are computationally sophisticated, they are not based on parametric statistics. Consequently, their validity is not sample-size dependent, but dependent on the representativeness of participants in the sorting task. While the location of items on the map is analytically determined, the meaning of the dimensions (i.e., the two axis labels) must be interpreted. We used a standard approach to interpret the derived dimensions by engaging the entire research team and by examining the substantive differences between items at the extremes of the dimensions.
The Institutional Review Board at the University of Alabama at Birmingham approved this study.
During the 4 independent NGT sessions, a total of 61 ideas were generated. Twenty-nine of these ideas were selected by at least 2 participants as most important or feasible; they are all listed in Table 1. The other 32 ideas were discarded and not incorporated into the resulting Web-based program’s content (a list of these discarded ideas is presented in Supplemental Digital Table 1, which may be found at http://----------------------------). [INSERT LINK HERE] As shown in Table 1, the top-rated idea among experts was to “include content to help learners become aware of [their] own biases and attitudes…” (item 17, sum of the ratings = 9); among community physicians the two top-rated ideas were “describe habits that are unique…” (item 13, sum of the ratings = 7), and “present objective data on the cultures …” (item 12, sum of the ratings = 7); among residents it was to “provide access to resources…” (item 6, sum of the ratings = 6); and, among medical students it was to “identify critical questions that would permit taking cultural history …” (item 1, sum of the ratings = 17). Table 1 shows the organization of the 29 ideas selected by at least 2 participants as most important or feasible, classified by family (i.e., small group).
The 45 clinician-educators and health disparities researchers organized the 29 ideas in a median of 3 piles (range 1–11); each pile had a median of 6 ideas (range 3–10).
The multidimensional scaling (MDS) and hierarchical cluster analysis revealed two possible solutions, on two or three dimensions. Although the three-dimension solution better explained the data than did the two-dimension solution (higher RSQ, 0.97 vs. 0.91; lower stress measure, 0.07 vs. 0.15), we present here the two-dimension solution, as it was the most parsimonious and interpretable.
The cluster analysis revealed ten small groups, or families, of cognitively most-similar items, each within four larger groups, or neighborhoods, of cognitive similarity. The decision to interpret a ten-cluster solution was informed by examining the pattern of the agglomeration coefficients and by visually inspecting the two-dimension plot and the dendrogram to determine the sequence of cluster formation. The cognitive map resulting from the MDS and cluster analysis is shown in Figure 2.
We propose that the four larger groups can be viewed as neighborhoods of related themes: (1) patient’s cultural background (i.e., information on cultures, habits, values); (2) provider and health care (e.g., clinical skills, awareness of one’s bias, patient-centeredness, and professionalism), communication skills (e.g., history, stereotype avoidance, and health disparities epidemiology); (3) cross-culture (e.g., idiomatic expressions, examples of effective communication); and (4) resources to manage cultural diversity. These neighborhoods are shown in Table 1 and Figure 2, which give a fuller picture of them than could be presented in this paragraph. We base our interpretation of the mapping results on the substantive orientation and thematic consistency of the ideas within these clusters of families and among neighborhoods.
The patient’s background neighborhood includes three distinct families: cultural background (e.g., religion, customs, moral values), culture and health (e.g., impact of religion and culture behavior and adherence to cultural standards), and social background (e.g., health patterns, views on complementary and alternative medicine, region of the country).
The “provider and health care neighborhood includes three families: clinical skills (e.g., awareness of one’s bias, patient-centered, professionalism), communication skills (e.g., history, stereotype avoidance), and health disparities epidemiology (e.g., prevalence, causes, medications).
The cross-culture neighborhood includes two families: cross-cultural communication (e.g., idiomatic expressions, barriers, effective communication examples) and cross-cultural resources.
The “resources to manage cultural diversity neighborhood consists of two families: translator guides/instructions and community resources.
In Figure 2, we conceptualize the x-axis as a dimension of patient centeredness (Patient Centrism) with a neighborhood of issues more intrinsic to the patient (Patient’s Background, on the left side of the figure) and a neighborhood of issues more extrinsic to the patient (Resources, on the right side). Similarly, we conceptualize the y-axis as a dimension of health-care centeredness (Provider and Health- System Centrism) with a neighborhood of issues more intrinsic to health care (Provider/Health Care, on the upper portion of the figure) and a neighborhood of issues more extrinsic to health care (Resources and Cross-Culture, on the lower portion of the figure).
We arrived at these interpretations by examining the ideas at the extremes on each axis to determine similarities and differences. For instance, with respect to the dimension of patient centeredness (the Patient Centrism axis), differences between the two items farthest apart were relatively obvious. Item 8 (“Provide a list of available translator services”) is not under the control of the patient or would be the same across all patients. Item 15 (“Provide information on religious principles of particular cultures and how these might influence medical care”) is very individualized to the patient and not the same from patient to patient. However, this approach was more difficult in the case of the dimension of health care centeredness (the Provider/Health-System-Centrism axis), where the variance between extremes was not as great. Consequently, we had to examine other items near the extremes to find meaning. Ultimately, we concluded that the items near the extremes on this axis also related to the level of control of the physician or health care system.
Importantly, the Patient’s Background and the Provider/Health Care neighborhoods seem to be the most important, as illustrated by the higher overall sums of ratings of 56 and 55, respectively, compared with the Cross-Culture or Resources neighborhoods, with sums of ratings of 25 and 23, respectively.
As explained earlier, in a formative evaluation of concepts important and feasible for inclusion in a cultural competence curriculum, medical students, residents, community physicians, and academic physicians identified and organized 29 ideas along the two axes of patient centeredness and provider/health-care-system centeredness (illustrated in Figure 2). The resulting cognitive map was interpreted to include four neighborhoods of concepts: patient’s background, provider/health care, cross-culture, and resources to manage cultural diversity; each of them contained a family of ideas related to the neighborhood’s concept. We extended the methods and analytical approach from the social sciences to effectively outline a curriculum for cultural competence training. Our goal with this process was not to identify new areas of content in cultural competence (although in our study, religion emerged as an important dimension not specifically defined in prior frameworks42), but rather to understand in a more structured fashion the ways in which target groups conceptually framed and organized existing cultural competence content.
The organization and prioritization of information is critical for mental processing, learning, and problem solving, i.e., learning with understanding. The concept of learning with understanding is fully supported by a body of evidence summarized in the National Research Council’s seminal document Brain, Mind, Experience, and School.43 Learners gain a deeper understanding of a theme when they organize ideas, understand facts and ideas in the context of a conceptual framework, and organize knowledge in ways that facilitate retrieval and application.
An approach similar to ours can guide the process of content development, which includes the identification of learning needs as well as organization and prioritization of the content. Our study does not address which modality is best suited for curriculum delivery. However, the product of this formative assessment was, in fact, an online curriculum that our institution created entitled “Cultural Competence Online for Medical Practice (CCOMP)” (www.c-comp.org), which is available in MedEdPortal.44 Within the Web site, teaching points are organized according to the dimensions and neighborhoods we discovered through this process. Furthermore, the overall importance of the neighborhoods (expressed as the overall sums of ratings) also informed the Web design. The size of the visual display of the content of the Patient’s Background and the Provider/Health Care neighborhoods was twice as large as the Cross-Culture or Resources neighborhoods. Though we focused on the topic of cultural competence, a similar strategy may work well for others developing curriculum in other topic areas, especially in areas that are ill-defined. Similarly, cognitive mapping can allow the inclusion of perspectives of critical stakeholders such as patients, providers, or health care systems.
We noted significant differences among groups with regard to the ideas rated highest in importance, perhaps stemming from the group members’ own skills at their individual career stages. However, we recognize these differences as potential strengths for curriculum developers. These differences present an opportunity for tailoring content to particular learners or groups of learners based on their professional level. Their unique responses may also provide a foundation from which to review, refocus, or update materials, especially as culture shifts, people change, the health care system evolves, and resources become more or less available.
The use of the NGT in curriculum development is emerging in medical education; however, the use of cognitive mapping methods is a novel application in medical education research. Building from the body of knowledge from this study, our group has used cognitive mapping methods, multidimensional scaling and hierarchical cluster analysis as key processes in medical education.18,45 The attributes that contribute to successful ward-attending teaching rounds identified by Castiglioni et al 18 were organized using cognitive mapping methods from the faculty perspective 45 and from the medical students’ and residents’ perspectives 37 both published in abstract form. For example, Salanitro et al found that faculty grouped the attributes of successful rounds into six unique clusters, which were learning climate, explaining the clinical rationale, respect, managerial skills, and setting expectations.45 Roy et al reported that trainees grouped the same attributes into five unique clusters, which were role modeling, learning environment, teaching process, team management, and expectations.37 The results of those two studies are guiding an ongoing faculty development program. A review of cognitive mapping methods in the social sciences is available elsewhere.25,27
We acknowledge that there may be other, even more meaningful, interpretations of our cognitive map that we did not consider. Though we convened a multidisciplinary group, the sample was relatively small and may not be representative for all U.S. trainees or practicing physicians, and there were certainly perspectives that were not brought to the table or may have been excluded from our expansive discussions. However, our data show great diversity of ideas among groups and sufficient overlap to conclude that we reached some level of saturation. We felt that the final agreed-upon interpretation of the cognitive map best embodied the purpose and outcomes of our formative method.
Curriculum development is difficult, particularly in an area like cultural competence, where multiple frameworks exist. In this study, we set forth one approach for engaging learners, elucidating important topical domains, reaching some level of consensus, and organizing the information in a way that is both logical and helpful in curriculum development in cultural competence. Our approach illustrates one method, technique, or process that may be used by educators to identify content for initial development in any number of clinical or practice areas. It can also be used to revisit or reconsider that content over time for purposes of update or revision in light of new advances or improvements in care.
The authors thank Michael Schoen, PhD, Trudi Horton, PhD, Analia Castiglioni, MD, and Brita Roy, MD, MPH for critical revisions of the manuscript.
Funding/Support: This study was funded by a National Heart, Lung, and Blood Institute (K07 HL081373-01) grant to Dr. Estrada as part of the National Consortium for Multicultural Education for Health Professionals. The funding source did not make any restrictions on the scope or reporting of the study.
Other disclosures: None
Ethical approval: This study was approved by the institutional review boards of the University of Alabama at Birmingham.
Disclaimer: The opinions expressed in this article are those of the authors alone and do not reflect the views of the Department of Veterans Affairs.
Previous presentations: The abstract of an earlier version of this article was presented at the 2007 Southern Society of General Internal Medicine, New Orleans, Louisiana.
Ms. Katie Crenshaw, University compliance coordinator and was assistant director, Division of Continuing Medical Education, University of Alabama at Birmingham, at the time the study was conducted.
Dr. Richard M. Shewchuk, Professor of health services administration, School of Health Related Professions, University of Alabama at Birmingham, Birmingham, Alabama.
Dr. Haiyan Qu, Research assistant professor of health services administration, School of Health Related Professions, University of Alabama at Birmingham, Birmingham, Alabama.
Dr. Lisa J. Staton, Program director and outpatient clinic director, Department of Internal Medicine, and associate professor, University of Tennessee College of Medicine Chattanooga, Chattanooga, Tennessee.
Dr. Judy Ann Bigby, Secretary of health and human services, Executive Office of Health and Human Services, Commonwealth of Massachusetts, Boston, Massachusetts.
Dr. Thomas K. Houston, Center for Health Quality, Outcomes and Economic Research, Bedford VAMC, Bedford, Massachusetts; professor of Quantitative Health Sciences and medicine chief, Division of Health Informatics and Implementation Science; and assistant dean for continuing medical education/medical education research, University of Massachusetts Medical School, Worcester, Massachusetts.
Dr. Jeroan Allison, Department of Quantitative Health Sciences, associate vice provost for health disparities, and professor of quantitative health sciences, University of Massachusetts Medical School, Worcester, Massachusetts.
Dr. Carlos A. Estrada, Veterans Affairs National Quality Scholars Program; scientist, Deep South Center on Effectiveness, REAP Center for Surgical, Medical Acute Care Research and Transitions, Birmingham Veterans Affairs Medical Center, and professor of medicine and director, Division of General Internal Medicine, University of Alabama at Birmingham, Birmingham, Alabama.