Search tips
Search criteria 


Logo of nihpaAbout Author manuscriptsSubmit a manuscriptHHS Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
Acad Med. Author manuscript; available in PMC 2012 March 1.
Published in final edited form as:
PMCID: PMC3046368

What Should We Include in a Cultural Competence Curriculum? An Emerging Formative Evaluation Process to Foster Curriculum Development

Ms. Katie Crenshaw, JD, MSEd, Dr. Richard M. Shewchuk, PhD, Dr. Haiyan Qu, PhD, Dr. Lisa J. Staton, MD, Dr. Judy Ann Bigby, MD, Dr. Thomas K. Houston, MD, MPH, Scientist, Dr. Jeroan Allison, MD, MSc, Vice chair, and Dr. Carlos A. Estrada, MD, MS, Director



To identify, prioritize, and organize components of a cultural competence curriculum to address disparities in cardiovascular disease.


In 2006, four separate nominal group technique sessions were conducted with medical students, residents, community physicians, and academic physicians to generate and prioritize a list of concepts (i.e., ideas) to include in a curriculum. Afterward, 45 educators and researchers organized and prioritized the concepts using a card-sorting exercise. Multidimensional scaling (MDS) and hierarchical cluster analysis produced homogeneous groupings of related concepts and generated a cognitive map. The main outcome measures were the number of cultural competence concepts, their relative ranks, and the cognitive map.


Thirty participants generated 61 concepts, 29 were identified by at least 2 participants. The cognitive map organized concepts into four clusters, interpreted as: (1) patient’s cultural background (e.g.,, information on cultures, habits, values); (2) provider and health care (e.g., clinical skills, awareness of one’s bias, patient-centeredness, and professionalism), communication skills (e.g., history, stereotype avoidance, and health disparities epidemiology); (3) cross-culture (e.g., idiomatic expressions, examples of effective communication); and (4) resources to manage cultural diversity (e.g., translator guides, instructions and community resources). The MDS two-dimensional solution demonstrated a good fit (stress=0.07; R2=0.97).


A novel, combined approach allowed stakeholders’ inputs to identify and cognitively organize critical domains used to guide development of a cultural competence curriculum. Educators may use this approach to develop and organize educational content for their target audiences, especially in ill-defined areas like cultural competence.

The Institute of Medicine has suggested that a culturally-sensitive approach to care that avoids bias, prejudice, and stereotyping may help reduce health care disparities.1 Multiple frameworks have been proposed to improve cultural competence among health care providers.210 Cultural competence training has covered many different topics, has taken any number of different shapes, and has been shown to improve provider knowledge, attitudes, and skills, although patient outcomes are not frequently assessed.11,12 However, no clear consensus is available for the organization of content that ought to be included in a cultural competence curriculum for physicians or physicians-in-training.

Processes more frequently employed in the social sciences have been useful in gathering and organizing data that can shape educational products. For instance, nominal group technique (NGT) 13,14 offers advantages over traditional focus group discussions by allowing equal participation, avoiding participant’s inhibition in informal group discussion, avoiding focusing on one particular area, reducing apprehension from public speaking, generating a greater number of ideas, and minimizing inefficiencies and process loss.1517 NGT has been used in medical education to assess trainees’ perceptions of successful teaching rounds,18 to evaluate curricula,19 and to identify key attributes of an ideal continuing medical education (CME) institution to create a roadmap for excellence,20 among others. In medical practice, NGT has been used to identify intervention strategies to improve osteoporosis treatment in nursing homes, 21 identify reasons for lack of medication adjustment in patients with seemingly uncontrolled hypertension, 22 to identify barriers for diabetes self-management, 17 and to achieve consensus among stakeholders in guidelines development.23 Importantly, NGT does not limit participants to a pre-determined framework; rather, it forces participants to consider any and all options.

In addition, grouping methods allow the visual representation of mental processes, such as the data gathered during NGT, for instruction. The goal is to reveal the underlying cognitive dimensions, or “cognitive map,” that describes the spatial representation of how the items generated during NGT are related to each other and how ideas are prioritized within each cluster.2427 Card-sorting, hierarchical cluster analysis, and multidimensional scaling have been used to organize information and understand group similarities,2832 to understand consumer behavior,33 to design the contextual navigation of Web sites,28,34,35 to provide insight on how journals are cognitively organized,26 and to understand motivation in schools.36 However, to our knowledge, such methods have not been used in medical education research.18,37

We set out to inform the development and organization of a cultural competence curriculum that would increase health care professionals’ knowledge of health disparities and fundamental issues of cultural competence as a step in enhancing cardiovascular care for minority groups. We did so by engaging groups of relevant stakeholders (including cultural competence experts, physicians, and physicians-in-training) in a robust, formative assessment to identify, prioritize, and organize the critical components of cultural competence.22 The combined approach we describe is a novel application in medical education research; the qualitative and quantitative approach is specifically designed to guide the process of content development and organization of information.


We report below the development of the Cultural Competence Curriculum program, an Internet-based educational intervention for physicians and physicians in training aimed at reducing health disparities in cardiovascular health.

Information gathering: NGT

First, we used NGT to generate and prioritize a list of concepts deemed important to include in the curriculum. As explained earlier, NGT is a highly structured, small-group-meeting approach used to elicit and organize answers to a specific question.18 NGT involves the following steps: (1) silent, written generation of responses to a specific question; (2) round-robin recording of ideas; (3) discussion for clarification only; and (4) voting on the rank order of the top three ideas, usually with respect to importance, feasibility, or other characteristics. (These steps are described in much greater detail elsewhere 18). Figure 1 is a graphical description of the question development, the NGT groups, and how the responses were processed to the question that was developed.

Figure 1
A graphical representation of the question development, the nominal group technique groups, and how the responses to the question that was developed were processed. The study goal was to identify, prioritize, and organize components of a cultural competence ...

Question development

To formulate the central question for our NGT sessions, we convened a panel of clinician-educators, nurses, minority community physicians, minority health researchers, health services researchers, social scientists, and distance-learning experts. The panel reviewed the literature and composed and pilot-tested the question among medical students, faculty members, and a community physician.

NGT sessions

Between February and March 2006, we conducted four separate NGT sessions with medical students (n = 9), internal medicine residents (n = 7), primary care community physicians (n = 7), and disparities or educational researchers (n = 7). The small sample size was by design and is justified, given NGT methodology where the process allows for idea saturation quickly with smaller numbers of participants, provided the demographic composition of the groups broadly represents the target end-user. We tried to have participants who represented a mixture of age, gender, and ethnicity; however, by design, we did not ask or collect information on such characteristics, since such inquiries could have influenced the responses during the NGT sessions.

We recruited two convenience samples of fourth-year medical students and internal medicine house officers from The University of Alabama at Birmingham, Birmingham, Alabama. We recruited community physicians from the Alabama Practice-Based CME Network, a group in regular communication with the academic medical center, for purposes of exploring educational need in the community and easy delivery of educational materials and activities addressing those needs. Participants were recruited via fax and awarded a $100 honorarium. Finally, we recruited investigators funded by the National Heart, Lung, and Blood Institute.38 NGT participants were informed that the purpose of the session was to generate their ideas concerning curricular content, not format, for a cultural competence program focusing on cardiovascular disease risk and treatment; the program would be available to medical students, residents in training, and practicing physicians.

The final question formulated by the panel and posed to NGT participants was “What sorts of things could be included in a curriculum that focuses on cultural competence training for physicians?” Once all responses to the question had been given, each group was asked to identify the three most important ideas and the three most feasible ideas that the responses contained. Experienced facilitators conducted the sessions.

Organizing the ideas: card sort

The ideas from the NGT sessions that received at least 2 votes as “most important” or “most feasible” were then organized by a group consisting of a convenience sample of 45 educators and disparities researchers. These participants physically grouped and ranked the ideas, which were written on index cards. These educators and researchers hailed from 19 cities and 14 states in the United States. There were 29 general internal medicine physicians, one pediatrician, three specialists, four nurses, four PhDs, and four from other professions. Of these participants, 26 were women, 19 were men; there were 24 Caucasians, 8 African Americans, 6 Hispanics, 3 Asians, 3 unknown, and 1 “other.”

Each participant worked independently for 10–20 minutes to

  • examine the cards together as a deck,
  • sort them into piles based on their own perceptions or criteria for how they should be grouped,
  • create at least two but no more than 10 piles,
  • place cards that were not in some way related to each other into a pile labeled “cards cannot be grouped,”
  • note whether two or more ideas reflected the same concept, and
  • make no attempt of assigning a label to each pile.

Last, within each pile, participants ranked the ideas, again based on their own criteria, by placing the most important idea at the top of the pile and the least important idea at the bottom of the pile. We did not provide an incentive or honoraria.

Information analysis: cognitive mapping

To analyze the results of the card sort, we used multidimensional scaling (MDS) and hierarchical cluster analysis (HCA).22 The goal of MDS is to reveal the psychological dimensions in the data that can meaningfully describe the underlying cognitive constructs. The result, a “cognitive map,” is a spatial representation of how the ideas are considered to be similar to or different from each other.26,39 The coordinates defining each item’s position from the MDS analysis are used as input data in the cluster analysis. The goal of HCA is to facilitate the interpretation of the MDS map by dividing a set of objects into a smaller number of homogeneous groups on the basis of their similarity.40 Enclosed within lines, the smaller homogeneous groups represent “families,” and clusters of groups represent “neighborhoods.” 41 The map in our study reflected how the card sorters cognitively organized the areas to include in the cultural competency training curriculum for physicians.

The data from the card-sorting task were first used to create a matrix for each participant. This co-occurrence matrix indicated the number of times each item was sorted with all other items (29×29 matrix). The individual co-occurrence matrices were then aggregated across all participants to form a group co-occurrence matrix. This matrix indicated the number of times all participants sorted each reason into similar or dissimilar piles. The group co-occurrence matrix was then analyzed with MDS (using the ALSCAL algorithm) and with cluster analysis to derive the cognitive map. This iterative process derived an optimal geometric solution that reflects the proximity of the data and the relative ordering from the distances observed in the matrix.24

The number of dimensions in an MDS solution is determined by its interpretability and analysis. Perhaps the most important is the interpretability. A two-dimension solution is usually recommended because additional dimensions make interpretation more difficult.41 MDS analysis results in the R-square (RSQ) and stress statistics. The RSQ statistic reflects the extent to which the map corresponds to the actual proximity data, and values of RSQ >.90 suggest high correspondence. The stress statistic reflects the “badness of fit,” with acceptable stress values being >.01 and <.20 (<.01 indicates a degenerate solution, >.20 indicates poor fit). 40,41

Although MDS and cluster analysis are computationally sophisticated, they are not based on parametric statistics. Consequently, their validity is not sample-size dependent, but dependent on the representativeness of participants in the sorting task. While the location of items on the map is analytically determined, the meaning of the dimensions (i.e., the two axis labels) must be interpreted. We used a standard approach to interpret the derived dimensions by engaging the entire research team and by examining the substantive differences between items at the extremes of the dimensions.

The Institutional Review Board at the University of Alabama at Birmingham approved this study.


The results of the nominal group technique (NGT) and the cognitive mapping are shown in Table 1 and Figure 2.

Figure 2
Representation of ideas, “families,” and “neighborhoods” to include in a cultural competence curriculum. The x-axis is a dimension of patient centeredness (Patient Centrism) with a neighborhood of issues more intrinsic ...
Table 1
Organization of 29 Ideas Generated by Participants in Four Independent Nominal Group Technique (NGT) Sessions *,

Nominal group technique

During the 4 independent NGT sessions, a total of 61 ideas were generated. Twenty-nine of these ideas were selected by at least 2 participants as most important or feasible; they are all listed in Table 1. The other 32 ideas were discarded and not incorporated into the resulting Web-based program’s content (a list of these discarded ideas is presented in Supplemental Digital Table 1, which may be found at http://----------------------------). [INSERT LINK HERE] As shown in Table 1, the top-rated idea among experts was to “include content to help learners become aware of [their] own biases and attitudes…” (item 17, sum of the ratings = 9); among community physicians the two top-rated ideas were “describe habits that are unique…” (item 13, sum of the ratings = 7), and “present objective data on the cultures …” (item 12, sum of the ratings = 7); among residents it was to “provide access to resources…” (item 6, sum of the ratings = 6); and, among medical students it was to “identify critical questions that would permit taking cultural history …” (item 1, sum of the ratings = 17). Table 1 shows the organization of the 29 ideas selected by at least 2 participants as most important or feasible, classified by family (i.e., small group).

Organization and analysis of ideas: the cognitive map

The 45 clinician-educators and health disparities researchers organized the 29 ideas in a median of 3 piles (range 1–11); each pile had a median of 6 ideas (range 3–10).

The multidimensional scaling (MDS) and hierarchical cluster analysis revealed two possible solutions, on two or three dimensions. Although the three-dimension solution better explained the data than did the two-dimension solution (higher RSQ, 0.97 vs. 0.91; lower stress measure, 0.07 vs. 0.15), we present here the two-dimension solution, as it was the most parsimonious and interpretable.

The cluster analysis revealed ten small groups, or families, of cognitively most-similar items, each within four larger groups, or neighborhoods, of cognitive similarity. The decision to interpret a ten-cluster solution was informed by examining the pattern of the agglomeration coefficients and by visually inspecting the two-dimension plot and the dendrogram to determine the sequence of cluster formation. The cognitive map resulting from the MDS and cluster analysis is shown in Figure 2.

We propose that the four larger groups can be viewed as neighborhoods of related themes: (1) patient’s cultural background (i.e., information on cultures, habits, values); (2) provider and health care (e.g., clinical skills, awareness of one’s bias, patient-centeredness, and professionalism), communication skills (e.g., history, stereotype avoidance, and health disparities epidemiology); (3) cross-culture (e.g., idiomatic expressions, examples of effective communication); and (4) resources to manage cultural diversity. These neighborhoods are shown in Table 1 and Figure 2, which give a fuller picture of them than could be presented in this paragraph. We base our interpretation of the mapping results on the substantive orientation and thematic consistency of the ideas within these clusters of families and among neighborhoods.

The patient’s background neighborhood includes three distinct families: cultural background (e.g., religion, customs, moral values), culture and health (e.g., impact of religion and culture behavior and adherence to cultural standards), and social background (e.g., health patterns, views on complementary and alternative medicine, region of the country).

The “provider and health care neighborhood includes three families: clinical skills (e.g., awareness of one’s bias, patient-centered, professionalism), communication skills (e.g., history, stereotype avoidance), and health disparities epidemiology (e.g., prevalence, causes, medications).

The cross-culture neighborhood includes two families: cross-cultural communication (e.g., idiomatic expressions, barriers, effective communication examples) and cross-cultural resources.

The “resources to manage cultural diversity neighborhood consists of two families: translator guides/instructions and community resources.

In Figure 2, we conceptualize the x-axis as a dimension of patient centeredness (Patient Centrism) with a neighborhood of issues more intrinsic to the patient (Patient’s Background, on the left side of the figure) and a neighborhood of issues more extrinsic to the patient (Resources, on the right side). Similarly, we conceptualize the y-axis as a dimension of health-care centeredness (Provider and Health- System Centrism) with a neighborhood of issues more intrinsic to health care (Provider/Health Care, on the upper portion of the figure) and a neighborhood of issues more extrinsic to health care (Resources and Cross-Culture, on the lower portion of the figure).

We arrived at these interpretations by examining the ideas at the extremes on each axis to determine similarities and differences. For instance, with respect to the dimension of patient centeredness (the Patient Centrism axis), differences between the two items farthest apart were relatively obvious. Item 8 (“Provide a list of available translator services”) is not under the control of the patient or would be the same across all patients. Item 15 (“Provide information on religious principles of particular cultures and how these might influence medical care”) is very individualized to the patient and not the same from patient to patient. However, this approach was more difficult in the case of the dimension of health care centeredness (the Provider/Health-System-Centrism axis), where the variance between extremes was not as great. Consequently, we had to examine other items near the extremes to find meaning. Ultimately, we concluded that the items near the extremes on this axis also related to the level of control of the physician or health care system.

Importantly, the Patient’s Background and the Provider/Health Care neighborhoods seem to be the most important, as illustrated by the higher overall sums of ratings of 56 and 55, respectively, compared with the Cross-Culture or Resources neighborhoods, with sums of ratings of 25 and 23, respectively.


As explained earlier, in a formative evaluation of concepts important and feasible for inclusion in a cultural competence curriculum, medical students, residents, community physicians, and academic physicians identified and organized 29 ideas along the two axes of patient centeredness and provider/health-care-system centeredness (illustrated in Figure 2). The resulting cognitive map was interpreted to include four neighborhoods of concepts: patient’s background, provider/health care, cross-culture, and resources to manage cultural diversity; each of them contained a family of ideas related to the neighborhood’s concept. We extended the methods and analytical approach from the social sciences to effectively outline a curriculum for cultural competence training. Our goal with this process was not to identify new areas of content in cultural competence (although in our study, religion emerged as an important dimension not specifically defined in prior frameworks42), but rather to understand in a more structured fashion the ways in which target groups conceptually framed and organized existing cultural competence content.

Implications for curriculum development and organization

The organization and prioritization of information is critical for mental processing, learning, and problem solving, i.e., learning with understanding. The concept of learning with understanding is fully supported by a body of evidence summarized in the National Research Council’s seminal document Brain, Mind, Experience, and School.43 Learners gain a deeper understanding of a theme when they organize ideas, understand facts and ideas in the context of a conceptual framework, and organize knowledge in ways that facilitate retrieval and application.

An approach similar to ours can guide the process of content development, which includes the identification of learning needs as well as organization and prioritization of the content. Our study does not address which modality is best suited for curriculum delivery. However, the product of this formative assessment was, in fact, an online curriculum that our institution created entitled “Cultural Competence Online for Medical Practice (CCOMP)” (, which is available in MedEdPortal.44 Within the Web site, teaching points are organized according to the dimensions and neighborhoods we discovered through this process. Furthermore, the overall importance of the neighborhoods (expressed as the overall sums of ratings) also informed the Web design. The size of the visual display of the content of the Patient’s Background and the Provider/Health Care neighborhoods was twice as large as the Cross-Culture or Resources neighborhoods. Though we focused on the topic of cultural competence, a similar strategy may work well for others developing curriculum in other topic areas, especially in areas that are ill-defined. Similarly, cognitive mapping can allow the inclusion of perspectives of critical stakeholders such as patients, providers, or health care systems.

We noted significant differences among groups with regard to the ideas rated highest in importance, perhaps stemming from the group members’ own skills at their individual career stages. However, we recognize these differences as potential strengths for curriculum developers. These differences present an opportunity for tailoring content to particular learners or groups of learners based on their professional level. Their unique responses may also provide a foundation from which to review, refocus, or update materials, especially as culture shifts, people change, the health care system evolves, and resources become more or less available.

Uses of the knowledge gained in this study

The use of the NGT in curriculum development is emerging in medical education; however, the use of cognitive mapping methods is a novel application in medical education research. Building from the body of knowledge from this study, our group has used cognitive mapping methods, multidimensional scaling and hierarchical cluster analysis as key processes in medical education.18,45 The attributes that contribute to successful ward-attending teaching rounds identified by Castiglioni et al 18 were organized using cognitive mapping methods from the faculty perspective 45 and from the medical students’ and residents’ perspectives 37 both published in abstract form. For example, Salanitro et al found that faculty grouped the attributes of successful rounds into six unique clusters, which were learning climate, explaining the clinical rationale, respect, managerial skills, and setting expectations.45 Roy et al reported that trainees grouped the same attributes into five unique clusters, which were role modeling, learning environment, teaching process, team management, and expectations.37 The results of those two studies are guiding an ongoing faculty development program. A review of cognitive mapping methods in the social sciences is available elsewhere.25,27


We acknowledge that there may be other, even more meaningful, interpretations of our cognitive map that we did not consider. Though we convened a multidisciplinary group, the sample was relatively small and may not be representative for all U.S. trainees or practicing physicians, and there were certainly perspectives that were not brought to the table or may have been excluded from our expansive discussions. However, our data show great diversity of ideas among groups and sufficient overlap to conclude that we reached some level of saturation. We felt that the final agreed-upon interpretation of the cognitive map best embodied the purpose and outcomes of our formative method.


Curriculum development is difficult, particularly in an area like cultural competence, where multiple frameworks exist. In this study, we set forth one approach for engaging learners, elucidating important topical domains, reaching some level of consensus, and organizing the information in a way that is both logical and helpful in curriculum development in cultural competence. Our approach illustrates one method, technique, or process that may be used by educators to identify content for initial development in any number of clinical or practice areas. It can also be used to revisit or reconsider that content over time for purposes of update or revision in light of new advances or improvements in care.

Supplementary Material


The authors thank Michael Schoen, PhD, Trudi Horton, PhD, Analia Castiglioni, MD, and Brita Roy, MD, MPH for critical revisions of the manuscript.

Funding/Support: This study was funded by a National Heart, Lung, and Blood Institute (K07 HL081373-01) grant to Dr. Estrada as part of the National Consortium for Multicultural Education for Health Professionals. The funding source did not make any restrictions on the scope or reporting of the study.


Other disclosures: None

Ethical approval: This study was approved by the institutional review boards of the University of Alabama at Birmingham.

Disclaimer: The opinions expressed in this article are those of the authors alone and do not reflect the views of the Department of Veterans Affairs.

Previous presentations: The abstract of an earlier version of this article was presented at the 2007 Southern Society of General Internal Medicine, New Orleans, Louisiana.

Contributor Information

Ms. Katie Crenshaw, University compliance coordinator and was assistant director, Division of Continuing Medical Education, University of Alabama at Birmingham, at the time the study was conducted.

Dr. Richard M. Shewchuk, Professor of health services administration, School of Health Related Professions, University of Alabama at Birmingham, Birmingham, Alabama.

Dr. Haiyan Qu, Research assistant professor of health services administration, School of Health Related Professions, University of Alabama at Birmingham, Birmingham, Alabama.

Dr. Lisa J. Staton, Program director and outpatient clinic director, Department of Internal Medicine, and associate professor, University of Tennessee College of Medicine Chattanooga, Chattanooga, Tennessee.

Dr. Judy Ann Bigby, Secretary of health and human services, Executive Office of Health and Human Services, Commonwealth of Massachusetts, Boston, Massachusetts.

Dr. Thomas K. Houston, Center for Health Quality, Outcomes and Economic Research, Bedford VAMC, Bedford, Massachusetts; professor of Quantitative Health Sciences and medicine chief, Division of Health Informatics and Implementation Science; and assistant dean for continuing medical education/medical education research, University of Massachusetts Medical School, Worcester, Massachusetts.

Dr. Jeroan Allison, Department of Quantitative Health Sciences, associate vice provost for health disparities, and professor of quantitative health sciences, University of Massachusetts Medical School, Worcester, Massachusetts.

Dr. Carlos A. Estrada, Veterans Affairs National Quality Scholars Program; scientist, Deep South Center on Effectiveness, REAP Center for Surgical, Medical Acute Care Research and Transitions, Birmingham Veterans Affairs Medical Center, and professor of medicine and director, Division of General Internal Medicine, University of Alabama at Birmingham, Birmingham, Alabama.


1. Unequal Treatment: Confronting Racial and Ethnic Disparities in Health Care. Institute of Medicine, National Academy of Sciences; Washington, D.C: 2002.
2. Lie D, Boker J, Cleveland E. Using the tool for assessing cultural competence training (TACCT) to measure faculty and medical student perceptions of cultural competence instruction in the first three years of the curriculum. Acad Med. 2006. [Accessed November 19, 2010]. pp. 557–564 . [PubMed]
3. Betancourt JR, Green AR. Commentary: linking cultural competence training to improved health outcomes: perspectives from the field. Acad Med. 2010. [Accessed November 19, 2010]. pp. 583–585. [PubMed]
4. Smith WR, Betancourt JR, Wynia MK, et al. Recommendations for teaching about racial and ethnic disparities in health and health care. Ann Intern Med. 2007;147(9):654–665. [PubMed]
5. Betancourt JR. Cross-cultural Medical Education: Conceptual Approaches and Frameworks for Evaluation. Acad Med. 2003. [Accessed November 19, 2010]. pp. 560–569. [PubMed]
6. Betancourt JR. Cultural competence and medical education: many names, many perspectives, one goal. Acad Med. 2006. [Accessed November 19, 2010]. pp. 499–501. [PubMed]
7. Weech-Maldonado R, Dreachslin JL, Dansky KH, De Souza G, Gatto M. Racial/ethnic diversity management and cultural competency: the case of Pennsylvania hospitals. J Healthc Manag. 2002;47(2):111–124. discussion 124–116. [PubMed]
8. Teal CR, Street RL. Critical elements of culturally competent communication in the medical encounter: a review and model. Soc Sci Med. 2009;68(3):533–543. [PubMed]
9. Saha S, Beach MC, Cooper LA. Patient centeredness, cultural competence and healthcare quality. J Natl Med Assoc. 2008;100(11):1275–1285. [PMC free article] [PubMed]
10. The Office of Minority Health. National Standards on Culturally and Linguistically Appropriate Services (CLAS) [Accessed November 13, 2010]. Available at:
11. Kutob RM, Senf JH, Harris JM., Jr Teaching culturally effective diabetes care: results of a randomized controlled trial. Fam Med. 2009;41(3):167–174. [PubMed]
12. Beach MC, Cooper LA, Robinson KA, et al. Strategies for improving minority healthcare quality. Evid Rep Technol Assess (Summ) 2004;(90):1–8. [PubMed]
13. Brief 7. Gaining Consensus Among Stakeholders Through the Nominal Group Technique. [Accessed November 13, 2010. 2006]. Available at
14. Paulus PB, Huei-Chuan Y. Idea Generation in Groups: A Basis for Creativity in Organizations. Organizational Behavior & Human Decision Processes. 2000;82(1):76–87.
15. Van de Ven AH, Delbecq AL. The nominal group as a research instrument for exploratory health studies. Am J Public Health. 1972;62(3):337–342. [PubMed]
16. Gallagher M, Hares T, Spencer J, Bradshaw C, Webb I. The nominal group technique: a research tool for general practice? Fam Pract. 1993;10(1):76–81. [PubMed]
17. Miller D, Shewchuk R, Elliot TR, Richards S. Nominal group technique: a process for identifying diabetes self-care issues among patients and caregivers. Diabetes Educ. 2000;26(2):305–310. 312, 314. [PubMed]
18. Castiglioni A, Shewchuk RM, Willett LL, Heudebert GR, Centor RM. A pilot study using nominal group technique to assess residents’ perceptions of successful attending rounds. J Gen Intern Med. 2008;23(7):1060–1065. [PMC free article] [PubMed]
19. Mihalic AP, Morrow JB, Long RB, Dobbie AE. A validated cultural competence curriculum for US pediatric clerkships. Patient Educ Couns. 2010;79(1):77–82. [PubMed]
20. Kristofco R, Shewchuk R, Casebeer L, Bellande B, Bennett N. Attributes of an ideal continuing medical education institution identified through nominal group technique. J Contin Educ Health Prof. 2005;25(3):221–228. [PubMed]
21. Levine DA, Saag KG, Casebeer LL, Colon-Emeric C, Lyles KW, Shewchuk RM. Using a modified nominal group technique to elicit director of nursing input for an osteoporosis intervention. J Am Med Dir Assoc. 2006;7(7):420–425. [PMC free article] [PubMed]
22. Safford MM, Shewchuk R, Qu H, et al. Reasons for not intensifying medications: differentiating “clinical inertia” from appropriate care. J Gen Intern Med. 2007;22(12):1648–1655. [PMC free article] [PubMed]
23. Raine R, Sanderson C, Hutchings A, Carter S, Larkin K, Black N. An experimental study of determinants of group judgments in clinical guideline development. Lancet. 2004 Jul 31–Aug 6;364(9432):429–437. [PubMed]
24. Schiffman SS, Reynolds ML, Young FW. Introduction to Multidimensional Scaling. Academic Press Inc; 1981.
25. Whaley AL, Longoria RA. Preparing card sort data for multidimensional scaling analysis in social psychological research: a methodological approach. J Soc Psychol. 2009;149(1):105–115. [PubMed]
26. Shewchuk RM, O’Connor SJ, Williams ES, Savage GT. Beyond rankings: using cognitive mapping to understand what health care journals represent. Soc Sci Med. 2006;62(5):1192–1204. [PubMed]
27. Mugavin ME. Multidimensional scaling: a brief overview. Nurs Res. 2008;57(1):64–68. [PubMed]
28. Pretlow C. Computers in Libraries. Vol. 28. Information Today Inc; 2008. 10 Web Tools to Create User-Friendly Sites; pp. 14–17.
29. Tobar LM, Andres PML, Lapena EL. WebA: A tool for the assistance in design and evaluation of websites. J Univers Comput Sci. 2008;14(9):1496–1512.
30. Villa J. Time for a Marketing Facelift? Mark Health Serv. 2009;29(3):30–31. [PubMed]
31. Giguere G. Collecting and analyzing data in multidimensional scaling experiments: A guide for psychologists using SPSS. Tutor Quant Meth Psychol. 2006;2(1):27–38.
32. Frisby CL. The Use of Multidimensional Scaling in the Cognitive Mapping of Cultural Difference Judgments. School Psych Rev. 1996;25(1):77–93.
33. Desarbo WS, Kim J, Choi SC, Spaulding M. A Gravity-Based Multidimensional Scaling Model for Deriving Spatial Structures Underlying Consumer Preference/Choice Judgments. J Consum Res. 2002;29(1):91–100.
34. Bussolon S. Card sorting, category validity, and contextual navigation. J Inform Architecture. 2009;1(2):5–29.
35. Harper S, Michailidou E, Stevens R. Toward a Definition of Visual Complexity as an Implicit Measure of Cognitive Load. ACM Trans Appl Percept. [Accessed November 13, 2010]. 2009. Available at:
36. McInerney DM, Ali J. Multidimensional and Hierarchical Assessment of School Motivation: Cross-Cultural Validation. Educ Psychol. 2006;26(6):717–734.
37. Roy B, Salanitro AH, Willett LL, et al. Using Cognitive Mapping to Identify Attributes Contributing to Successful Ward-Attending Rounds - a Resident and Student Perspective. J Gen Intern Med. 2010;25(S3):S424–425.
38. Carter-Pokras O, Bereknyei S, Lie D, Braddock CH. For the National Consortium for Multicultural Education for Health Professionals. Surmounting the unique challenges in health disparities education: a multi-institution qualitative study. J Gen Intern Med. 2010;25 (S2):108–114. [PMC free article] [PubMed]
39. Shewchuk R, O’Connor SJ. Using Cognitive Concept Mapping to Understand What Health Care Means to the Elderly: An Illustrative Approach for Planning and Marketing. Health Marketing Quarterly. 2002;20(2):69–88. [PubMed]
40. Aldenderfer MS, Blashfield RK. Cluster Analysis. Newbury Park, CA: Sage University Press; 1984.
41. Kruskal JB, Wish M. Multidimensional Scaling. Newbury Park, CA: Sage University Press; 1978.
42. AAMC. Association of American Medical Colleges. Tool for Assessing Cultural Competence Training (TACCT) [Accessed November 13, 2010]. Available at:
43. Bransford JD, Brown AL, Cocking RR. Committee on Developments in the Science of Learning and Committee on Learning Research and Educational Practice, Commission on Behavioral and Social Sciences and Education, National Research Council. Washington, D.C: National Academy Press; 2004. How people learn : Brain, Mind, Experience, and School. Expanded.
44. AAMC. Association of American Medical Colleges. MedEdPortal. [Accessed November 13, 2010]. Available at
45. Salanitro AH, Castiglioni A, Willett LL, et al. Using cognitive mapping to Identify faculty perspectives for successful ward-attending rounds. J Invest Med. 2009;57(1):382–383.