|Home | About | Journals | Submit | Contact Us | Français|
Language: | French
Historically, academic success has been a major outcome for evaluating the effectiveness of pharmacy education programs and admission criteria. In other words, students’ overall grades and/or specific course grades have determined academic success. However, there is a disconnection between students’ grades and their performance during practicums or in practice. It was postulated that professionalism might be an alternative outcome for measuring graduates’ abilities.
To construct an objective measure of professional attitudes and behaviours for recently graduated pharmacists.
A self-report instrument was developed using the American Board of Internal Medicine’s 6-tenet definition of professionalism. Four months after completing the doctor of pharmacy degree (PharmD), pharmacists were asked to complete an online version of the professionalism instrument. The Rasch Measurement Model (Winsteps, Chicago, Illinois) was used to construct a measure from the responses. Using data that fit the model, the Rasch Measurement Model can build the interval-level measurements needed for future inferential statistical interpretations of ordinal-level data gathered with this instrument.
Twenty-seven PharmD graduates completed the 15-item instrument. The Rasch Measurement Model was used to construct continuous, linear measures of pharmacist professionalism from these instrument rating scale data. Most of the scales functioned without modification, but 2 of the scales functioned only after being collapsed. One person and one item “misfit” the Rasch Measurement Model and were omitted from the analysis. After these adjustments, the data fit the model well. As a measurement tool, this instrument was overwhelmingly unidimensional; the linear model explained 99.9% of the variation in the data using principal contrast analysis. Item separation was 3.64 logits, person separation was 2.31 logits, and reliability was 0.93 by Cronbach’s α.
The Rasch Measurement Model was used to construct an objective measure of pharmacists’ professionalism. The results of this pilot project suggest a promising outcome measure for evaluating pharmacy graduates soon after completion of university.
Historiquement, le rendement scolaire était le principal résultat pour évaluer l’efficacité des programmes d’enseignement de la pharmacie et des critères d’admission. En d’autres mots, les notes globales des étudiants et (ou) leurs notes pour un cours particulier déterminaient le rendement scolaire. Cependant, il y a un écart entre les notes des étudiants et leur performance durant les stages ou l’exercice de leur profession. On a émis l’hypothèse selon laquelle le professionnalisme pourrait être un résultat de rechange pour mesurer les compétences des diplômés.
Créer une mesure objective de l’attitude et des comportements professionnels des pharmaciens récemment diplômés.
Un outil d’auto-évaluation a été développé à l’aide de la définition du professionnalisme selon les six principes de l’American Board of Internal Medicine. Quatre mois après avoir obtenu leur doctorat professionnel en pharmacie (Pharm. D.), les pharmaciens devaient répondre aux questions de la version en ligne de l’outil d’évaluation du professionnalisme. Le modèle de Rasch (Winsteps, Chicago, Illinois) a servi à concevoir une mesure à partir des réponses. En utilisant des données adaptées au modèle, le modèle de Rasch peut générer les mesures dotées d’une échelle d’intervalle nécessaires pour les futures interprétations statistiques inférentielles des données ordinales collectées à partir de cet instrument.
Au total, 27 détenteurs d’un diplôme de Pharm. D. ont rempli le questionnaire à 15 items. Le modèle de Rasch a été utilisé pour générer des mesures linéaires continues du professionnalisme des pharmaciens en se servant des données issues des échelles de notation de l’outil. La plupart des échelles ont produit des résultats appropriés, mais deux ont donné des résultats appropriés seulement une fois groupées. Les données issues d’une personne et d’un item ne concordaient pas avec le modèle de Rasch et ont été exclues de l’analyse. À la suite de ces ajustements, les données cadraient bien avec le modèle. Cet instrument, en tant qu’outil de mesure, était extraordinairement unidimensionnel; le modèle linéaire expliquait 99,9 % de la variation des données selon l’analyse du contraste principal. Les valeurs suivantes ont été observées : répartition des items, 3,64 logits; répartition des personnes, 2,31 logits; et fiabilité (déterminée par le coefficient alpha de Cronbach), 0,93.
Le modèle de Rasch a été utilisé pour générer une mesure objective du professionnalisme des pharmaciens. Les résultats de ce projet pilote laissent croire à un avenir prometteur pour cet outil d’évaluation des pharmaciens nouvellement diplômés.
(Traduction par l’éditeur]
Measurement is a process that we use and take for granted daily. A ruler is an excellent example for elaborating the fundamentals of measurement. A ruler has defined units that are both continuous and additive, and it functions along one length (or dimension) at a time.
In contrast to ordinal scales, an interval-level measure requires equal units between sequential increases in the scale. In the early 20th century, a leading theoretician in modern psychometrics, Louis L Thurstone, extensively elaborated on this “unit of measure” and its requirements.1,2 These requirements were parameter linearity (such that a unidimensional instrument can be used for measuring), arbitrary units (with an arbitrary but equal measurement unit between consecutive instrument items), absolute scaling (i.e., scaling that is independent of both the people doing the measuring and the items being measured on the scale), and item fit (whereby items possess both rational/theoretic and empiric support for use). In the example suggested above, a ruler measures only one length at a time, which indicates parameter linearity. Whether the ruler is marked in centimetres or inches (i.e., units are arbitrary), all measurements will consistently have the same units, and those units will be equal in size, regardless of which instrument item is being measured (absolute scaling). Lastly, Thurstone’s item fit is illustrated by the ruler’s ability to measure length well, but weight poorly. All measurements, including measurements of abstract concepts such as behaviour, ability, or performance, must have the same characteristics as the ruler.
These properties were examined and highlighted for psychometric measurement over a century ago; however, psychological and nonphysical measurements have remained problematic.3 Until the mid-1960s, traditional measurement analyses in the social sciences remained deficient, ignoring Thurstone’s requirements. In 1964, Luce and Tukey addressed the concept of conjoint measurement,4 which mathematically supported the need for a continuous and equal-sized unit scale that would allow addition and subtraction. In the example of the ruler, conjoint measurement indicates that 2 measured lengths (using the same units, either centimetres or inches) can be added together. They are measurements and consequently are both continuous and additive. Measurement according to these principles has been termed objective or fundamental measurement.5
In 1960, Rasch6 devised a means of achieving conjoint measurement for nonphysical attributes by examining candidates’ knowledge using tests. However, he did not simply count the number of correct answers as though all of the questions were equally difficult. Instead, he mapped each question’s difficulty and each person’s ability to correctly answer questions along a scale that had a single dimension (similar to the example of the ruler). Importantly, the Rasch Measurement Model provides a theoretical framework to build scales that can measure. It uses the different probabilities of items of various difficulties and each person’s ability to complete those items. The model creates an interval scale from ordinal data by logarithmically transforming the probabilities onto a continuous scale. Many decades earlier, the father of modern statistical science, R A Fisher, had suggested using logarithms for this very purpose. Using the Rasch Measurement Model provides a means of objective measurement through the creation of a continuous measure of a nonphysical attribute on a unidimensional or linear scale. The requirements for conjoint measurement are therefore satisfied.
Just as psychological measurement quantifies nonphysical attributes, clinical medicine should assess the nonphysical attributes of clinicians. A thorough assessment of such attributes should use the previously stated properties of measurement. The physical sciences have advanced using objective measurements and applied mathematics. Length, density, and sound frequencies are but a few of the measurements that have advanced the physical sciences. Clinical medicine (and pharmacy) must not be a “soft science” but instead should maintain the same stringent requirements of measurement that are used by our colleagues within applications of the biological sciences.7 Measures of thoughts and actions of practising pharmacists can potentially be constructed into objective measures, after which these continuous measures can be used with parametric statistical tests. Using interval measures rather than ordinal data significantly improves the performance of the parametric statistical tests that require them.8
Within pharmacy education, investigations of students’ academic success have defined success primarily with grade-point averages attained throughout and upon completion of pharmacy education.9–12 This is probably because completion of a course results in a grade, an outcome that can be easily and predictably accessed. However, grades are not measures. Grades are determined from percentages, are often converted into letter grades or whole numbers, and (in the case of letters) may be permutated back into predetermined numbers before mathematical averaging (i.e., to determine a grade-point average). Once these changes have been made, grade-point averages cannot be interpreted as interval-level data, as required for measurement.13 Fundamental measurement is needed to accurately assess students’ success.
Although pharmacy may be a rigorous field of study, pharmacy faculty and preceptors suggest that increasing numbers of pharmacy students lack empathy, social maturity, and a commitment to professional standards during the experiential components of their studies.14,15 Professional attributes and behaviours may not be accurately reflected in course grades, which instead may predominantly reflect students’ capabilities to successfully complete traditional knowledge-based didactic courses. Professionalism has evolved to serve a significant and explicit role in pharmacy education and may thus provide an important outcome within professional education. Regardless of a pharmacy graduate’s practice setting after the initial pharmacy education program, professionalism should be common to all graduates. An objective measure of pharmacist professionalism is therefore needed. The purpose of this study was to develop and test a measure of pharmacists’ professionalism.
For this study, a professionalism instrument was developed (Figure 1), with instrument items based on the American Board of Internal Medicine’s tenets of professionalism, later adapted for use in pharmacy.16 This theoretical framework served as the anchor for content validity in building the instrument. Instrument items were developed for each tenet, denoting altruism, accountability, excellence, duty, honour and integrity, and respect for others. A previously constructed measure of “student study and learning habits” used both attitudinal and behavioural questions successfully.17 Therefore, this professionalism instrument also contained both attitudinal and behavioural items related to each professional tenet.
An advantage of the Rasch Measurement Model is the information that it provides about the manner in which respondents used an instrument’s rating scales. Respondents may use a rating scale similarly to or differently from the way in which the investigators intended.18 Understandably, the rating scale may not function properly if the responses do not follow the pattern of probabilities that the model infers. A Rasch Measurement Model analysis could illustrate whether a rating scale is functioning properly within the model.
Four months after graduation, an e-mail invitation was sent to pharmacist alumni of the University of Toledo (using the institution’s contact information for each graduate), asking them to participate in an online survey about their professional attitudes and behaviours. A second e-mail request was sent 2 weeks after the first. The institutional ethics board approved this investigation, and informed consent was obtained from all participants.
Six different rating scales were used for the 15 items of the instrument (Table 1). The Rasch Measurement Model analyzed each scale separately, displaying 6 separate rating-scale evaluations. The category step calibrations for rating scales should differ by about 1.4 to 5.0 units and should progress in a stepwise fashion as the category rankings increase.18 When these requirements are not met and a rating scale does not function properly, collapsing the rating scale categories may allow the rating scale to function properly.
The response data generated with this professionalism instrument were either nominal or ordinal. Data that reasonably fit the Rasch Measurement Model were transformed into interval-level measurements. For this analysis, Winsteps software was used (2001 version; Linacre, Chicago, Illinois). Within the professionalism instrument, some items used different rating scales, so each rating scale was analyzed separately within the Rasch Measurement Model. The software accommodates dichotomous and various ordinal rating scales within the same instrument.19 To describe a true measure, the Rasch Measurement Model criteria and the results reported here should address rating scale step calibrations, person and item fit, principal contrast analysis, and a resulting variable map. Together, these variables demonstrate reliability and construct validity.13
E-mail messages were sent to a total of 98 alumni, with 6 messages bouncing back. A total of 27 individuals participated in the online survey.
The step calibrations and peak probabilities for every category of each rating scale are provided in Table 1. Only the rating scales for items 7 and 9 were problematic. However, once they were collapsed, both of these rating scales became functional.
Apart from the rating scale, individual items may not perform as expected for a number of reasons. Confusing terms or sentence structure may be a problem, or participants may interpret an item differently from the way in which the investigator intended. Misreading, guessing, or inadvertently selecting the wrong scale category may also occur. Rasch measurement analysis takes both item and person eccentricities (behaviour) into account, and allows investigators to remove items or people who “misfit” the model.
The Rasch Measurement Model analyzes both person and item fit. A “misfit” of either a person or an item implies poor data fit to the Rasch Measurement Model and consequent diminished measurement ability for the instrument. Any z-score values greater than 2.0 for the “in-fit” or “out-fit” represent items and people that “misfit” the proposed Rasch Measurement Model.18 Within the professionalism instrument, the mean z-score and standard deviation for person fit was 0.0 ± 0.8, whereas the mean item fit was 0.0 ± 0.9. However, one item and one person did misfit the model and were subsequently removed. The revised item separation was 3.64 logits, the revised person separation was 2.31 logits, and the revised reliability was 0.93 by Cronbach’s α. Item separation represents the number of units on the “ruler” that was constructed, whereas person separation describes the number of distinct groups into which the cohort was divided. Reliability (using a scale of 0 to 1), as commonly reported by Cronbach’s α, describes the internal consistency of the data and the likely repeatability of the evaluation. This instrument measured 3.5 units, divided the overall group of participants into 2 very distinct groups, and showed excellent reproducibility.
Within the Rasch Measurement Model, unidimensionality can be confirmed through principal contrast analysis, where the primary variable explains more than 60% of the variability in the measurement instrument. In this case, the model explained 99.9% of the variation in the data when this professionalism instrument was used. Beyond person and item fit, unidimensionality was confirmed.
The Rasch Measurement Model scales items from most difficult to least difficult, and scales each person from more to less able. That is, each person in this analysis was scaled above item responses that agreed with professionalism and below item responses that disagreed with professionalism. By logarithmic transformation of probabilities, a linear scale can be obtained. Only when data fit the Rasch Measurement Model, as they did here, can a measure be produced.
The variable map is a distinct feature of the Rasch Measurement Model (Figure 2). The linear map presents a professionalism construct showing each person (on the left) and each item (on the right). The equal-sized units of this linear measure are logits (or logarithm-odd units) and are intervallevel measures. Within the map, items are plotted from easiest at the bottom to most difficult at the top. As the items move from the bottom of the linear construct to the top, so does the level of professionalism. For each person, there is a scaling from the lowest level of professionalism to the highest. On the scale, items should vary (i.e., item separation) and be of similar units to the people. Only with similar units can the items truly separate the people into justifiably different groups (i.e., person separation).
For this map, participants were coded as age-sex-postgraduate training, with age presented in years, sex presented as a binary number, and postgraduate training presented as a 2-letter abbreviation. For example, “23-2-HR” means a 23-year-old woman who went on to a hospital residency, and “32-1- ” means a 32-year-old man who did not proceed with any postgraduate training. Compared with a statistical comparison such as a t test, this variable map provides more complete information about how age, sex, and/or postgraduate training were distributed within this construct of professionalism. The relative positions of individuals within the map illustrates that neither sex nor age appeared to affect this measure of professionalism, whereas postgraduate training did appear to have a considerable effect. Professionalism was greater among participants with postgraduate training, whether it was a residency, fellowship, or certificate. These results agree with what might be assumed rationally: the magnitude of professionalism should not differ according to sex or age, but postgraduate training should have an effect.
Measurement is the attempt to discover numeric relations between magnitudes of attributes; it is not an attempt to construct conventional numeric relations where they do not otherwise exist. Developing a measurement instrument that defines a linear, unidimensional, interval-level scale is foundational to any statistical application of data collected. Bypassing this important step in construct validity may lead to grave errors in the application of multidimensional and non-interval scales. The Rasch Measurement Model addresses fundamental measurement properties and creates a linear scale for measurement that is independent of both the items and the people in the analysis.
The Rasch measurement instrument for pharmacist professionalism is both “item-free” and “persons-free”. That is, the instrument’s functionality does not depend on which of the items are used as a measurement tool nor on the nature of the sample participants used in construction of the measurement tool.19 The instrument does not vary with the sample of people evaluated and can thus be applied to populations that differ from the original sample. Therefore, this invariant professionalism instrument should function very similarly in other cohorts of recently graduated pharmacists.
Using grades as the only quantitative outcome for pharmacy education programs can be problematic. This instrument provides an objective measure of pharmacist professionalism that could serve as an advantageous outcome measurement for pharmacy graduates. To date, no other objective measure of professionalism has been constructed. As well, no prior instrument for pharmacy students can be applied as an outcome measure for pharmacy education. The Rasch Measurement Model may be used as an additional measure of pharmacy students’ competency in an area that is currently not evaluated objectively. In addition, this tool for assessing pharmacists’ professionalism has potential for use in formally assessing the professionalism of residents in training or during the process of selecting from among residency candidates. The applicability of this instrument for these potential indications requires further study.
This measurement instrument has some limitations, and 3 issues merit elaboration. First, the Rasch Measurement Model requires fewer participants than other measurement techniques. While the sample size in the current study was small, the results are promising, because the data fit the model so well. Second, only about one-third of the invited pharmacists responded to the request to participate in this study. It is possible that there were differences between responders and nonresponders that would lead to bias in the model. Third, few instrument items assessed each and every tenet of the 6-tenet definition of professionalism. A greater number of items for each tenet might improve the measurement, but at the cost of participants having to complete a longer and more arduous instrument. Furthermore, experienced practising pharmacists well beyond recent graduation may characterize all or many of the tenets of professionalism, although the narrow range of items may not capture the totality of their inherent professionalism. This instrument should therefore be limited to recent graduates.
The Rasch Measurement Model instrument appears to be a reasonably objective measure of pharmacists’ professionalism, with excellent reliability. This instrument could be used as an outcome measure for pharmacy education in the future. Further study is required to better define the utility of this measurement tool for assessing pharmacists’ and pharmacy students’ degree of professionalism.
We thank Glenn D Whiteway, BScPharm, for assistance with reviewing this manuscript before submission.