|Home | About | Journals | Submit | Contact Us | Français|
To describe the conceptual development of a measure for assessing pharmacist knowledge of herbal and dietary supplements.
A standardized approach to constructing a multiple-choice competency examination following 8 pre-specified criteria (eg, specifying the target spectrum of herbal and dietary supplements) was used to create an item bank. The quality of each item was evaluated by 5 herbal and dietary supplement content experts based on specific criteria in 3 rounds of review.
From 122 initial items, 56 items were retained for the item bank representing 4 content areas: efficacy/effectiveness, safety, drug-supplement interactions, and regulation. The experts tended to agree that the constructed items represented a wide range of difficulty.
The initial development of a conceptually based item bank/measure of pharmacist herbal and dietary supplement knowledge lays the groundwork for a large-scale validation study. The measure should be useful as a standalone tool and as a component of a knowledge, attitude, and behavior survey for the assessment of pharmacist traits related to herbal and dietary supplements.
The marketing of herbal and dietary supplements has led to dramatic growth in patient use of these products in the last 2 decades.1-4 As health care professionals whose expertise focuses on pharmacotherapy, pharmacists have an important role in evaluating the safety, efficacy, and effectiveness of herbal and dietary supplement products and guiding patients concerning their appropriate use.5 However, few studies have investigated pharmacists' knowledge level of herbal and dietary supplements and the extent to which pharmacist knowledge and attitudes towards such products contribute to patient counseling.
Researchers in the field of health care have a tradition of developing and conducting knowledge, attitude, and behavior surveys to gather information, evaluate current practice, and effect curricular refinement.6 A review of knowledge, attitude, and behavior studies reported that pharmacists perceived they did not possess sufficient knowledge of herbal and dietary supplements.7 Problematically, there was little evidence to support the availability of conceptually based, validated measures to assess a pharmacist's knowledge concerning herbal and dietary supplements, either as a standalone measure or within the context of a knowledge, attitude, and behavior survey. The purpose of this study was to report the initial development of a measure to assess pharmacist knowledge of herbal and dietary supplements. This represents part of a larger study to develop and validate measures of pharmacists' knowledge, attitude, and counseling on herbal and dietary supplements, which will be used to evaluate the interrelationships between these traits/constructs of interest.
A survey instrument that measures knowledge is typically created to capture respondent knowledge and understanding of scientific and clinical content areas.6 A format, such as multiple-choice or true/false items, provides a structure for a metric that is used to assess an individual's total score. The score is intended to represent the participant's actual knowledge. Regardless of the type of item structure, there is always a best or most correct answer for each item on a knowledge measure. The advantage of using multiple-choice items over true/false items is that multiple-choice items incur less chance of “blind guessing,” which refers to the situation where there is an equal probability of a respondent endorsing each option, where the fewer the options, the higher the likelihood that a respondent's guess will result in the correct answer. Multiple-choice items also demonstrate certain response patterns that are more informative than a dichotomous scoring structure (ie, scoring “1” for choosing the keyed option and scoring “0” for selecting any incorrect distractor options).8 However, the advantage of adopting multiple-choice items for actual knowledge assessment in a survey can be jeopardized when problematic, defective, or biased items and/or their options (choices) are used. To minimize this phenomenon, well-established procedures for item generation and review should be followed. Subsequently, an intensive statistical evaluation of the item responses is then implemented.8
In this study, herbal and dietary supplements refer to those products that follow the definition of Dietary Supplement Health and Education Act of 1994 (DSHEA): “a product (other than tobacco) that is intended to supplement the diet that bears or contains one or more of the following dietary ingredients: a vitamin, a mineral, an herb or other botanical, an amino acid, a dietary substance for use by man to supplement the diet by increasing the total daily intake or a concentrate, metabolite, constituent, extract, or combinations of these ingredients.”9 Knowledge-based items were created to cover the most frequently used and popular products based upon literature searches, existing measures, web sites, and internal expert opinion.
A structured method of constructing knowledge items was used that mirrored Pray and Popovich's study for a standardized competency examination.10 An internal review of the items was conducted by 2 experts with herbal and dietary supplement teaching experience in the pharmacy curriculum, postgraduate continuing education, and herbal and dietary supplements counseling with patients and other health care providers. The items were constructed with 8 criteria in mind.
The target population was practicing pharmacists who possess a broad range of knowledge about herbal and dietary supplements used in the United States.
The herbal and dietary supplements measure was created as a diagnostic assessment, ie, formative assessment, to determine the extent to which pharmacists have mastered herbal and dietary supplement-related knowledge in their formal and continuing professional education.
The top 10 herbal products and top 10 supplements, as well as current issues related to herbal and dietary supplements were identified from literature, government web sites, and marketing authorities.11-13 Herbals included St. John's wort, valerian, Ginkgo biloba, chamomile, saw palmetto, ginger, ginseng, garlic, feverfew, and soy products. Co-enzyme Q, creatine, folic acid, melatonin, fish oils/fatty acids, multivitamins, vitamin E, minerals (ie, zinc, calcium/magnesium), glucosamine, and chondroitin constituted the non-herbal dietary supplements.14-17
The multiple-choice items were designed with the 4-option response structure in mind, labeled as A, B, C, D (ie, 1 best answer as the keyed option and 3 alternative option as distractors) were constructed.
Initially, the items were designed to include 6 themes, in no particular ranking of importance: efficacy/effectiveness, safety, drug-supplement interactions, regulation, information retrieval /evaluation, patient counseling.
Objectives of the measure were to discern levels of knowledge about herbal and dietary supplements that would enable pharmacists to help patients and consumers make informed decisions about their use. These include the ability to:
In some respects, the application of evidence-based practice can be recognized as an extension of a systematic approach to drug information.18 Originally, the recommended systematic approach for pharmacists to respond to a drug information request included 5 steps: classifying the request, securing background information, performing a systematic search, creating and providing the response, and classifying the information requested.19 Although more limited literature and evidence on herbal and dietary supplements is available compared to conventional medicine, the stepwise approach (ie, consulting tertiary and secondary sources, followed by primary resources) of drug information retrieval and evaluation in conventional medicine was adopted in this study.20
Specifically, the target literature resources included: (1) tertiary resources, eg, the most popular textbooks containing content related to herbal and dietary supplements (ie, Handbook of Nonprescription Drugs, 14th and 15th editions; Nonprescription Product Therapeutics, 2nd edition)21,22 and books with name recognition23,24; (2) full-text computer bases, review articles, and secondary resources, which are indexing and abstracting services of the primary literature7,25-36; (3) research studies published in biomedical journals from the primary literature (ie, using English language; 1966-2006; key terms included herbal and dietary supplements [or specific ingredient name], efficacy, adverse reactions, and/ or evidence-based medicine) as identified in PubMed37-39; and (4) official web sites related to herbal and dietary supplements (eg, Office of Dietary Supplements, National Institutes of Health, and reports posted by AHRQ Evidence-based Practice Centers, The Cochrane Library).9,25-54
Three cognitive domains adopted from Bloom's taxonomy (ie, knowledge, comprehension, application) for item construction were utilized to categorize items.55 Specifically, the knowledge domain included activities related to one's knowledge, memory, and ability to know the correct answer. The comprehension domain contained activities for translating, interpreting, and extrapolating knowledge. The application domain included the activities of knowing when/why to apply information, especially for a new situation.55
A 31-item checklist for item creation included 4 main categories related to item writing quality: content concern, style concern, the written stem, and item alternatives.56 For instance, in terms of content, items were intended to reflect specific content, evaluate a single mental behavior as intended by the test specifications, avoid being “opinion-based,” and not perceived as being “picky.”56 During the item generation process, this checklist was used as a template to reduce item writing problems.
A 2-way table that specified the content themes to be tested and the desired cognitive domains was created prior to generating the set of items. The expected proportions of herbal and dietary supplements items related to each content theme and cognitive domains were established (Table (Table1).1). The marginal proportion of items related to each cognitive domain (ie, knowledge, comprehension, application) were pre-specified at 45%, 33%, and 22%, respectively. The marginal proportion of each content theme was projected as follows: efficacy/effectiveness, 30%; safety, 20%; drug-supplement interactions, 12%; regulation, 15%; information, 12%; and patient counseling, 11%). The intention was to place greater emphasis on the knowledge and comprehension cognitive domains than on the application domain. Three themes (ie, drug interactions, information retrieval and evaluation, patient counseling) constituted approximately 11% of all items in the item bank. Thus, the numbers of items for each theme were initially generated using the guidelines in Table Table1.1. For instance, approximately 7% of all items were created to address the topic of efficacy/effectiveness in the cognitive domain of application.
For the constructed items, 2 separate item reviews were performed. First, draft items were reviewed by lay persons who were not practicing pharmacists. They proofread the items and made suggestions regarding item structure, meaning, and flow. The second content review process was conducted by internal and external experts, including 5 pharmacy-based faculty members and health professional educators who evaluated all items. Two out of 3 external experts (ie, who were not faculty members in the College of Pharmacy) were involved in each round of the “external review,” using the same review process and electronic content validation format.
The internal experts reviewed the items based on the following criteria: (1) representativeness of the construct; (2) ability to update information in items; and (3) ability of items to span different levels of difficulty. Based on feedback, more items were created or revised to meet the intended proportion related to the content theme and cognitive domain. The first and second drafts of items in the item bank were distributed to external experts to assess the quality of constructed items. At this stage of the process, the reviewers were asked to provide the keyed response, assign the cognitive domain, assign the corresponding content theme, specify the level of representativeness of the item, specify the level of its difficulty, and provide any editorial suggestions for revision.56 Accordingly, an item was retained in the item bank only if it met all the following criteria for each round of external reviews: at least one expert answered correctly, not all experts guessed, and the mean of representativeness rating was at least 3.5. Upon receiving feedback from external experts, items were retained, revised, or removed for the next draft of items after consultation with the internal experts. This initial development of the measure was approved by the Institution Review Board of the University of Illinois at Chicago.
Figure Figure11 presents the item development process for the knowledge measure. Originally, 122 items were created for the item bank. Ten items were added to meet the desired proportion of items related to each content theme and cognitive domain in the first draft of items in the item bank. Twenty-one items constructed on the theme of patient counseling appeared to assess the pharmacists' practice experience. Upon re-evaluation, these were deemed out of the scope for a knowledge-specific measure for herbal and dietary supplements. Hence, these items were removed from the first draft of the item bank. After completion of each of the 3 rounds of expert review, there were 111, 98, and 56 items retained, respectively, in the item bank.
When grouping the items based on their cognitive domain and specified content themes (Table (Table2),2), the proportion of items created to assess respondents' comprehension domain were consistent (ie, 35.1%, 36.7%, 33.9% in the first, second, and third drafts for the item bank, respectively) with the expected proportion (ie, 33%). The proportion of items developed for the other themes deviated somewhat from the pre-specified proportions in Table Table1.1. In the second draft of items, 13 items (ie, 13.3% of the total item pool) were answered incorrectly by 2 external experts (data not shown); these items were dropped. The remaining 85 items were considered further because at least 1 expert answered the item correctly. Twenty-nine items were removed from the item bank because those items: (1) were answered correctly by experts due to self-reported guessing; (2) were unrepresentative (ie, mean representativeness rating < 3.5); (3) contained out-of-date information; or (4) were too subjective (ie, opinion-based items). For example, questions addressing the regulatory requirement for good manufacturing practices for herbal and dietary supplements were removed because the legal requirement to do so was just put into effect on June 22, 2007.57 None of the items in the theme of information retrieval/evaluation were retained in the third review of the item bank. Of the 56 remaining items, approximately half of the items related to factual knowledge (ie, 50 % assigned in knowledge domain). Only 16 % of the retained items measured the application domain (Table (Table2),2), which was considered the most complex domain in terms of higher mental processes.
Table Table33 presents the results of the “external” expert content validation process at each review step. Of the 56 items in the third draft of the item bank, 44 (79%) were answered correctly without self-reported guessing by 2 external experts. External experts tended to agree that the items were difficult (ie, mean of difficulty = 3.4, on a scale on which 1 = extremely easy to 5 = extremely difficult) but representative (ie, mean of representativeness = approximately 4.0, on a scale on which 1 = not representative and 5 = very representative).
The goal of this study was to apply a systematic approach to develop a measure that assesses pharmacist herbal and dietary supplements-related knowledge. Items were written consistent with guidelines and content themes, producing 56 items for the final item bank that forms the basis for the herbal and dietary supplement knowledge measure.8,10,58 None of the items corresponding to the theme of information retrieval/evaluation were retained in the item bank after compiling the experts' opinion. Upon deliberation, the internal expert panel agreed this theme represented a process that was not specific to herbal and dietary supplements and not an outcome that represented actual herbal and dietary supplements knowledge. Therefore, 4 content themes (efficacy, safety, drug interaction, regulation) concerning herbal and dietary supplements were retained. These 4 themes were consistent with the issues pharmacists confront with regard to common herbal and dietary supplements products according to expert opinion and guidelines proposed by the American Dietetic Association and the American Pharmacist Association (ADA/APhA).5,30,59 The proportion of the total number of items related to each content theme and the cognitive domain of the corresponding draft of items was guided by the proposed proportion of items. Although a smaller proportion of items were retained for the application domain, the domain met the rule of thumb of at least 10 items per cognitive domain.58
When survey design is employed, the purpose of a measure should be clearly delineated, with items formulated to capture the constructs of interest.8,60 To our knowledge, no previous measures of pharmacists' knowledge level of herbal and dietary supplements have followed a structured approach in constructing a standardized competency examination. The results presented in this manuscript represent the first stage of development. In the second stage, the measure will be piloted with a representative group of pharmacists, followed by a large scale validation study. These results will help with the development of a short form derived from the item bank. In the future, the item bank can be reviewed, updated, and/or revised periodically and new short form measures can be developed from the pool of items in the herbal and dietary supplement knowledge-based item bank.
At the present stage, measure development has relied on expert opinion in the selection and refinement of the items germane to each theme. Psychometric analysis will need to be conducted to further evaluate how well the items were constructed in the item bank. Although fewer items in the application domain were retained in the final item bank, sufficient domain coverage is expected and items can be added to the item bank in the future.
This study reports the initial development of a conceptually based item bank/measure of herbal and dietary supplement knowledge. An item bank of 56 items was developed covering 4 content areas relating to herbal and dietary supplements (ie, efficacy, safety, drug interactions, regulations) and 3 cognitive domains, including knowledge, comprehension, and application. The next phases include large-scale validation, psychometric analysis, and development of a short form version of the measure based on the item bank.
The authors would like to thank the following reviewers and external experts for providing their insights and comments on the draft items in the item bank: Heather Boon, Maria Charisse De Leon, John Hessian, W. Steven Pray, and Matthew Sargent.