Figure presents the item development process for the knowledge measure. Originally, 122 items were created for the item bank. Ten items were added to meet the desired proportion of items related to each content theme and cognitive domain in the first draft of items in the item bank. Twenty-one items constructed on the theme of patient counseling appeared to assess the pharmacists' practice experience. Upon re-evaluation, these were deemed out of the scope for a knowledge-specific measure for herbal and dietary supplements. Hence, these items were removed from the first draft of the item bank. After completion of each of the 3 rounds of expert review, there were 111, 98, and 56 items retained, respectively, in the item bank.
Item development process of the knowledge measure.
When grouping the items based on their cognitive domain and specified content themes (Table ), the proportion of items created to assess respondents' comprehension domain were consistent (ie, 35.1%, 36.7%, 33.9% in the first, second, and third drafts for the item bank, respectively) with the expected proportion (ie, 33%). The proportion of items developed for the other themes deviated somewhat from the pre-specified proportions in Table . In the second draft of items, 13 items (ie, 13.3% of the total item pool) were answered incorrectly by 2 external experts (data not shown); these items were dropped. The remaining 85 items were considered further because at least 1 expert answered the item correctly. Twenty-nine items were removed from the item bank because those items: (1) were answered correctly by experts due to self-reported guessing; (2) were unrepresentative (ie, mean representativeness rating < 3.5); (3) contained out-of-date information; or (4) were too subjective (ie, opinion-based items). For example, questions addressing the regulatory requirement for good manufacturing practices for herbal and dietary supplements were removed because the legal requirement to do so was just put into effect on June 22, 2007.57
None of the items in the theme of information retrieval/evaluation were retained in the third review of the item bank. Of the 56 remaining items, approximately half of the items related to factual knowledge (ie, 50 % assigned in knowledge domain). Only 16 % of the retained items measured the application domain (Table ), which was considered the most complex domain in terms of higher mental processes.
Proportion of Herbal and Dietary Supplement Items Related to Each Content Theme for Each Iteration of Item Development
Table presents the results of the “external” expert content validation process at each review step. Of the 56 items in the third draft of the item bank, 44 (79%) were answered correctly without self-reported guessing by 2 external experts. External experts tended to agree that the items were difficult (ie, mean of difficulty = 3.4, on a scale on which 1 = extremely easy to 5 = extremely difficult) but representative (ie, mean of representativeness = approximately 4.0, on a scale on which 1 = not representative and 5 = very representative).
Summary Results of “External” Expert Content Validation