Fifty general practitioners in Sydney, Australia, completed self administered questionnaires,2
in which they rated their understanding of each of seven terms used in evidence based medicine as “Would not be helpful for me to understand,” “I don't understand but would like to,” “I already have some understanding,” and “I understand this and could explain to others.” We considered the last response to represent full understanding (self rating of competence). Participants sealed their responses in an envelope before participating in a structured interview with JY (who was unaware of their self rating), in which they were asked to explain each term as if to a medical student. Unprompted comments were recorded (see box on bmj.com). The study was approved by the Central Sydney Area Health Service Ethics Review Committee.
Three independent experts in evidence based medicine had been asked to identify criteria essential for showing that the participant knew the correct meaning of the term (criterion based assessment; see table on bmj.com). During interviews with general practitioners, JY ticked any criterion met by participants' verbal explanations. To demonstrate competence in understanding number needed to treat, for example, participants had to include in their verbal responses the concept that this represents the number of patients needed to be treated to achieve one good outcome or prevent one bad outcome and that it is the reciprocal of absolute risk reduction.
Participants' verbal explanations almost never met the essential criteria (table). Although self ratings were modest, only one participant's explanation met all essential criteria, and this for only one term, positive predictive value.
We could not calculate sensitivity and specificity of self rated competence for any terms other than positive predictive value as only one respondent met objective criteria for competence. We calculated positive and negative predictive values for each term to assess the probability of competence given a positive or negative self rating. The predictive value of a positive self rating was 8% for positive predictive value but zero for the other six terms (table). As no participants demonstrated competence exceeding their self rating, the predictive value of a negative self rating was 100% for all terms.