In the past decade, Evidence Based Medicine (EBM) has become mainstream (
1). EBM can be defined as “the conscientious, explicit, and judicious
use of current best evidence in making decisions about the care
of individual patients” (
2). In it’s original conception, EBM asked practitioners to be able
to access the literature at the point of need. This process included
four steps. First, formulate a clear clinical question from the patient’s
problem. Second, search the primary literature for relevant
clinical articles. Third, critically appraise the evidence for validity
and finally, implement these findings in practice (
3).
Physicians lack the time to perform all of the steps outlined above. In
an observational study, participating physicians spend less than two
minutes seeking the answer to a clinical question (
4). In addition to the lack of time, many clinicians may also lack the skills
needed to search, analyze and synthesize the primary literature (
5). While it could be argued that clinicians could or ought to learn these
skills, there are already other professionals with theses skills. In
a modern team-based approach to healthcare it is natural for others
to do the information synthesis and analysis. In the McColl study cited
above, 37% of respondents felt the most appropriate way to practice
EBM was to use “guidelines or protocols developed by colleagues
for use by others.” Only 5% felt that “identifying
and appraising the primary literature” was the
most appropriate. With others doing searching and synthesizing, assembling
the evidence no longer needs to be done as the need arises. It is
access to these pre-digested forms of information that most readily supports
the use of evidence in clinical decision-making.
There are many electronic products, which have been specifically designed
to provide this sort of synthesized information to the clinician. For
our purposes we’ll call these bedside information tools. Because
they have been designed to provide information at the point of care, they
tend to be action oriented. They are designed to answer questions
directly related to patient care rather than background questions. Because
these resources are electronic they can be updated much more
frequently than print resources and it can be easier to search for and
share information.
As with other forms of secondary literature, there are a number to choose
from. With limited financial resources, libraries and other institutions
cannot afford to subscribe to or purchase all available bedside
information tools. Providing access to all the bedside information tools
is not a practical solution. The question then is, which bedside information
tools should be selected and supported within an institution
or library? In order to make this decision we need a method for evaluating
these products.
When looking for evaluation methods one can turn to evaluation methods
used to evaluate other information retrieval (IR) systems. IR evaluation
methods can roughly be grouped into system-centered or user-centered. Popular
system centered metrics for evaluating IR systems are recall
and precision, but because of their focus on document retrieval and
relevance, recall and precision may not be suitable at all to measure
whether or not a given system can answer clinical questions. In a 2002 study, Hersh
found no correlation between recall and precision measures
and the ability of students to answer clinical questions using MEDLINE (
6). Recall and precision, therefore are not appropriate for the evaluation
of these bedside information tools.
Another System-Centered evaluation is system usage. While these measurements
can show how much a system is used relative to others, the measure
is only use, not successful use. What can be extrapolated beyond that
the product is being used is questionable. Usage data can also only
be collected on products available for use, so one cannot collect data
on products only being considered for purchase. System usage is also
inappropriate for the evaluation of these bedside information tools.
User-Centered evaluations look at the user and the system as a whole when
evaluating IR systems. Task-Oriented evaluations ask users attempt
a task on the system, then asks was the user able to complete the task. A
Task-Oriented evaluation of bedside information tools asks the question “Can
users find answers to clinical questions using this
tool?” By having users attempt to complete these tasks and measuring
their success we can answer this question for these products. By
looking at both the system and the user these evaluations measure whether
or not an IR system can do what it is designed to do.
User-satisfaction goes one step further and asks not only “Did
the system do what it was designed to do?” but also “Was
the user satisfied with the way it did it?” By measuring user
satisfaction an evaluation can identify not only those products, which
successfully answer clinical questions, but also those products that
the users enjoy using the most.
Evaluating and selecting products for others to use can pose a challenge. Evaluators
need to be explicit in including users in these evaluations. Although
products must meet minimum standards of quality, by not
addressing users in an evaluation, one risks choosing a product that does
not suite the user’s needs. Therefore, a user-centered, task-oriented
approach is needed to evaluate bedside information tools.
The specific products evaluated in this project are ACP’s PIER, DISEASDEX from Thomson Micromedex, FIRSTConsult from Elsevier, InfoRetriever and UpToDate. These products were selected for practical considerations. These products
are market peers, and would consider each other to be their competition.
This article has been 
