|Home | About | Journals | Submit | Contact Us | Français|
The research sought to establish a rubric for evaluating evidence-based medicine (EBM) point-of-care tools in a health sciences library.
The authors searched the literature for EBM tool evaluations and found that most previous reviews were designed to evaluate the ability of an EBM tool to answer a clinical question. The researchers' goal was to develop and complete rubrics for assessing these tools based on criteria for a general evaluation of tools (reviewing content, search options, quality control, and grading) and criteria for an evaluation of clinical summaries (searching tools for treatments of common diagnoses and evaluating summaries for quality control).
Differences between EBM tools' options, content coverage, and usability were minimal. However, the products' methods for locating and grading evidence varied widely in transparency and process.
As EBM tools are constantly updating and evolving, evaluation of these tools needs to be conducted frequently. Standards for evaluating EBM tools need to be established, with one method being the use of objective rubrics. In addition, EBM tools need to provide more information about authorship, reviewers, methods for evidence collection, and grading system employed.
Health care providers require quick retrieval of information to efficiently answer questions related to patient care . Evidence-based medicine (EBM) tools, designed for use at the point of care (POC), meet these needs by providing high-quality and synthesized information at the patients' “bedside” . To support clinical information needs, medical and hospital libraries may wish to subscribe to multiple POC tools, owing to the variety in topics covered, the options provided, and the audiences targeted by each tool. To avoid redundancy and to offer the best products for their users, libraries need to periodically reassess available EBM POC resources . What criteria should librarians utilize to evaluate these POC tools? Because these resources are used to make clinical decisions, resources providing the best evidence relevant to patient care would be most desirable. Those who use these tools “need to know how much confidence they can place in the recommendations” . One role of medical librarians is to evaluate the quality of EBM resources so that users can have confidence in the information upon which they base clinical decisions. This is a weighty responsibility, and librarians have an important challenge determining which EBM POC resources to make available to users.
The Texas A&M University Medical Sciences Library (MSL) is 1 of 5 libraries on the College Station campus of Texas A&M University. MSL houses 120,000 print titles and 1,600 serials, with a collection budget of more than $1.8 million. One of the library's user groups is the Texas A&M Health Science Center (TAMHSC), which includes the TAMHSC College of Medicine, College of Nursing, School of Graduate Studies, Rangel College of Pharmacy, and School of Rural Public Health. The director of the MSL charged a team of 3 librarians in 2008 to evaluate and determine which EBM POC subscription-based tools the library should keep or purchase, in order to reduce overlap and to ensure that user needs were met. In 2008, MSL subscribed to 5 EBM POC tools: Stat!Ref's ACP PIER, DynaMed, Cochrane Library, Essential Evidence Plus, and MD Consult's First Consult. The librarians were asked to evaluate all subscription EBM POC resources. No directions were given as to how to perform this evaluation. To determine which tools to evaluate and to find best practices for evaluation, the librarians conducted a literature review.
The literature review included searching MEDLINE (PubMed), CINAHL (EBSCO), Library Information Science Technology Abstracts (LISTA) (EBSCO), and Google Scholar. Search terms included combinations of “evidence based medicine tools,” “point of care tools,” “bedside tools,” “evaluation,” and names of specific tools. The researchers were particularly interested in studies reviewing EBM tools to which the library currently subscribed. UpToDate, Epocrates, and Ovid's Clin-eguide were tools in which users had expressed interest and were also used as search terms. Articles excluded from the search were opinion papers, either letters to editors or journal columns.
Fourteen research-based articles were found evaluating specific features of EBM POC tools for clinical medicine (Table 1). Found studies focused mainly on EBM tools that physicians use. One article was found on tools for clinical laboratory specialists . The current status of each tool mentioned in the articles was verified. Some tools' names had been changed since the article was written, such as InfoPOEMs/InfoRetriever, which is now Essential Evidence Plus [2, 5–7]. Other products included in studies no longer exist, such as MAXX . Negative reviews of the tools were often later resolved. For instance, a 2003 review of UpToDate found the information for patients weak , and UpToDate has since created a version specific for patients . In another example, DynaMed was criticized for not clearly grading its evidence , and DynaMed producers responded by creating a description of the levels of evidence used in the product .
Retrieved articles varied by who acted as the tool reviewer (the researcher versus the user) and by the platform used during the evaluation (computer versus mobile device). Nearly 80% of the articles included 1 or more clinicians (MDs) as reviewers. Backgrounds of authors varied among medical librarians, practicing physicians, clinical faculty, medical informaticians, graduate students, and research coordinators. The evaluations also took place in a wide range of locations including hospitals, private practice, laboratories, and libraries. Some authors evaluated both free tools, such as Google, and subscription-based tool, such as DynaMed [2, 7, 8, 13–19]. Differences were evident in how EBM tools were defined and compared. One study divided tools into two categories . The first category, “evidence-based resources,” included bibliographic databases and summary resources. Resources such as PubMed, Cochrane Database of Systematic Reviews, First Consult, InfoPoems, and DynaMed were referenced in this group. The second group, defined as “review resources,” incorporated “evidence-based content” without an “explicit process to ensure that all relevant literature is selected and appraised” . Examples included AccessMedicine, MD Consult, Stat!Ref, and UpToDate . Other evaluations did not differentiate between bibliographic versus synthesized tools, comparing resources like PubMed, MEDLINE, and Cochrane to UpToDate [5, 7, 15, 17–19]. Another evaluation defined three EBM tools as “e-textbooks” .
Most evaluation metrics in these studies were designed to determine whether answers to clinical questions were retrieved in the tools. Either preselected questions or those formulated at the point of care were used in the evaluations. Methods varied as to how preselected questions were gathered. Some studies additionally factored in the average search time required to locate the answer. One article used a scoring system created by the authors to assess overall content in each tool.
Due to the wide variation in the studies' researchers, methods, and evaluated tools, comparisons between studies and their respective recommendations proved difficult. Overall conclusions varied. Some reviews rated no product higher than another [18, 20], multiple studies rated UpToDate highest [2, 9, 14, 16, 19], and one rated DISEASEDEX highest . UpToDate's easy-to-use interface was cited as a reason for its popularity. Campbell acknowledged that participants in her study had used UpToDate previously and therefore were already familiar with the search interface . Kim explained that “bibliographic databases such as PubMed and Medline [were] difficult to navigate, and current evidence-based synopses or summary sources [had] small databases that cover[ed] a limited content” . Two of the studies that found DynaMed valuable included the product's creator as one of the authors [8, 13].
While most previous evaluations of EBM tools focused on health professionals' and researchers' perspectives on the ability of a tool to answer a clinical question, the librarians in this study did not feel qualified to determine if a tool sufficiently answered a clinical question. Also, the researchers wanted to ultimately develop a process with which they could regularly evaluate the tools in their collection. Due to these factors, the goal of this study was to assess the attributes, the standards and clarity of evidence grading, and the level of transparency (authorship or review processes) of the tools. The following objectives were identified to achieve these goals:
In the spring of 2009, a list of EBM tools was created from those found in the literature review, from tools that patrons had requested, and from tools to which Texas A&M University subscribed. As illustrated in the literature review, definitions for and categories of EBM tools varied. Tools that were subscription based, claimed to be evidence based, claimed to be POC, and focused on clinical content were chosen for the review. Because the original charge from the library director was to compare subscription-based products, free resources were excluded. Evidence of the first three criteria could be found on the product's website or brochures. Emphasis on clinical content was determined by reading product descriptions on the site. A sample of academic medical and hospital libraries' holdings was also reviewed to ensure tools were not missed and to compare the researchers' holdings with that of the sample. The South Central Academic Medical Libraries consortium (SCAMeL), of which MSL is a member, was selected as the sample. Libraries in the consortium vary in size and users or programs supported, providing a spectrum of medical libraries appropriate for the sample that the researchers desired. Each SCAMeL library's website was searched, based on the screening criteria, to find a listing of EBM tools available to their users. No additional tools were found when the SCAMeL holdings were compiled.
The tools found were then were entered into a form made using Zoho Creator, a web-based software product that creates databases with forms and tables . Each tool was screened by at least two reviewers, using an online form created in Zoho Creator . Three screeners discussed discrepancies before coming to a consensus about tools to exclude. This form was completed in 2009 and again in 2010 to show changes in subscriptions.
Based on the screening criteria, the researchers chose to include the following tools in their evaluation: ACP PIER, BMJ Point of Care, Clin-eguide, DynaMed, Epocrates, Essential Evidence Plus, First Consult, and UpToDate. The authors chose to exclude any tool that was strictly a pharmaceutical resource, as the target for this evaluation was clinical medicine.
Each tool was evaluated in two ways: by rating common attributes on a general evaluation form and by determining the tool's ability to provide a treatment summary on commonly diagnosed conditions.
The general evaluation form, produced in Zoho Creator, was developed to analyze all parts of the resource and initially included four sections: general characteristics, scope and content, search options, and results presentation. The evaluation form questions were adapted from a presentation by Trumble in 2004 (in which one of the authors participated) . As the evaluation progressed, it became apparent that an important difference between these tools was how the evidence was graded. Evidence that has been compiled and evaluated by authoritative bodies, or in other words, “graded,” has the most weight in EBM [2, 23]. Because the reviewers wanted to examine more closely how the evidence was graded, a fifth section was added to the form.
The final evaluation form included the following sections: Section A included measures of general characteristics of the tool, including availability of platform compatibility, simultaneous users, mobile devices accommodated, free version, and available customizations. Section B parameters were the scope and content, including availability of patient handouts, continuing education credits, practice guidelines, and help screens. Section C covered the search options (browsing, guided searches for pharmaceuticals, advanced), navigation, and presentation of the results (readability, adjustments, print options, and other outputs). Section D focused on quality control, including listing of authors or reviewers' names, credentials, and affiliations; the process of becoming an author or reviewer; updating schedule; and any biases (affiliations were considered to constitute a conflict of interest). Section E included the standards used for grading and the clarity of the grading levels.
During the general evaluation, the researchers also noted that the consistency with which the evidence summaries were graded varied between tools. In summer of 2010, the investigators wanted a snapshot of the extent to which grades were provided for common clinical topics in each tool. A list of common primary diagnoses was located in the Centers for Disease Control and Prevention's “National Ambulatory Medical Care Survey: 2005 Summary” . The investigators selected the top ten diagnoses to search across each EBM tool. For those terms that were very broad (such as malignant neoplasms), librarians selected the most common form of the disease (such as skin cancer). The final list of ten terms searched was: hypertension, upper respiratory infection, arthritis, skin cancer, diabetes, low back pain, ear infection, asthma, high cholesterol, and sinusitis.
A Zoho Creator form was made for each clinical topic addressing the following criteria: Was a section on the topic provided? Was the evidence graded? Was it summarized? Were summary tables provided? For each question, a yes or no option was listed, as well as the number of clicks to find the answer, and a notes field. The number of clicks was counted from the main page to the appropriate section of the results page. Two librarians searched each term in every tool.
Table 2 summarizes the results of the general evaluation. In terms of platforms available for tools, all but two tools were compatible with PC and Mac computers. Several tools mentioned mobile availability, with DynaMed listing the most mobile platforms. Customizations for all tools were limited to changing the font size and customizing patient handouts. Subscription options were fairly consistent across tools, with only ACP PIER using a charge-per-seat model, while others had switched to unlimited seat access. All evaluated tools included some practice guidelines in their summaries. Six offered some continuing education credits, mostly to physicians. Little variation was seen in search options and presentation of results across tools. However, one obvious difference noted in the search for pharmaceuticals was that some tools offered additional options, such as pill identification, indications, and counter indications.
The quality control of the evidence in the resources was of particular interest to investigators. Most tools listed the authors of individual summaries. DynaMed and UpToDate specified the process to become an author. Most of the tools provided the authors' credentials and affiliations. Two resources, DynaMed and Essential Evidence Plus, explained the process to become an editor, and five of the resources (BMJ Point of Care, DynaMed, Epocrates, First Consult, and UpToDate) listed the editors' names and credentials. Six of the tools claimed to update their summaries in six months or less. The standard of grading was described in five of the tools: One used Strength of Recommendation Taxonomy (SORT) ; two used Grading of Recommendations Assessment, Development and Evaluation (GRADE) ; and two created their own grading systems. Three of the tools did not specify a grading process or standard.
Table 3 summarizes the results of searching for 10 specific diagnoses treatment summaries in each resource with the exception of BMJ Point of Care, which provides the disease content for Epocrates. Rather than evaluating the same content twice, the investigators chose to search the content solely in Epocrates. A total of 70 treatment summaries were retrieved, 10 for each of the 7 tools. Evidence supporting possible treatments was often summarized in narrative form (average 8.1 times per tool), but not often in table form (average 2.3 times per tool). Transparency in authorship varied, with 5 out of 7 listing individual authors, 4 providing credentials, and 4 providing affiliations. One bias was noted while reviewing the affiliations, with a regional director for Merck listed as an author for ACP PIER. The average number of authors listed in a summary was 1.6 for those 5 tools that listed authors. Fifty-two of the 70 retrieved searches (74%) graded the evidence. DynaMed had the most up-to-date summaries (updated on average within 19 days), while First Consult had the least up to date (updated on average within 449 days). Six tools claimed to update summaries within 6 months or less. For the 10 topics searched, however, only DynaMed met this claim.
While some articles in the literature review favored one EBM tool, the major finding of this review was that current EBM POC tools were quite similar relative to their content coverage and search options. Reporting of quality control measures was where researchers noted the greatest difference between tools. Level of transparency of authorship, evidence selection process, and grading standards varied greatly between tools. Transparency in processes is vital to evaluating reliability and completeness to determine possible biases in the evidence. This need for transparency in evidence summaries has been expressed by those developing standards for practice guidelines and systematic reviews [23, 26].
Two rubrics for the review were created: one to evaluate tool characteristics and another to assess the evidence presented in treatment summaries. The Zoho Creator software provided a user friendly and efficient way to develop and complete the rubrics. Overall, the general evaluation rubric was easy to develop due to the resource assessment skill sets common to most librarians. The elements in sections A to C were quickly agreed upon and the information for them found. However, elements in sections D and E were much more difficult to define and locate. The reviewers needed to contact customer service providers in some cases to receive clarification on grading standards. The second rubric, clinical summaries evaluation, was more challenging to create. Determining the clinical topics to search required some research and discussion. Some measures included in the rubric were ultimately omitted from the final assessment. For instance, because the number of clicks was difficult to standardize between screeners, the number of clicks to find the treatment section and to find the evidence grading were omitted from the findings.
Limitations of the study lie in what was absent from the review. Understanding the factors affecting SCAMeL institutions' selection of EBM POC tools could have been valuable. Factors such as product cost, budget, and user input might have impacted the selection process more than attributes of the tools themselves. Another omission from this review was free EBM tools. If assessed, these tools might have changed the findings of the study. Lastly, one element that should have been included in the tool evaluation was an analysis of potential bias based on editor affiliations. Only the affiliations of the evidence summary authors were considered.
The span of time, eighteen months, covered by this review was both a strength and a limitation. Researchers had an opportunity to observe how quickly EBM POC tools can change and to see trends in the SCAMeL collections and in tool development. Due to product changes during this time frame, both the literature review and information about the tools were updated multiple times.
Strengths of this review included systematic evaluation of tools' characteristics and quality control processes, as well as in-depth assessment of grading systems. By focusing on those elements that affect the reliability and value of the synthesized evidence, the reviewers highlighted the need for EBM POC tools to be more transparent.
The rubrics developed in this study can be used by other librarians to evaluate EBM tools. Because the rubrics themselves do not weigh one element over another, the importance of each element could be adapted according to the priorities and needs of specific libraries. As products change often, these evaluations need to be conducted on a regular basis, to discover the similarities and differences in tools and to make more informed subscription decisions. For example, in this case, the reviewers' recommendation to the library director was to drop at least one of the currently subscribed tools due to redundancy.
One further area of investigation is to determine how clinicians and students perceive the grading of evidence. How important is it to EBM POC users that the evidence is graded, and how well do they understand the grading scales? If the meaning of the grade is unclear, users may be hesitant to apply the recommendations. What are the mental processes by which users apply the evidence found in EBM POC tools? Focus groups may be helpful to determine users' preferences of tools and impressions of evidence grading.
EBM POC tools remain an important resource for health care practitioners, librarians, and patients. Due to the dynamic nature of these tools, evaluation needs to be continuous. Many tools have similar audiences and content. The grading of the evidence, however, varies among resources and requires standards for evaluation. Tool creators need to more openly communicate guidelines for authorship, reviewers, methods of evidence collection, and grading systems. Medical librarians play an important role in evaluating the quality of POC tools and can develop assessment tools, such as rubrics, to aid in this evaluation.
Christina Seeger, AHIP, previously at Texas A&M's MSL, now the pharmacy librarian at Feik School of Pharmacy, University of the Incarnate Word, assisted with this research project.
IRPThis article has been approved for the Medical Library Association's Independent Reading Program <http://www.mlanet.org/education/irp/>.