The results of this analysis showed that there was a difference in easily accessed content between the subscription-based resource and the publicly available resource. UpToDate returned a minimum of 1 document for 11.6% more questions than NGC, whereas NGC retrieved more documents when content was found for a question. The results also demonstrated that the combination of the knowledge resources covered more than each knowledge resource separately, thus confirming what many librarians experience every day: more than one resource is often required to answer a given question ().
Although NGC returned at least 1 document in 11% fewer questions, the average number of documents found in NGC for all questions was larger than the average number of documents found in UpToDate. For example, multiple documents were retrieved from NGC on the subject of heart failure, but only one was retrieved from UpToDate. This concentration of guidelines on selected topics may be due to the absence of curriculum and editorial control at NGC due to its focus on providing content reflecting active areas of the work the producers of guidelines initiated. Dictating which diseases need guidelines is not in NGC's purview; NGC simply brings submitted guidelines together in one place. The process of determining whether their guidelines provide unique contributions rests with the guideline authors rather than NGC. The potentially associated problem of redundant guidelines may be analogous to the recognized problem of redundant clinical trials. Cumulative meta-analyses have shown that many clinical trails were unnecessary and were conducted after prior trials had already settled a research question [19
The current study observed peculiarities in the UpToDate search engine. In addition to the inability to accept two search terms, initial informal observations led to revising the protocol to substitute “cancer” for the keyword “neoplasms” due to problems with the UpToDate engine's failure to process MeSH terms that are not common text-words in articles. Similarly, the influence of disease category might be due to variation among disease categories in the mapping between the indexing terms of the two resources and the MeSH terms that were used in the searching protocol. For example, the question “What is the name of that skin disease on the feet that produces moist skin and little holes?” with the very vague assigned keyword of “skin disease” resulted in 0 UpToDate documents and 285 NGC documents. This problem, and the authors' inability to completely anticipate or identify all problematic terms, underscores the need for formal research of mapping [20
This analysis had several limitations. First, it was not meant to determine definitively whether or not the content could be found in the online resources, but rather could the content be found in an automated or nonexpert fashion. Assessing less robust searching methods as performed by nonexperts or by automated searches, such as those that might be incorporated into electronic medical records, would be valuable for future system development.
Second, no judgment regarding the quality of the content was attempted. This was a much more difficult question and concerns not only whether or not the required information was present (often what is required is debatable), but also whether or not it was in a form clinicians could use. The authors felt that the current results were an optimistic estimation, underscoring the need to improve existing resources. Third, this analysis was limited to only two of the many clinical knowledge resources currently available and to questions generated by primary care clinicians. It would be reasonable to presume that alternative knowledge resources or a pool of questions from other clinical specialties would have yielded different results.