PubMed is one of the most important information sources for biomedical researchers. It supports an efficient processing of keyword and constraint queries. However, finding relevant articles from PubMed is still challenging because it is hard to express the user’s specific intention in the given query interface, and a keyword query typically retrieves a large number of results. For example, the keyword “breast cancer” returns more than two hundred thousand articles. Adding a few more constraints could narrow down the search results but is still likely to return more results that the user can easily handle. The user can sort the results according to publication date, author’s first or last name, or journal name, but sorting them by some notion of relevance is hard.
To improve the search quality on PubMed, researchers have studied querying methodologies for PubMed, such as how to use controlled vocabulary, MeSH terms, or background knowledge to formulate proper PubMed queries [1
]. Re-organizing the search results using ontologies or clustering techniques has been explored to provide better presentation of the results to the users [3
]. Text mining researchers have also tried to compute the global importance of articles using the citation information and have applied it to rank the results as done in Google [4
]. However, users’ specific intentions are typically widely varied even with the same keyword query. For example, with a query “breast cancer”, one user may be interested in finding genetic-study related papers while another user may want to find the latest cancer treatments. Thus ranking according to the global importance often does not meet the users’ specific information needs.
Researchers have also applied machine learning techniques to find relevant articles by ranking the articles according to the learned relevance function [8
]. However, the process of learning and ranking is usually done offline without being integrated with the PubMed’s keyword queries, and the users have to provide a large amount of training articles to get a reasonable learning accuracy.
Finally, relevance feedback, a well established technique in IR to improve retrieval performance [10
], has been applied on PubMed (e.g., MiSearch, a recent relevance feedback system for PubMed [12
]). However, existing relevance feedback systems use classification methods and thus are limited to two level relevance judgements (relevant or not).
This paper proposes a novel multi-level relevance feedback system for PubMed, called RefMed, which supports both ad-hoc keyword queries and a multi-level relevance feedback in real time on PubMed.
Figure shows the search process in RefMed. RefMed first accepts a keyword query (Step 1) and returns initial results (Step 2) as done in PubMed. While browsing the resulting documents, the user makes relevance judgments on some of them (Step 3). The number of relevance levels is set to three as default but can be adjusted depending on the user’s preference. Once the user “pushes the feedback,” the system induces a relevance function from the feedback using the RankSVM [13
] and returns top-k results ranked according to the function (Step 4). The user can repeat this process until she receives satisfying results. This process of learning and ranking is done in real time.
To the best of our knowledge, RefMed is the first “multi-level” relevance feedback system for PubMed. The new technical contributions of RefMed are as follows.
• RefMed supports the multi-level relevance feedback by using the RankSVM as the learning method, and thus it achieves higher accuracy with less feedback. Traditional relevance feedback systems use classification methods for learning (e.g., SVM, Bayesian learning) and thus are limited to two levels of relevance judgments (i.e., relevant or not). RankSVM is one of the most actively researched algorithms for learning ranking functions in the machine learning community and is regarded as the most accurate methodology when the size of training data is relatively small [13
]. In a real time relevance feedback system such as RefMed, the amount of user feedback, i.e., training data, is typically small. Thus, we adopted the RankSVM as the learning method.
• RefMed “tightly” integrates the RankSVM into a relational database management system (RDBMS) to support keyword queries and relevance feedback in the same framework and to minimize the response time. Specifically, we develop and integrate new SQL expressions for learning and predicting ranking into DBMS. The tight coupling of RankSVM and DBMS improves the processing time substantially by running the RankSVM directly on the data tables instead of files. The new SQL expressions also facilitates the application development process by running the rank learning and predicting operations within SQL.
• An efficient parameter selection method for RankSVM is proposed, which tune the parameter without performing validation. Validation is a necessary process in learning with RankSVM in order to tune the soft margin parameter C. However, it is not feasible to perform the validation in RefMed, as no validation set is given during the search process. By the parameter selection method, RefMed estimates the best parameter to achieve a high learning accuracy without performing validation.
Methods section overviews the RankSVM and presents the integration of RankSVM within SQL and our parameter selection method. Result section demonstrates RefMed, and reports experiment results. We report (1) the learning accuracy of RankSVM against Rocchio with different amounts of feedback and relevance levels, (2) the query processing time of the tight coupling against a loose coupling, and (3) the accuracy of our parameter selection method against the cross validation and other parameter selection methods.