|Home | About | Journals | Submit | Contact Us | Français|
William Hsu contributed to the initial design and implementation of the web-based interface and gateway. He is the primary author of the MEDINFO 2007 submission.
Sameer Antani Ph.D. is the lead researcher on the Content-Based Image Retrieval project at the National Library of Medicine, NIH. He has been directly involved with the reported research with several components developed by him, or under his direction or mentorship.
L. Rodney Long M.A. jointly participates with Dr. Antani on several aspects of the project which include but are not limited to image segmentation, image transmission, and performance evaluation.
Leif Neve is a developer on the project and directly contributes to the development, maintenance, and enhancements in SPIRS.
George R. Thoma Ph.D. is the Chief of the Communications Engineering Branch and jointly participates on the project with Mr. Long and Dr. Antani. He contributes to the reasearch and provides guidance on biomedical imaging projects at the National Library of Medicine, NIH.
With the increasing use of images in disease research, education, and clinical medicine, the need for methods that effectively archive, query, and retrieve these images by their content is underscored. This paper describes the implementation of a Web-based retrieval system called SPIRS (Spine Pathology & Image Retrieval System), which permits exploration of a large biomedical database of digitized spine x-ray images and data from a national health survey using a combination of visual and textual queries.
SPIRS is a generalizable framework that consists of four components: a client applet, a gateway, an indexing and retrieval system, and a database of images and associated text data. The prototype system is demonstrated using text and imaging data collected as part of the second U.S. National Health and Nutrition Examination Survey (NHANES II). Users search the image data by providing a sketch of the vertebral outline or selecting an example vertebral image and some relevant text parameters. Pertinent pathology on the image/sketch can be annotated and weighted to indicate importance.
During the course of development, we explored different algorithms to perform functions such as segmentation, indexing, and retrieval. Each algorithm was tested individually and then implemented as part of SPIRS. To evaluate the overall system, we first tested the system’s ability to return similar vertebral shapes from the database given a query shape. Initial evaluations using visual queries only (no text) have shown that the system achieves up to 68% accuracy in finding images in the database that exhibit similar abnormality type and severity. Relevance feedback mechanisms have been shown to increase accuracy by an additional 22% after three iterations. While we primarily demonstrate this system in the context of retrieving vertebral shape, our framework has also been adapted to search a collection of 100,000 uterine cervix images to study the progression of cervical cancer.
SPIRS is automated, easily accessible, and integratable with other complementary information retrieval systems. The system supports the ability for users to intuitively query large amounts of imaging data by providing visual examples and text keywords and has beneficial implications in the areas of research, education, and patient care.
Medical imaging has become an important tool in not only objectively documenting patient presentation and clinical findings, but also understanding and managing various diseases. Image data provides tangible visual evidence of disease manifestation. For example, in spine X-rays, shape deformations of the vertebral body (e.g., protrusions of the bone) are often indicative of osteophytes, which commonly appear in conditions such as osteoarthritis. As medical imaging becomes more widely used in patient care and generates larger amounts of data, tools that allow users to retrieve and manipulate such data effectively are becoming essential. Hospitals and medical centers have been adopting technology such as Picture Archiving and Communication Systems (PACS), Hospital Information Systems (HIS), and Radiological Information Systems (RIS) to assist in the digital collection, organization, and storage of patient data. The goal of these systems is to make patient data more accessible; in reality, the amount of data that is entered and stored in these systems have created a new challenge for effective information indexing and retrieval. Historically, PACS have limited users to query by certain keywords (e.g., unique patient identifier, fields in the image header). However, these keywords often may not capture valuable visual characteristics contained within the image, thereby limiting the power of posed clinical queries and reducing the overall usefulness of the data. As imaging data becomes more abundant, technologies are needed to extract and structure the content of medical images and enable more efficient image retrieval for patient care, education, and research.
Over the past two decades, content-based image retrieval (CBIR) systems have been researched to address the problem of indexing and retrieval of visual data in a variety of domains . Rather than limiting queries to textual keywords, CBIR allows the user to provide a visual query consisting of a sketch, an image, a color characteristic, texture sample, or a specified region of interest (ROI), which is then used to find other similar images in a database. Here, similarity is defined as minimizing the difference between features of interest in the query image and indexed images in a database. The problem of retrieving information based on image content has been researched by various groups since the early 1990’s resulting in the development of tools such as Query by Image Content (QBIC) , Virage , and Blobworld . CBIR in medicine has been an active area of research , but only a small number of proposed systems have been demonstrated in the clinical environment. While large biomedical image collections exist, such as the National Cancer Imaging Archive (NCIA) or the Lung Imaging Database Consortium (LIDC) created under the aegis of the Cancer Imaging Program1 at the U.S. National Cancer Institute (NCI), these efforts have concentrated on data collection and transmission but have left development of applications to the research community. Early efforts in medical CBIR include Comparison Algorithm for Navigating Digital Imaging Databases (CANDID) , which is based on a technique similar to the n-gram method for searching free-text documents: a signature is computed that represents the content of the image in an abstract sense using texture features. A preliminary evaluation showed that the technique used in CANDID correctly classified computed tomography (CT) lung images of patients with scleroderma and vasculitis. I2C  integrates tools for defining image analysis routines based on specific image classes; some of the algorithms are interactive, while others are automated. This system has been implemented in a mini-PACS system and deployed on the web. ASSERT  requires a physician to delineate the ROI containing the pathology on which the features are indexed. The system has been evaluated clinically on high-resolution CT lung images. More recently, Image Retrieval for Medical Applications (IRMA)  has been developing a retrieval system that supports semantic and formalized queries. For example, IRMA would be able to automatically determine the staging of a patient’s therapy or retrieve a set of images with similar diagnostic findings. While the IRMA system is a global feature system, which means it extracts a single feature vector from the entire image, efforts are underway to develop a multilevel distributed CBIR system  that supports both global and local features. These developments are pioneering efforts in CBIR and have helped demonstrate its potential use in the healthcare environment. However, widespread adoption and application of CBIR in the clinical environment have yet to be achieved. This lack of utilization is attributed to several limitations: (i) the difficulty of integrating current implementations with existing healthcare systems ; (ii) the amount of human expert intervention required for image indexing (e.g., I2C and ASSERT); and (iii) the lack of evaluation using a large set of text and imaging data. CBIR has the potential of improving many areas in healthcare, including research (e.g., finding the prevalence of a pathological feature in a large survey collection), education (e.g., creation of teaching files), and clinical decision support (e.g., diagnosing a patient given a set of image findings).
The goal of our work is to develop a retrieval system that implements recent developments in feature representation, efficient indexing, and similarity matching; supports whole and partial shape matching, which enables a wide range of meaningful queries to be posed; and utilizes a distributed framework, which is customizable to the needs and constraints of healthcare environments. The result, Spine Pathology & Image Retrieval System (SPIRS)2, provides a Web-based interface for performing image retrieval on a database of digitized spine x-rays using the morphological shape of the vertebral body. A query editor enables users to pose queries by sketching a unique shape or selecting or modifying an existing shape from the database. Additional text fields enable users to supplement visual queries with other relevant data (e.g., anthropometric data, quantitative imaging parameters, patient demographics). These hybrid text-image queries may be annotated with pertinent pathologies by selecting and weighting local features to indicate importance. Results appear in a customizable window that displays the top matching results and related patient data. SPIRS addresses current limitations of CBIR implementations by: (i) utilizing open standards to communicate among components, which can be extended to support data encryption to meet privacy regulations; (ii) implementing algorithms that automate shape extraction and representation; (iii) utilizing feature indexing methods for efficient retrieval; (iv) combining text and image feature queries for hybrid queries; and (v) performing evaluation of a variety of whole and partial shape retrieval algorithms using a large biomedical dataset. Much work has been done in the past on visual querying paradigms, as reviewed in . SPIRS utilizes the query-by-example paradigm, in which users provide an example shape or image to the system as a method of finding similar images. In addition, it combines visual and text queries to provide users with greater flexibility in retrieving relevant results. SPIRS is a working proof-of-concept that demonstrates the capability of accommodating large amounts of text and imaging data that is expected in the modern research and healthcare environments.
The system’s distributed architecture, shown in Figure 1, consists of four components: (i) the client applet, which provides a front-end for users to pose queries and interact with results; (ii) a gateway that acts as an mediator between client and server-side components; (iii) the indexing and retrieval server, which performs the feature representation and similarity matching; and (iv) the databases containing images and associated text data. The following sections describe the process by which imaging features are extracted and indexed and how the user interacts with SPIRS to perform queries and interact with the results.
At the U.S. National Library of Medicine (NLM), the focus of CBIR research has been to develop systems capable of performing a range of queries on large medical multimedia databases comprising various biomedical images and patient health data. One such database contains digitized spine x-rays and associated metadata from a nationwide survey, the National Health and Nutrition Examination Survey (NHANES)3, conducted regularly by the National Center for Health Statistics (NCHS) at the Centers for Disease Control and Precention (CDC) in the United States. The goals of NHANES include estimating prevalence of selected diseases, monitoring disease trends, and studying the relationship between nutrition and health. The Lister Hill National Center for Biomedical Communications, an intramural research and development division of the NLM, maintains data from the second survey, NHANES II, which was collected between 1976 and 1980 and featured over 20,000 participants. Each participant’s record includes over 2,000 textual data points such as health questionnaire answers, anthropometric information, and results from a physical exam (Table I). This textual data is stored in a MySQL relational database. Supplementing the textual data is a collection of 17,000 cervical and lumbar spine x-rays that were taken from patients aged 25 – 74. These images were originally on film and subsequently digitized using a 146 dpi scanner resulting in 140 Gigabytes (GB) of data . The collection is considered valuable to radiologists, bone morphometrists and researchers interested in osteoarthritis and musculoskeletal diseases, and medical educators. A panel of domain experts reviewed a sample of the data and identified 23 key biomedical features exhibited in the x-rays.
Of these, anterior osteophytes (AO), subluxation in the cervical spine and spondylolisthesis in the lumbar spine, and disc space narrowing (DSN) were determined to be frequently occurring and reliably detectable using vertebral shape characteristics. However, consistently identifying images that exhibit particular types and severities of these pathologies is extremely tedious for such a large collection. SPIRS and related tools , developed to analyze the images and populate the database, automate many tasks and enable retrieval of the images using sketches of the vertebral boundary with the desired feature. While the intent of the NHANES dataset is not clinical decision support, the combination of health survey and imaging data is useful in retrieving examples of abnormal vertebral images for research and education. For instance, SPIRS can be used to determine what types of features (e.g., protrusion on the anterior edge of the cervical vertebra) are consistently associated with a particular symptom (e.g., neck discomfort).
This paper focuses on identifying images in the NHANES II dataset that contains AOs using SPIRS. AOs typically appear on the two anterior “corners” in the sagittal view as bony protrusions, shown in Figure 2. Depending on their appearance, AOs may be classified into one of two groups and assigned a grade of severity. The Macnab classification assigns an osteophyte as either a claw, which occurs when a spur of triangular shape rises from the vertebral rim and curves toward the adjacent disk, or a traction, which protrudes horizontally and does not extend across the inter-vertebral disc space. The different grades of severity are: slight, moderate, and severe, depending on the degree of narrowing as depicted in Table II [14–16]. When the user provides SPIRS with a query shape, it is used to retrieve other similar shapes in the NHANES database along with associated text data and Macnab classification.
The SPIRS framework is also being used to query an image database of cervicographic images (cervigrams) created by the National Cancer Institute (NCI) and the National Library of Medicine (NLM) for the study of uterine cervix cancer . The database contains approximately 100,000 cervigrams taken as part of the Guanacaste and ALTS projects that study the natural history of human papillomavirus infection and cervical neoplasia. In addition to cervigrams, correlated clinical, cytologic, and molecular information are also available. Unlike NHANES II, the intent of this dataset is to be used in clinical practice as a method for identifying whether patients have precursors to cervical cancer given the appearance of their cervigram.
In our continuing research, we have explored and implemented various algorithms to extract, index, and retrieve shapes of vertebral bodies from x-ray images. Segmentation is the process of characterizing features (e.g., shape, color, texture) that can be represented in a unique form that is useful for CBIR. In spine x-ray images, the vertebral shape is an image characteristic of high significance. We have considered the combination of manual and semi-automated algorithms including active contour segmentation, LiveWire, active shape models (ASMs), and hierarchical segmentation  to obtain the boundaries of the vertebral body. Here, we examine two types of hierarchical segmentation algorithms: Zamora hierarchical segmentation , which is based on an ASM but captures the shape of bone spurs by using a combination of Generalized Hough Transform (GHT), and a deformable model (DM), and Howe hierarchical segmentation , which combines the GHT with two active appearance model (AAM) steps applied initially to the entire set of vertebra and then to each individual vertebra. The performance of these algorithms is presented in Section 4.
Automated segmentation methods are necessary to make CBIR practical because the time required for manual segmentation would be prohibitive for large datasets such as NHANES II. However, their outputs need to be comparable to those done manually by an expert. To this end, an evaluation tool has been developed called PathVa (Pathology Validation) , which is used to collect ground truth data. Content experts log into the imaging database, review images, mark the pathology data, and validate or create boundary segmentations. The tool has also been useful in finding other pathologies inside the NHANES II dataset, such as spinal stenosis, which was not originally studied from the dataset.
While the aforementioned algorithms are specific to extracting shape boundaries, different image characterization methods may be used to extract other features such as color and texture. For example, in uterine cervix images, color tone and surface texture are two major features used by physicians to differentiate tissue types and to identify the developing stage of cervical neoplasia. Therefore, in addition to defining the boundaries of different regions, these images are characterized using a combination of color moments descriptor, Discrete Wavelet Transform, and lesion size.
Once segmented, the vertebral shape boundaries are then treated as closed polygons and represented in a variety of forms, such as Polygon Approximation, Fourier Descriptors, or geometric shape properties . Traditionally, these representations are stored as a feature vector and compared individually with the query. However, for large image archives, this linear comparison approach would be impractical. SPIRS therefore uses a coordinate tree to efficiently index shapes and optimize retrieval time. Embedded with the indexing process are appropriate distance measures, which identify matching shapes . Several similarity measures such as Procrustes distance, Fourier descriptors, shape features, invariant moments, polygon approximation for tangent space matching, and token evaluation in multi-scale space have been implemented for matching whole spine x-ray shapes . However, these algorithms have performed with only 56% accuracy in retrieving pathologically relevant vertebrae images . To improve retrieval results, a partial shape matching algorithm has been explored with Procrustes distance and Fourier descriptors; this algorithm is described further in Section 3.2.
The client, shown in Figure 3, is a Java applet that runs in a Web-browser and provides the user with the necessary functionality to query for particular shapes of interest and interact with the results. The applet has been developed with the following considerations: (i) it is a thin client that provides the look-and-feel of a standalone application but uses minimal computing resources; (ii) it does not require installation of files onto the local drive; (iii) is easily modifiable without affecting core-CBIR functionality; and (iv) it is platform independent. The client interface is comprised of the following components, described below.
SPIRS utilizes standard Web communication interfaces to enable interaction with its components. The gateway is a Java servlet (e.g., Apache Tomcat) that acts as a mediator between clients, which could be applets or remote applications, and server-side components. It manages multiple simultaneous connections (users) as a separate session and queues requests to the core CBIR engine. The servlet automatically generates the appropriate statements in structured query language (SQL) query syntax for interacting with a MySQL database. It also is responsible for packaging and transmitting responses back to the requesting client. Other types of databases may be supported by customizing the gateway configuration file to support the different database schema and query syntax. In addition, the core architecture of the gateway makes interaction with other geographically distributed CBIR systems possible: recent work has combined the IRMA 4 system, which is a system located in Germany for CBIR of medical images using overall (global) image intensity features with the SPIRS shape matching engine [10, 25]. While it appears as if the IRMA system provides shape query support for subtle localized pathology, in actuality, its interface is using the SPIRS server located at the NIH in the USA for computing shape-similarity of spine x-rays. Queries executed within IRMA are routed to SPIRS via the gateway. Information is transmitted between systems using XML (extensible markup language). The entire transaction is divided into three primary events—querystatus, query, and queryresult. Each element in the XML file is designed for a particular event. The <querystatus> element is used to determine whether a desired service is available and to obtain a list of currently available services. The <query> element contains information about shape queries such as the vertebra contour, partial shape indices and weights, maximum number of results desired, and range of similarity result scores. SPIRS responds with the <queryresult> element populated with matching image-vertebra tuples and similarity scores.
Although much CBIR research has focused on overcoming technical challenges such as developing accurate and efficient retrieval algorithms, in order for widespread CBIR adoption and use, systems need to provide users with the ability to exploit the capabilities of these algorithms in managing large biomedical multimedia databases. With this goal, we believe that SPIRS allows users to pose meaningful queries that have both clinical and research applicability. The following sections describe how users pose queries and interact with the results.
Figure 4 and Figure 5 illustrate how users interact with the system to pose a query. In this particular example, a researcher is interested in finding any association as to whether male participants in the survey data who exhibit a particular protrusion on the anterior of their cervical vertebra complain of neck pain. The user would first utilize the Query Menu to find the initial shape of interest, whether by using a random image, selecting a shape from a specific participant’s image, or specifying a specific vertebra type and number (e.g., C3). Once a query shape has been selected, the user may modify the shape to accentuate a particular feature (e.g., move the points to create the desired protrusion) in the Query Editor (Figure 4a) and demarcate a subset of the shape to emphasize in the query using the partial shape functions (Figure 4b). The user may also input additional textual and numerical data to further constrain the returned results such as limiting the results to only male patients (Figure 4c). The user also selects the desired parameters to be returned. Once the query is executed, the results are displayed as cropped images depicting the shape and sorted by similarity where smaller values indicate that the shape is more similar to the query shape (Figure 4d). Selecting a particular shape returns the entire patient image (Figure 5a) from which the cropped image was extracted. If the user selected to have patient data returned with the query, a separate tabbed pane (Figure 5b) provides that information (e.g., whether the patient experiences continual neck pain). The user may iteratively pose queries by selecting a returned shape and using that shape as the basis of a new query.
A problem with many current CBIR approaches for medical images is that they operate on the image or region of interest (ROI) as a whole. Whole shape matching methods have been demonstrated to be more effective in identifying shapes with gross similarity because these algorithms operate over the entire shape. They are ineffective, however, in matching subtle shape characteristics, which may have critical pathology. It has been observed that pathologies of interest on a vertebral outline are often localized along a short interval on the boundary. For example, osteophytes are only expressed along the “corners” on the vertebral boundary as seen in the sagittal view. Partial shape matching (PSM) has been implemented in SPIRS allowing users to sketch or identify only the local interval of interest on the vertebral boundary. For instance, consider a user interested in identifying patients in the NHANES II data with a claw AO. In the query editor, the user would highlight the subset of points that comprise the spur, have the capability of altering its appearance by dragging the shape points, weight these points greater than the other points in the shape, and execute the PSM query. PSM has been demonstrated as an effective method to classify the severity of a pathology depicted in the query shape .
Figure 6 illustrates how the system handles a query and how query execution integrates with the overall CBIR system. When the user formulates a hybrid text-image query, the query is separated into its visual and textual components. Relevant features and annotations are extracted from the visual component and matched against the indexed features using specific similarity computation methods. Parameters from the textual component are executed against the database containing survey and patient data. The results of these individual searches are then combined before the results are presented to the user. The set of returned results consists of the intersection of the set of spine images with matching vertebrae shape profiles and the set of matching responses to text query components. For example, a user who is researching the correlation between certain shape features and the degenerative spine disease spondylosis may be interested in the intersection of results between a visual query depicting a traction spur and survey participants who have been diagnosed with spondylosis.
To ensure the accuracy and utility of SPIRS in the research environment, we have evaluated the system in two parts: first, each component is evaluated individually, and then the components are evaluated as a complete framework. While many of the results for the individual components have been reported in previous publications, we summarize their results here. We then present results of the overall system in retrieving similar vertebral shapes and describe preliminary results showing how the SPIRS framework can be used to search uterine cervix images.
Accurate segmentations are critical to the system’s ability to capture subtle features that may be used for retrieval. In this work, two algorithms were explored: Zamora and Howe. The Zamora algorithm was tested using a random subset of 100 cervical and 100 spine images from the NHANES dataset. Success is defined as whether the converged solution is near (20 pixels for cervical spine, 50 pixels for lumbar spine) the actual boundary. While this measure is subjective and relies on the visual judgment of a domain expert, it is still useful for gauging the general abilities of the algorithm. It achieved 75% success for cervical spine and up to 49% for lumbar spine. For the Howe algorithm, the AAM was trained using a leave-one-out strategy (train on 99 images, test on 1 image) and achieved 65% accuracy for cervical and 67% for lumbar vertebra.
The performance of the indexing scheme using kD-tree was compared with a metric tree. A sample of 2,812 shapes was divided into groups of random sizes (434, 902, 1,654, and 2,812). Each set was indexed in the original shape space using a metric tree and optimal embedding into a kD-tree. After indexing, every shape was used as queries and the k-nearest neighbor vertebral images were retrieved using the Euclidean shape distance. Retrieval using a kD-tree consistently outperformed that of the metric tree, but both approaches were sublinear in complexity .
To evaluate the overall performance of the system, three board certified radiologists were asked to (i) review and label a subset of 200 vertebrae for anterior osteophytes using the classifications and degree of severity discussed previously, (ii) generate 24 queries using exemplar cases from this set, and (iii) on (unranked) set of vertebrae that may be considered “similar” on the particular expressed pathology on each query vertebral outline. The evaluation studied the top 10 vertebral shapes returned by SPIRS and compared their pathology classification to the query shape classification. The retrieval algorithms achieved 68% relevance (precision and recall) when querying for specific osteophyte type (claw or traction) or severity (slight, moderate, severe) using PSM based on Procrustes distance method. The shortcomings in the performance have been linked: (i) to erroneous determination of the query semantics and (ii) to limitations of the shape matching algorithm. A corner-guided PSM using dynamic programming in initial experiments has shown a precision of 100% for the top match that gradually drops down to 85% as larger number of matches are considered . In addition, SPIRS allows users to select a matching result as the basis of a new query. Traditionally, user interactivity has helped in minimizing similar problems with text retrieval, and user feedback has often been analyzed and employed to improve retrieval relevance. By allowing users the option to iteratively query and refine results, SPIRS implements a basic form of relevance feedback, which will be enhanced with a novel advanced weighted hierarchical feedback method using short-term memory  and other approaches as they are migrated from laboratory prototype routines. To quantify the improvements of using relevance feedback , eight queries were selected to have a user perform relevance feedback on a subset of 2,000 shapes from the NHANES II dataset. The retrieval accuracy (number of relevant retrieved images over the total number of retrieved images) was calculated for all eight selected queries before and after each feedback refinement for the first three feedback iterations. An additional 22% performance improvement was observed through use of the linear updating approach after three iterations.
Work is also ongoing to evaluate the feasibility of using the SPIRS framework with the uterine cervix images. A preliminary study involves retrieving images in the database that contain the same region type as the query image. The SPIRS implementation was modified to search for similarity based on color, texture, and lesion size rather than shape. The user interface was also customized to incorporate additional tools for working with cervigrams. A set of 120 images with 422 expert-marked regions (e.g., acetowhite, blood, mucus) was presented to the system, and the fraction of the top five returned images that contained the same region type was computed. The system was able to differentiate between three types of tissue (acetowhite, blood, columnar epithelium) with up to 64% accuracy . While additional improvements are needed to address the algorithm’s ability to distinguish different tissues that have similar appearances, the evaluation demonstrates the feasibility of using the SPIRS framework in this domain.
Tools for image management and CBIR systems that not only minimize the semantic gap  but also are sensitive to depicted pathology are increasingly necessary to interact with the growing volume of biomedical imaging data. In CBIR systems, images are characterized either using their content, which is captured through measurements of low level features such as color, texture, shape (edges), or by their location relative to some identifiable entity on the image. These image descriptors are quantified using a variety of particular techniques that attempt to ensure some basic properties of similarity:
The descriptors can be applied at different scales that can be broadly categorized as global or local descriptors. Global descriptors are those that attempt to capture the visual content characteristic of the entire image on the assumption that sufficient separation can be achieved between different entities in the image database on those descriptors alone. Local descriptors, on the other hand, can be applied at various other image “levels” or “regions of interest” and attempt to not only separate images on the computed characteristic, but also support localized queries depending on the level of detail desired. Historically, global features have been used to characterize images in systems with a heterogeneous collection of images such as IRMA. However, when images result from a study that focuses on a particular aspect it is likely that the resulting database will be homogeneous. For example, the NHANES II collection has two kinds of images (cervical and lumbar spine) that are very similar within each group and require localized descriptors at various levels of detail to distinguish amongst them. Another example is ASSERT, which showed that the use of local features significantly improved the retrieval performance of HRCT images with pathologies similar to the query image . This distinction between global and local descriptors is also applied to the vertebral outlines in the case of spine images and the AO pathology. Whole shape feature extraction algorithms are a type of global descriptors at this image scale and partial shape feature extraction algorithms are a type of local descriptors attempting to capture subtle pathology expressed along the vertebral shape.
Traditionally, computer-aided diagnosis (CAD) systems classify abnormalities in an image and assist in determining their degree of severity. This aspect is beginning to appear in CBIR systems as segmented (extracted) features are used for pathology classification on the images in the database as well as the query image. Unlike traditional text databases, however, an image database is often a collection of images stored on a file server. Relevant fields in text traditional text databases hold pointers to these images. This configuration, commonly found in PACS, RIS, and HIS, is challenging for use with CBIR applications. One reason is that the image “indices” are stored as long strings of numbers called feature vectors (segmented features and feature measurements). These “indices” are entities on which further computation is necessary to determine the degree of similarity between a query image and a candidate on the database. Several approaches have been proposed for indexing multidimensional vectors and include coordinate trees and metric spaces . Our approach has been to use coordinate trees in SPIRS while we continue to evaluate other approaches.
In spite of the acknowledged importance of CBIR, several shortcomings in current approaches have prevented their widespread acceptance into medical research, practice, and education. We believe that a biomedical CBIR system should be easily accessible, extensible, and capable of supporting a rich set of segmentation, validation, indexing, query, retrieval, and visualization methods developed using open software and standards. With the SPIRS framework, we have implemented a system that incorporates these qualities to demonstrate how such system would perform on a large biomedical dataset such as NHANES II and the uterine cervix image database. In order to support unique requirements of different biomedical images it is necessary to either: (i) provide the capability to combine this system with others (possibly geographically distant) that have complementary features; or (ii) make the framework modular such that other databases and retrieval algorithms may be integrated. The SPIRS framework attempts to address these requirements through its client-server design, customizability of individual components (e.g., gateway, segmentation algorithms), use of a XML-based service protocol, and implementation in Java. The applet makes use of the Java Image I/O programming interface. While images in the NHANES II database are stored as JPEG (Joint Photographic Experts Group) format, SPIRS is capable of handling other common medical imaging formats such as DICOM (Digital Imaging and Communications in Medicine) provided the appropriate file reader packages. While most components in the framework require few changes to support different domains, the user interface needs the most customization to best suit a particular context. For example, in spine retrieval, the interface should be geared towards drawing and viewing vertebral shape boundaries. However, for cervical images, the interface should provide tools for identifying various cervix regions and weighing which features (color, texture, size) affect the results. Despite the advances that SPIRS has made, CBIR systems have yet to make inroads into routine clinical, biomedical research, or educational use. To better understand the underlying reasons,  presents a gaps-based framework to evaluate CBIR systems based on various characteristics broadly grouped into areas such as content, feature, performance, and usability.  details the results of evaluating SPIRS using this framework.
This work describes the basic implementation of the SPIRS framework and how it is being used in applications such as vertebral shape and how it can be extended to support uterine cervix image retrieval. Plans for SPIRS include a more exhaustive performance evaluation on a substantially larger dataset marked by multiple experts. In addition, a usability study will be conducted for ease of use, quality of expected outcome, value of SPIRS capability of correlating text and image data. As evidenced by the segmentation results, fully automated methods are still insufficient to accurately capture vertebral shapes (particularly lumbar spine) and osteophytes. In addition, overcoming the subjective evaluations such as those done to gauge the success of segmentations is an ongoing research topic . While we believe the development of PathVa provides the evaluating domain expert with the tools to make the best determination of whether the segmentation is accurate, more objective quantitative measures are needed. Furthermore, using human experts judge segmentations or classify vertebral shapes introduces an element of potential bias. Currently, we address this issue by having a consensus opinion from at least two expert radiologists examining the vertebral shapes.
SPIRS is a Web-based distributed content-based image retrieval framework that supports hybrid visual and text queries, and may be applied to various applications in medicine. It implements novel shape representation and similarity matching embedded with an index tree that allows efficient retrieval. SPIRS aims to capture query semantics through support of advanced mechanisms like multiple partial shape matching and iterative querying that provides simple yet effective relevance feedback to the system. SPIRS is built using open standards and is simultaneously developed as a service, which enables its integration with other complementary information retrieval systems. Initial evaluation of the system has shown that the combination of partial shape matching and relevance feedback significantly improves the system’s ability to retrieve similar results. Since its original release in August, 2006, SPIRS has undergone several revisions that have added support for additional retrieval algorithms (i.e., Fourier descriptors) and an improved, cleaner user interface for posing visual and text queries. Future goals for the project include: (i) additional data collection, assimilation, and validation; (ii) system feature enhancements; (iii) improved retrieval quality by learning from user feedback; and (iv) improved user interaction and visualization. Extracting and validating vertebral shapes is an ongoing process that will (at regular intervals) add to the 7,200 shapes currently indexed. Planned feature enhancements include integration of a generalized shape segmentation toolbox, which is a currently a standalone application, incorporation of additional similarity algorithms, support for query of multiple vertebral shapes and disc space narrowing pathology, relevance feedback, and improved visualization algorithms.
This research was supported by the Intramural Research Program of the National Institutes of Health (NIH), National Library of Medicine (NLM), and Lister Hill National Center for Biomedical Communications (LHNCBC). The authors thank Dr. Alex Bui at the UCLA Medical Imaging Informatics Group for his valuable feedback.
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
4IRMA Web site: http://www.irma-project.org/index_en.php
Statements on Conflicts of Interest
No conflicts of interest noted.
William Hsu, Medical Imaging Informatics Group, University of California, Los Angeles.
Sameer Antani, National Library of Medicine, U.S. National Institutes of Health.
L. Rodney Long, National Library of Medicine, U.S. National Institutes of Health.
Leif Neve, National Library of Medicine, U.S. National Institutes of Health.
George R. Thoma, National Library of Medicine, U.S. National Institutes of Health.