Rationale and Objectives

Malignant pleural mesothelioma (MPM) is a neoplasm that grows circumferentially along the pleura. The tumor and concurrent pleural effusion may reduce lung function by restricting or preventing lung expansion. The purpose of this study was to provide objective evidence that pleurectomy/decortication (P/D) allows trapped lung to re-expand, quantify the re-expansion based on computed tomography (CT) scans, and investigate whether the expansion persists after surgery.

Materials and Methods

A database of 12 patients demonstrating unilateral MPM was collected. Each patient underwent a pre-surgical CT scan, surgical debulking by P/D, and two post-surgical CT scans (at one and four months). The lung volume was measured in each scan using an automated algorithm and compared for each patient across time.

Results

An increase in the ipsilateral post-surgical lung volume was observed for 10 of 12 patients (83%) one month after surgery. The median ipsilateral volume increase was 44% relative to the pre-surgical ipsilateral volume and 21% relative to the contralateral volume. A statistically significant change in ipsilateral lung volume was not observed between 1 -month and 4-month post-surgical scans, implying that the volume improvement persisted months after surgery.

Conclusions

Debulking of MPM with P/D substantially increased the ipsilateral lung volume relative to both the pre-surgical, ipsilateral volume and the contralateral lung volume. This improvement persisted months after surgery.

Rationale and Objectives

Studies that evaluate the lung-nodule-detection performance of radiologists or computerized methods depend on an initial inventory of the nodules within the thoracic images (the “truth”). The purpose of this study was to analyze (1) variability in the “truth” defined by different combinations of experienced thoracic radiologists and (2) variability in the performance of other experienced thoracic radiologists based on these definitions of “truth” in the context of lung nodule detection on computed tomography (CT) scans.

Materials and Methods

Twenty-five thoracic CT scans were reviewed by four thoracic radiologists, who independently marked lesions they considered to be nodules ≥ 3 mm in maximum diameter. Panel “truth” sets of nodules then were derived from the nodules marked by different combinations of two and three of these four radiologists. The nodule-detection performance of the other radiologists was evaluated based on these panel “truth” sets.

Results

The number of “true” nodules in the different panel “truth” sets ranged from 15–89 (mean: 49.8±25.6). The mean radiologist nodule-detection sensitivities across radiologists and panel “truth” sets for different panel “truth” conditions ranged from 51.0–83.2%; mean false-positive rates ranged from 0.33–1.39 per case.

Conclusion

Substantial variability exists across radiologists in the task of lung nodule identification in CT scans. The definition of “truth” on which lung nodule detection studies are based must be carefully considered, since even experienced thoracic radiologists may not perform well when measured against the “truth” established by other experienced thoracic radiologists.

RATIONALE AND OBJECTIVES: This article describes issues and methods that are specific to the measurement of change in tumor volume as measured from computed tomographic (CT) images and how these would relate to the establishment of CT tumor volumetrics as a biomarker of patient response to therapy. The primary focus is on the measurement of lung tumors, but the approach should be generalizable to other anatomic regions. MATERIALS AND METHODS: The first issues addressed are the various sources of bias and variance in the measurement of tumor volumes, which are discussed in the context of measurement variation and its impact on the early detection of response to therapy. RESULTS AND RESOURCES: Research that seeks to identify the magnitude of some of these sources of error is ongoing, and several of these efforts are described herein. In addition, several resources for these investigations are being made available through the National Institutes of Health-funded Reference Image Database to Evaluate Response to therapy in cancer project, and these are described as well. Other measures derived from CT image data that might be predictive of patient response are described briefly, as well as the additional issues that each of these metrics may encounter in real-life applications. CONCLUSIONS: The article concludes with a brief discussion of moving from the assessment of measurement variation to the steps necessary to establish the efficacy of a metric as a biomarker for response.

RATIONALE: Early detection of tumor response to therapy is a key goal. Finding measurement algorithms capable of early detection of tumor response could individualize therapy treatment as well as reduce the cost of bringing new drugs to market. On an individual basis, the urgency arises from the desire to prevent continued treatment of the patient with a high-cost and/or high-risk regimen with no demonstrated individual benefit and rapidly switch the patient to an alternative efficacious therapy for that patient. In the context of bringing new drugs to market, such algorithms could demonstrate efficacy in much smaller populations, which would allow phase 3 trials to achieve statistically significant decisions with fewer subjects in shorter trials. MATERIALS AND METHODS: This consensus-based article describes multiple, image modality-independent means to assess the relative performance of algorithms for measuring tumor change in response to therapy. In this setting, we describe specifically the example of measurement of tumor volume change from anatomic imaging as well as provide an overview of other promising generic analytic methods that can be used to assess change in heterogeneous tumors. To support assessment of the relative performance of algorithms for measuring small tumor change, data sources of truth are required. RESULTS: Very short interval clinical imaging examinations and phantom scans provide known truth for comparative evaluation of algorithms. CONCLUSIONS: For a given category of measurement methods, the algorithm that has the smallest measurement noise and least bias on average will perform best in early detection of true tumor change.

We describe methods and issues that are relevant to the measurement of change in tumor uptake of 18F-fluorodeoxyglucose (FDG) or other radiotracers, as measured from positron emission tomography/computed tomography (PET/CT) images, and how this would relate to the establishment of PET/CT tumor imaging as a biomarker of patient response to therapy. The primary focus is on the uptake of FDG by lung tumors, but the approach can be applied to diseases other than lung cancer and to tracers other than FDG. The first issue addressed is the sources of bias and variance in the measurement of tumor uptake of FDG, and where there are still gaps in our knowledge. These are discussed in the context of measurement variation and how these would relate to the early detection of response to therapy. Some of the research efforts currently underway to identify the magnitude of some of these sources of error are described. In addition, we describe resources for these investigations that are being made available through the Reference Image Database for the Evaluation of Response project. Measures derived from PET image data that might be predictive of patient response as well as the additional issues that each of these metrics may encounter are described briefly. The relationship between individual patient response to therapy and utility for multicenter trials is discussed. We conclude with a discussion of moving from assessing measurement variation to the steps necessary to establish the efficacy of PET/CT imaging as a biomarker for response.

Rationale and Objectives

Computer-aided diagnostic (CAD) systems fundamentally require the opinions of expert human observers to establish “truth” for algorithm development, training, and testing. The integrity of this “truth,” however, must be established before investigators commit to this “gold standard” as the basis for their research. The purpose of this study was to develop a quality assurance (QA) model as an integral component of the “truth” collection process concerning the location and spatial extent of lung nodules observed on computed tomography (CT) scans to be included in the Lung Image Database Consortium (LIDC) public database.

Materials and Methods

One hundred CT scans were interpreted by four radiologists through a two-phase process. For the first of these reads (the “blinded read phase”), radiologists independently identified and annotated lesions, assigning each to one of three categories: “nodule ≥ 3mm,” “nodule < 3mm,” or “non-nodule ≥ 3mm.” For the second read (the “unblinded read phase”), the same radiologists independently evaluated the same CT scans but with all of the annotations from the previously performed blinded reads presented; each radiologist could add marks, edit or delete their own marks, change the lesion category of their own marks, or leave their marks unchanged. The post-unblinded-read set of marks was grouped into discrete nodules and subjected to the QA process, which consisted of (1) identification of potential errors introduced during the complete image annotation process (such as two marks on what appears to be a single lesion or an incomplete nodule contour) and (2) correction of those errors. Seven categories of potential error were defined; any nodule with a mark that satisfied the criterion for one of these categories was referred to the radiologist who assigned that mark for either correction or confirmation that the mark was intentional.

Results

A total of 105 QA issues were identified across 45 (45.0%) of the 100 CT scans. Radiologist review resulted in modifications to 101 (96.2%) of these potential errors. Twenty-one lesions erroneously marked as lung nodules after the unblinded reads had this designation removed through the QA process.

Conclusion

The establishment of “truth” must incorporate a QA process to guarantee the integrity of the datasets that will provide the basis for the development, training, and testing of CAD systems.

Rationale and Objectives

To investigate the effects of choosing between different metrics in estimating the size of pulmonary nodules as a factor both of nodule characterization and of performance of computer aided detection systems, since the latters are always qualified with respect to a given size range of nodules.

Materials and Methods

This study used 265 whole-lung CT scans documented by the Lung Image Database Consortium using their protocol for nodule evaluation. Each inspected lesion was reviewed independently by four experienced radiologists who provided boundary markings for nodules larger than 3 mm. Four size metrics, based on the boundary markings, were considered: a uni-dimensional and two bi-dimensional measures on a single image slice and a volumetric measurement based on all the image slices. The radiologist boundaries were processed and those with four markings were analyzed to characterize the inter-radiologist variation, while those with at least one marking were used to examine the difference between the metrics.

Results

The processing of the annotations found 127 nodules marked by all of the four radiologists and an extended set of 518 nodules each having at least one observation with three-dimensional sizes ranging from 2.03 to 29.4 mm (average 7.05 mm, median 5.71 mm). A very high inter-observer variation was observed for all these metrics: 95% of estimated standard deviations were in the following ranges [0.49, 1.25], [0.67, 2.55], [0.78, 2.11], and [0.96, 2.69] for the three-dimensional, the uni-dimensional, and the two bi-dimensional size metrics respectively (in mm). Also a very large difference among the metrics was observed: 0.95 probability-coverage region widths for the volume estimation conditional on uni-dimensional, and the two bi-dimensional size measurements of 10mm were 7.32, 7.72, and 6.29 mm respectively.

Conclusions

The selection of data subsets for performance evaluation is highly impacted by the size metric choice. The LIDC plans to include a single size measure for each nodule in its database. This metric is not intended as a gold standard for nodule size; rather, it is intended to facilitate the selection of unique repeatable size limited nodule subsets.

Rationale and Objectives

The purpose of this study was to analyze the variability of experienced thoracic radiologists in the identification of lung nodules on CT scans and thereby to investigate variability in the establishment of the “truth” against which nodule-based studies are measured.

Materials and Methods

Thirty CT scans were reviewed twice by four thoracic radiologists through a two-phase image annotation process. During the initial “blinded read” phase, radiologists independently marked lesions they identified as “nodule ≥ 3mm (diameter),” “nodule < 3mm,” or “non-nodule ≥ 3mm.” During the subsequent “unblinded read” phase, the blinded read results of all radiologists were revealed to each of the four radiologists, who then independently reviewed their marks along with the anonymous marks of their colleagues; a radiologist’s own marks then could be deleted, added, or left unchanged. This approach was developed to identify, as completely as possible, all nodules in a scan without requiring forced consensus.

Results

After the initial blinded read phase, a total of 71 lesions received “nodule ≥ 3mm” marks from at least one radiologist; however, all four radiologists assigned such marks to only 24 (33.8%) of these lesions. Following the unblinded reads, a total of 59 lesions were marked as “nodule ≥ 3 mm” by at least one radiologist. 27 (45.8%) of these lesions received such marks from all four radiologists, 3 (5.1%) were identified as such by three radiologists, 12 (20.3%) were identified by two radiologists, and 17 (28.8%) were identified by only a single radiologist.

Conclusion

The two-phase image annotation process yields improved agreement among radiologists in the interpretation of nodules ≥ 3mm. Nevertheless, substantial variabilty remains across radiologists in the task of lung nodule identification.

Rationale and Objectives

Integral to the mission of the National Institutes of Health–sponsored Lung Imaging Database Consortium is the accurate definition of the spatial location of pulmonary nodules. Because the majority of small lung nodules are not resected, a reference standard from histopathology is generally unavailable. Thus assessing the source of variability in defining the spatial location of lung nodules by expert radiologists using different software tools as an alternative form of truth is necessary.

Materials and Methods

The relative differences in performance of six radiologists each applying three annotation methods to the task of defining the spatial extent of 23 different lung nodules were evaluated. The variability of radiologists’ spatial definitions for a nodule was measured using both volumes and probability maps (p-map). Results were analyzed using a linear mixed-effects model that included nested random effects.

Results

Across the combination of all nodules, volume and p-map model parameters were found to be significant at P < .05 for all methods, all radiologists, and all second-order interactions except one. The radiologist and methods variables accounted for 15% and 3.5% of the total p-map variance, respectively, and 40.4% and 31.1% of the total volume variance, respectively.

Conclusion

Radiologists represent the major source of variance as compared with drawing tools independent of drawing metric used. Although the random noise component is larger for the p-map analysis than for volume estimation, the p-map analysis appears to have more power to detect differences in radiologist-method combinations. The standard deviation of the volume measurement task appears to be proportional to nodule volume.

The purpose of this study was to develop and test a computerized method for the fully automated analysis of abnormal asymmetry in digital posteroanterior (PA) chest radiographs. An automated lung segmentation method was used to identify the aerated lung regions in 600 chest radiographs. Minimal apriori lung morphology information was required for this gray-level thresholding-based segmentation. Consequently, segmentation was applicable to grossly abnormal cases. The relative areas of segmented right and left lung regions in each image were compared with the corresponding area distributions of normal images to determine the presence of abnormal assymetry. Computerized diagnoses were compared with image ratings assigned by a radiologist. The ability of the automated method to distinguish normal from asymmetrically abnormal cases was evaluated by using receiver operating characteristic (ROC) analysis, which yielded an area under the ROC curve of 0.84. This automated method demonstrated promising performance in its ability to detect abnormal asymmetry in PA chest images. We believe this method could play a role in a picture archiving and communications (PACS) environment to immediately identify abnormal cases and to function as one component of a multifaceted computeraided diagnostic scheme.

Objective

An area of need in cancer informatics is the ability to store images in a comprehensive database as part of translational cancer research. To meet this need, we have implemented a novel tandem database infrastructure that facilitates image storage and utilisation.

Background

We had previously implemented the Thoracic Oncology Program Database Project (TOPDP) database for our translational cancer research needs. While useful for many research endeavours, it is unable to store images, hence our need to implement an imaging database which could communicate easily with the TOPDP database.

Methods

The Thoracic Oncology Research Program (TORP) imaging database was designed using the Research Electronic Data Capture (REDCap) platform, which was developed by Vanderbilt University. To demonstrate proof of principle and evaluate utility, we performed a retrospective investigation into tumour response for malignant pleural mesothelioma (MPM) patients treated at the University of Chicago Medical Center with either of two analogous chemotherapy regimens and consented to at least one of two UCMC IRB protocols, 9571 and 13473A.

Results

A cohort of 22 MPM patients was identified using clinical data in the TOPDP database. After measurements were acquired, two representative CT images and 0–35 histological images per patient were successfully stored in the TORP database, along with clinical and demographic data.

Discussion

We implemented the TORP imaging database to be used in conjunction with our comprehensive TOPDP database. While it requires an additional effort to use two databases, our database infrastructure facilitates more comprehensive translational research.

Conclusions

The investigation described herein demonstrates the successful implementation of this novel tandem imaging database infrastructure, as well as the potential utility of investigations enabled by it. The data model presented here can be utilised as the basis for further development of other larger, more streamlined databases in the future.