Search tips
Search criteria 


Logo of endoconnBioScientificaSociety for EndocrinologyEuropean Society for EndocrinologyHomeAboutSubmitAlertsEndocrine ConnectionsThis Article
Endocr Connect. 2017 January; 6(1): 33–38.
Published online 2017 January 9. doi:  10.1530/EC-16-0092
PMCID: PMC5302166

Is visual radiological evaluation of liver tumour burden in patients with neuroendocrine tumours reproducible?



Visual semi-quantitative assessment of liver tumour burden for neuroendocrine tumour liver metastases is often used in patient management and outcome. However, published data on the reproducibility of these evaluations are lacking.


The aim of this study was to evaluate the interobserver and intraobserver agreement of a visual semi-quantitative assessment of liver tumour burden using CT scan.


Fifty consecutive patients (24 men and 26 women, mean aged 54 years) were retrospectively reviewed by four readers (two senior radiologists, one junior radiologist and one gastroenterologist) who assessed the liver tumour burden based on a visual semi-quantitative method with four classes (0–10, 11–25, 26–50 and ≥50%). Interobserver and intraobserver agreement were assessed by weighted kappa coefficient and percentage of agreement. The intraclass correlation was calculated.


Agreement among the four observers for the evaluation of liver tumour burden was substantial, ranging from 0.62 to 0.73 (P < 0.0001). The intraclass coefficient was 0.977 (P < 0.0001). Intraobserver agreement was 0.78 and ICC was 0.97.


Reproducibility of the visual semi-quantitative evaluation of liver tumour burden is good and is independent of the level of experience of the readers. We therefore suggest that clinical studies in patients with neuroendocrine liver metastases use this method to categorise liver tumour burden.

Keywords: neuroendocrine tumour, CT scan, liver metastases, liver burden


Neuroendocrine neoplasms are a heterogeneous group of tumours arising from endocrine and nervous system cells. Two of the most common anatomical sites of origin are the gastrointestinal tract and the pancreatic islet cells, both grouped as gastroenteropancreatic neuroendocrine tumours, or GEP-NETs. They are frequently metastasised at diagnosis, and the liver is the most common site of metastases (1). The presence of liver metastases is a significant negative prognostic factor that depends on the site of the primary tumour, its histological differentiation and proliferative activity. At present, the latter is assessed by the number of mitoses per unit area of tumour or as the percentage of neoplastic cells immunolabelling for the proliferation marker Ki67. The system that was recently proposed for GEP-NETs by the European Neuroendocrine Tumor Society (ENETS) and recommended by WHO uses either mitotic rate or the Ki67 labelling index (2).

Except for neuroendocrine carcinomas (G3 tumours), curative surgical resection is the reference treatment for liver metastases (NELM), with 5-year overall survival rates ranging from 60 to 80%, low mortality and acceptable morbidity (2). If more than 90% of the liver tumour burden can be resected, debulking resection can also be considered to reduce symptoms (2).

When surgical resection cannot be considered, locoregional liver-directed therapies and/or medical therapies may be discussed. In these cases, the therapeutic choice is based on the previously mentioned tumour characteristics (primary tumour site, histological differentiation and tumour activity) as well as tumour progression in the liver, the presence of extrahepatic lesions and the liver tumour burden (2).

Several recently published studies have shown that liver tumour burden was an important factor for patient management and long-term outcome, especially for somatostatin analogue treatment, and also for transarterial chemoembolisation, bland embolisation or selective intraarterial radiation therapy (SIRT) (3, 4, 5, 6, 7, 8). For example, liver tumour burden has been shown to be an independent prognostic factor for survival and, in the Promid study, an important factor for the antiproliferative effect of octreotide LAR (4, 6, 9).

In most of these studies, the liver tumour burden was evaluated by a visual semi-quantitative assessment of the total tumour volume in the liver on CT scan and/or MRI, categorised into three to five classes by percentage. To our knowledge, there are no published data on the interobserver agreement for this assessment.

CT scan, because it enables the exploration of the most common metastatic sites, is the reference technique for initial evaluation and follow-up of NET-associated metastases, when using appropriate imaging techniques (triphasic CT scan acquisition including a late arterial phase), with a mean sensitivity for detection of NET liver metastases of 82% (range 78–100%) and a mean specificity of 92% range (83–100%) (10, 11, 12). MRI is generally more sensitive at detecting liver metastases because of the improved tissue contrast and is a non-radiant technique; nonetheless, MRI is less available and more expensive than CT and is generally not a standard NET imaging method (10, 11, 12).

Thus, the aim of our study was to evaluate the intraobserver and interobserver agreement of a visual semi-quantitative assessment of liver tumour burden, based on CT scan, because this is the most widely used imaging modality in oncology.

Patients and methods

Patient selection

This was a retrospective study and a waiver was obtained from the institutional review board.

From 2013 to 2014, all patients referred to our institution with a diagnosis of GEP-NETs were retrospectively reviewed. All patients in whom at least one synchronous liver metastasis was identified on baseline CT scan were included. Patients with previous systemic or locoregional liver therapies were excluded.

The final diagnosis of a neuroendocrine tumour was histologically confirmed by conventional and immunohistochemical techniques using chromogranin A and synaptophysin stain performed on the liver biopsy or the surgical specimen of the primary tumour. In patients with no pathologic proof of neuroendocrine liver metastases, the diagnosis was based on typical imaging features (13).

Demographic, clinical and biological data were collected in a retrospective review of the medical records.

CT studies

All patients underwent CT scan, performed with a 64-section scanner (VCT LightSpeed; GE Healthcare). The same MDCT protocol optimised for neuroendocrine tumours was performed in all patients. This included an unenhanced acquisition and two contrast-enhanced acquisitions (late arterial and portal venous phase) after intravenous administration of 2 mL/kg of non-ionic contrast medium (Xenetix, Guerbet, France).

Imaging analysis

Anonymous CT data were analysed by 4 readers blinded to the clinical and biological data: two senior abdominal radiologists (MZ and MPV) with 11 and 20 years of experience, respectively, one junior radiologist (ML) and one gastroenterologist (OH) with 10 years of experience in neuroendocrine tumour management.

Each reader was asked to assess the liver tumour burden based on a visual semi-quantitative scale as follows: 0–10, 11–25, 26–50 and more than 50% of tumour involvement of the liver, using arterial or portal venous phase acquisitions depending on the tumour conspicuity (Fig. 1). Intraobserver variability was assessed for one reader (MZ) by repeating the assessment of tumour burden in all patients six months later.

Figure 1
(A) 57-year-old woman with liver metastases from an ileal neuroendocrine tumour. Liver tumour burden was found to be less than 10% by all observers. (B) 58-year-old woman with liver metastases from a rectal neuroendocrine tumour. Liver tumour burden was ...

Statistical analysis

Quantitative variables were determined by mean and s.d. Categorical variables were determined by count and percentages. Frequencies were compared using a chi-square test. Interobserver agreement was assessed using weighted kappa values. A coefficient between 0.00 and 0.20 indicated slight agreement, 0.21 and 0.40 fair agreement, 0.41 and 0.60 moderate agreement, 0.61 and 0.80 substantial agreement and 0.81 and 1.00 almost perfect agreement (14). Furthermore, percentage of agreement and intraclass correlation coefficient (ICC) were calculated. A P value of 0.05 was considered to be significant. All analyses were performed using the Statistical Package for the Social Sciences software (SPSS Inc., version 20.0).


Patient characteristics

A total of 50 patients were included in the study, 24 men and 26 women, mean age 54 years (30–75).

The primary sites of the neuroendocrine tumours were the foregut (n = 28), the midgut (n = 15), the hindgut (n = 2) and indeterminate (n = 5). Forty patients had well-differentiated tumours (80%) and 10 had poorly differentiated tumours (20%). Thirteen (26%) tumours were G1, 33 (66%) were G2 and the remaining 4 (8%) were G3.

Liver tumour burden assessment, interobserver and intraobserver agreement

The details of the radiological assessment of liver tumour burden by the four readers are summarised in Table 1. There was no significant difference among the four liver tumour burden distribution (P = 0.87).

Table 1
Radiological semi-quantitative liver tumour burden assessment according to the 4 readers.

In a patient-based analysis, all four observers agreed on the liver tumour burden assessment in 29/50 patients (58%). All cases of disagreement concerned two contiguous classes.

The interobserver agreement was substantial for all observer pairs (kappa ranging from 0.62 to 0.73) (Table 2). Percentage of agreement ranged from 72 to 80%. Overall, ICC was 0.977 (P < 0.0001).

Table 2
Interobserver and intraobserver agreement.

The intraobserver agreement was substantial (kappa = 0.78), with a percentage of agreement of 84%; ICC was 0.97 (Table 2).


Liver tumour burden is an important criterion for the management of neuroendocrine liver metastases, both for prognosis and treatment (4, 6, 9, 15, 16).

Although there is no standardised imaging method to reliably measure liver tumour burden, the ENETS Consensus Guidelines state that the estimation of the percentage of liver tumour involvement by an experienced radiologist with a visual semi-quantitative method is the best option (2). However, to our knowledge, the reproducibility of this method has never been studied.

Our study demonstrates that there was substantial agreement in evaluating liver tumour burden on CT examinations among the four observers with different levels of experience. As expected, the strongest interobserver correlation was between the two senior radiologists and the weakest was between the junior radiologist and the gastroenterologist. However, the differences between pairs of observers were not significant. Intraobserver correlation was a bit better than all interobserver correlations.

In our study, we included consecutive patients with neuroendocrine liver metastases whatever the primary site, tumour grading or differentiation and the imaging features of the liver metastases. Thus, there were different patterns of liver metastases (well defined or infiltrative) in our patient population. Indeed, the evaluation of liver tumour burden is expected to be more difficult when metastases are infiltrative. Moreover, the distribution among the different classes by percentage of liver tumour burden was fair, which provides an assessment of the reproducibility in patients with various liver tumour burdens.

The thresholds for the classes (≤10, 11–25, 26–50 and >50% of tumour involvement) were chosen because they have already been used in other studies. One of the first studies to use the same percentages for the assessment of liver tumour burden showed a good correlation between plasma levels of chromogranin A and liver tumour burden (9). The clinical relevance of these classes has been confirmed in many studies. Hentic and coworkers have shown that a significant liver tumour burden defined as more than 25% was an independent predictor of poorer survival, whereas Bertani and coworkers showed that a liver tumour burden of less than 25% was an important factor for improved survival (4, 15). Rincke and coworkers showed that the antiproliferative effect of somatostatin analogues was greater in patients with a low (≤10%) liver tumour burden, and Palazzo and coworkers found that a low-to-moderate (≤25%) liver tumour burden was predictive of tumour stability under somatostatin analogue therapy, whereas Caplin and coworkers found an antiproliferative effect of lanreotide in patient with larger hepatic tumour volumes (6, 16). Kress and coworkers have shown that best morphological response to transarterial chemoembolisation or bland embolisation was obtained with limited liver involvement (<50%) (5). More recently, in a series of 48 patients undergoing SIRT for unresectable NELM, Saxena and coworkers showed that a low liver tumour burden (<25%) was associated with a partial or complete tumour response and thus improved survival (8).

We chose to evaluate liver tumour burden using a visual semi-quantitative assessment, and we showed that this easy method was highly reproducible. In several other studies using the same approach, liver tumour burden was calculated using four to six of the most amount of diseased CT slices (6, 9). We feel that our method using the whole liver was more reliable and thorough. Other authors have evaluated liver tumour burden by counting the number of liver metastases (more or less than five lesions) (17, 18). Although a correlation was found between the number of liver metastases and the prognosis, the clinical relevance of this method is probably limited because the number of liver metastases does not necessarily reflect tumour load. Other researchers have determined the percentage of the liver tumour burden using liver volumetry with three-dimensional image-reconstruction commercial software (8, 19). However, this method is complicated and often difficult in patients with a major liver tumour burden and when lesions are poorly defined and/or infiltrative.

Our study has certain limitations. We did not use a method of reference in our liver tumour burden assessment. For example, it would have been very difficult to use a pathological examination of the entire liver, because very few patients underwent liver transplantation or hepatectomy. Moreover, comparisons between imaging findings and histological count of NELM on thin serial slices have already been performed by Elias and coworkers who concluded that half the number of NELM were undetectable on preoperative imaging, with an accuracy of 38% for CT scan (20). We could have compared the visual semi-quantitative evaluation to a quantitative volume evaluation using dedicated software. However, as stated previously, obtaining a gold standard by an accurate quantitative evaluation is impossible when the liver tumour burden is significant or in case of infiltrative disease. Moreover, our primary goal was to assess the reproducibility of visual semi-quantitative assessment and not to assess its accuracy.

In conclusion, we showed that a visual semi-quantitative evaluation of liver tumour burden is very reproducible, regardless of the level of experience of the readers. Therefore, we think that clinical studies in patients with neuroendocrine liver metastases should use this method to categorise liver tumour burden.

Declaration of interest

The authors declare that there is no conflict of interest that could be perceived as prejudicing the impartiality of the research reported.


This research did not receive any specific grant from any funding agency in the public, commercial or not-for-profit sector.


1. Frilling A, Modlin IM, Kidd M, Russell C, Breitenstein S, Salem R, Kwekkeboom D, Lau WY, Klersy C, Vilgrain V, et al. Recommendations for management of patients with neuroendocrine liver metastases. Lancet Oncology 2014. 15 e8–e21. ()10.1016/S1470-2045(13)70362-0 [PubMed] [Cross Ref]
2. Pavel M, Baudin E, Couvelard A, Krenning E, Oberg K, Steinmuller T, Anlauf M, Wiedenmann B, Salazar R. ENETS consensus guidelines for the management of patients with liver and other distant metastases from neuroendocrine neoplasms of foregut, midgut, hindgut, and unknown primary. Neuroendocrinology 2012. 95 157–176. ()10.1159/000335597 [PubMed] [Cross Ref]
3. Gupta S, Johnson MM, Murthy R, Ahrar K, Wallace MJ, Madoff DC, McRae SE, Hicks ME, Rao S, Vauthey J-N, et al. Hepatic arterial embolization and chemoembolization for the treatment of patients with metastatic neuroendocrine tumors: variables affecting response rates and survival. Cancer 2005. 104 1590–1602. ()10.1002/cncr.21389 [PubMed] [Cross Ref]
4. Hentic O, Couvelard A, Rebours V, Zappa M, Dokmak S, Hammel P, Maire F, O’Toole D, Lévy P, Sauvanet A, et al. Ki-67 index, tumor differentiation, and extent of liver involvement are independent prognostic factors in patients with liver metastases of digestive endocrine carcinomas. Endocrine-Related Cancer 2010. 18 51–59. ()10.1677/ERC-09-0319 [PubMed] [Cross Ref]
5. Kress O, Wagner HJ, Wied M, Klose KJ, Arnold R, Alfke H. Transarterial chemoembolization of advanced liver metastases of neuroendocrine tumors–a retrospective single-center analysis. Digestion 2003. 68 94–101. ()10.1159/000074522 [PubMed] [Cross Ref]
6. Rinke A, Muller HH, Schade-Brittinger C, Klose KJ, Barth P, Wied M, Mayer C, Aminossadati B, Pape UF, Blaker M, et al. Placebo-controlled, double-blind, prospective, randomized study on the effect of octreotide LAR in the control of tumor growth in patients with metastatic neuroendocrine midgut tumors: a report from the PROMID Study Group. Journal of Clinical Oncology 2009. 7 4656–4663. ()10.1200/jco.2009.22.8510 [PubMed] [Cross Ref]
7. Roche A, Girish BV, de Baere T, Ducreux M, Elias D, Laplanche A, Boige V, Schlumberger M, Ruffle P, Baudin E. Prognostic factors for chemoembolization in liver metastasis from endocrine tumors. Hepatogastroenterology 2004. 51 1751–1756. [PubMed]
8. Saxena A, Chua TC, Bester L, Kokandi A, Morris DL. Factors predicting response and survival after yttrium-90 radioembolization of unresectable neuroendocrine tumor liver metastases: a critical appraisal of 48 cases. Annals of Surgery 2010. 251 910–916. ()10.1097/SLA.0b013e3181d3d24a [PubMed] [Cross Ref]
9. Arnold R, Wilke A, Rinke A, Mayer C, Kann PH, Klose KJ, Scherag A, Hahmann M, Müller HH, Barth P. Plasma chromogranin A as marker for survival in patients with metastatic endocrine gastroenteropancreatictumors. Clinical Gastroenterology and Hepatology 2008. 6 820–827. ()10.1016/j.cgh.2008.02.052 [PubMed] [Cross Ref]
10. de Mestier L, Dromain C, d’Assignies G, Scoazec JY, Lassau N, Lebtahi R, Brixi H, Mitry E, Guimbaud R, Courbon F, et al. Evaluating digestive neuroendocrine tumor progression and therapeutic responses in the era of targeted therapies: state of the art. Endocrine-Related Cancer 2014. 21 R105–R120. ()10.1530/ERC-13-0365 [PubMed] [Cross Ref]
11. Sundin A, Vullierme MP, Kaltsas G, Plockinger U. ENETS Consensus Guidelines for the Standards of Care in Neuroendocrine Tumors: radiological examinations. Neuroendocrinology 2009. 90 167–183. ()10.1159/000184855 [PubMed] [Cross Ref]
12. van Essen M, Sundin A, Krenning EP, Kwekkeboom DJ. Neuroendocrine tumours: the role of imaging for diagnosis and therapy. Nature Reviews Endocrinology 2014. 10 102–114. ()10.1038/nrendo.2013.246 [PubMed] [Cross Ref]
13. Dromain C, de Baere T, Lumbroso J, Caillet H, Laplanche A, Boige V, Ducreux M, Duvillard P, Elias D, Schlumberger M, et al. Detection of liver metastases from endocrine tumors: a prospective comparison of somatostatin receptor scintigraphy, computed tomography, and magnetic resonance imaging. Journal of Clinical Oncology 2005. 23 70–78. ()10.1200/JCO.2005.01.013 [PubMed] [Cross Ref]
14. Cohen J. Weighted kappa: nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin 1968. 70 213–220. ()10.1037/h0026256 [PubMed] [Cross Ref]
15. Bertani E, Fazio N, Botteri E, Chiappa A, Falconi M, Grana C, Bodei L, Papis D, Spada F, Bazolli B, et al. Resection of the primary pancreatic neuroendocrine tumor in patients with unresectable liver metastases: possible indications for a multimodal approach. Surgery 2014. 155 607–614. ()10.1016/j.surg.2013.12.024 [PubMed] [Cross Ref]
16. Palazzo M, Lombard-Bohas C, Cadiot G, Matysiak-Budnik T, Rebours V, Vullierme MP, Couvelard A, Hentic O, Ruszniewski P. Ki67 proliferation index, hepatic tumor load, and pretreatment tumor growth predict the antitumoral efficacy of lanreotide in patients with malignant digestive neuroendocrine tumors. European Journal of Gastroenterology and Hepatology 2013. 25 232–238. ()10.1097/MEG.0b013e328359d1a6 [PubMed] [Cross Ref]
17. Janson ET, Holmberg L, Stridsberg M, Eriksson B, Theodorsson E, Wilander E, Oberg K. Carcinoid tumors: analysis of prognostic factors and survival in 301 patients from a referral center. Annals of Oncology 1997. 8 685–690. ()10.1023/A:1008215730767 [PubMed] [Cross Ref]
18. Turner GB, Johnston BT, McCance DR, McGinty A, Watson RG, Patterson CC, Ardill JES. Circulating markers of prognosis and response to treatment in patients with midgut carcinoid tumours. Gut 2006. 55 1586–1591. ()10.1136/gut.2006.092320 [PMC free article] [PubMed] [Cross Ref]
19. Hur S, Chung JW, Kim HC, Oh DY, Lee SH, Bang YJ, Kim WH. Survival outcomes and prognostic factors of transcatheter arterial chemoembolization for hepatic neuroendocrine metastases. Journal of Vascular and Interventional Radiology 2013. 24 947–956; quiz 57. ()10.1016/j.jvir.2013.02.030 [PubMed] [Cross Ref]
20. Elias D, Lefevre JH, Duvillard P, Goere D, Dromain C, Dumont F, Baudin E. Hepatic metastases from neuroendocrine tumors with a ‘thin slice’ pathological examination: they are many more than you think. Annals of Surgery 2010. 251 307–310. ()10.1097/SLA.0b013e3181bdf8cf [PubMed] [Cross Ref]

Articles from Endocrine Connections are provided here courtesy of Bioscientifica Ltd.