Search tips
Search criteria 


Logo of nihpaAbout Author manuscriptsSubmit a manuscriptHHS Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
Cancer. Author manuscript; available in PMC 2010 November 1.
Published in final edited form as:
PMCID: PMC2767474

Inter-observer reproducibility of histologic parameters of melanoma deposits in sentinel lymph nodes: implications for management of patients with melanoma

Rajmohan Murali, MBBS, FRCPA,1,2,3 Alistair J. Cochran, MD, FRCP(Glasg.), FRCPath,*,4 Martin G. Cook, MD, FRCPath, FRCPA,*,5 Joseph D. Hillman, MD,*,4 Rooshdiya Z. Karim, MBBS, FRCPA,*,1,3 Marc Moncrieff, MD, FRCS,*,2 Hans Starz, MD,*,6 John F. Thompson, MD, FRACS, FACS,2,7 and Richard A. Scolyer, MD, FRCPA, FRCPath1,2,3



Histologic parameters of melanoma deposits in sentinel lymph nodes (SLNs) have been shown to be predictive of the presence or absence of tumor in non-SLNs and clinical outcome, but assessment of these parameters is prone to inter-observer variation.


Histologic sections of 44 SLNs containing metastatic melanoma were examined by 7 pathologists. Parameters assessed included cross-sectional area of tumor deposits, cross-sectional area of SLNs, percentage of SLN area involved by tumor calculated from the two previous parameters, estimated percentage of SLN area involved by tumor, tumor penetrative depth (TPD), location of tumor within the SLN, and presence of extracapsular spread (ECS). Levels of inter-observer agreement were measured using intraclass correlation coefficients (ICC).


There was good to excellent inter-observer agreement on measurement of quantitative parameters: maximum size of largest tumor deposits, calculated area of 3 largest tumor deposits, percent area of SLN involved by tumor and TPD (ICC 0.88, 0.73, 0.68 and 0.83, respectively). There was moderate agreement on the evaluation of subcapsular versus non-subcapsular location of tumor deposits (ICC = 0.50). Agreement on assessment of ECS was fair (ICC = 0.39).


Assessment of some of the quantitative parameters was highly reproducible between pathologists. However, evaluation of the location of tumor deposits within SLNs and assessment of ECS was less reproducible. Clearer definitions and training can be expected to improve the reproducibility of assessment. These results have important implications for the reliability and reproducibility of these parameters in staging, prediction of outcome, and clinical management of melanoma patients.

Keywords: Diagnosis, Histologic parameters, Inter-observer reproducibility, Melanoma, Pathology, Sentinel lymph node

The modern sentinel lymph node (SLN) biopsy procedure was developed in the late 1980s and early 1990s.1 The SLN biopsy (SLNB) procedure in melanoma patients is a highly accurate staging method and the tumor-harboring status of the SLN is the most important prognostic factor for melanoma patients with early stage disease.2-12 In clinically nodenegative patients, complete regional lymph node dissection (CLND) is now restricted in most centers to patients with demonstrated metastatic disease in SLNs, sparing the majority of patients major surgery with its associated anesthetic risks and potential morbidity (acute wound problems, nerve injury and chronic lymphedema).13

Only a minority (15-30%) of patients with positive SLNs have additional lymph node involvement in subsequent CLND specimens.14-25 If it could be reliably determined which patients were likely to harbor tumor in non-SLNs, the remaining patients could be safely spared CLND and its potential morbidity. Prior studies have evaluated clinical and pathologic features (features of the primary tumor, number of positive SLNs and histologic characteristics of SLN tumor deposits) in an attempt to predict which SLN-positive patients are likely to have tumor in regional non-SLNs. These studies found that patient age,26 gender,22 site of primary tumor,17, 24, 27 primary tumor (Breslow) thickness,17, 20, 22, 27, 28 Clark level of invasion,27 ulceration,20 primary tumor mitotic rate,18, 22 absence of regression,20 and the number of positive SLNs16, 22, 23, 27 were significantly predictive of the presence of metastatic melanoma in non-SLN. Histologic parameters of SLN metastases that have been assessed include the size of metastases, tumor penetrative depth (TPD, also known as maximum subcapsular depth and centripetal thickness), the location of SLN tumor deposits in the SLN, the percentage cross-sectional area of the SLN involved and the presence of extracapsular spread. Many of these parameters have been shown to be predictive of non-SLN status and clinical outcome (Table 1).15, 17, 21, 22, 26, 28-30 The power of individual features of melanoma metastases in SLN to predict tumor in non-SLN and survival reported in some studies has not been reproduced in others.18, 21, 22, 26, 31

Table 1
Studies of predictive parameters of metastatic melanoma in SLN

Accurate assessment, classification and measurement of the histologic characteristics of SLN tumor deposits requires pathologists to make subjective judgements, and is therefore prone to inter-observer variation. The amount of such variation can be assessed by determining a reliability index, which is an indicator of the level of agreement between observers with adjustment for the degree of agreement that could be expected on the basis of chance. It is usually expressed as an intraclass correlation coefficient (ICC) or kappa score.32-35 ICC/kappa equals 0 if the observed level of agreement could be expected by chance, and equals 1 if the observers always agree completely.

There have to date been no published studies reporting the inter-observer reproducibility of evaluation of histologic parameters of melanoma deposits in SLNs. Reproducible assessment of these parameters by pathologists is essential to ensure clinical applicability and predictive accuracy, as well as permitting comparison of different studies evaluating these approaches. Standardization is critical if these parameters are to be used to select individual patients who might safely be spared CLND.36 In this study, we attempted to determine the level of inter-observer agreement between pathologists at different institutions in the assessment of a range of histologic characteristics of SLN melanoma deposits. The aim was to assess the level of agreement in the assessment of these parameters, and to identify any areas of difficulty or poor agreement. Identification of such problem areas could potentially lead to clearer definitions and criteria, and/or quality assurance measures such as training slides or microscope consensus sessions, resulting in improved interobserver agreement and standardization in the evaluation of these parameters.


Slides from forty-four SLNs that contained metastatic melanoma deposits were retrieved from the records of the Department of Anatomical Pathology, Royal Prince Alfred Hospital, Sydney, Australia. The cases were derived from patients with primary cutaneous melanoma who underwent SLNB at the Sydney Melanoma Unit between January 2001 and December 2004. The cases were chosen at random, in order to represent a range of sizes and morphologies of tumor deposits. Three slides from each SLN (one stained with haematoxylin-eosin, and one each stained immunohistochemically with antibodies to S-100 protein and HMB-45) were chosen for assessment. The slides were de-identified and sent in turn to seven observers (RM, AJC, MGC, JH, RZK, RAS, and HS). The observers are pathologists with an interest in dermatopathology and experience ranging from less than five years to several decades. Each observer was also sent guidelines (Fig. 1) containing definitions of the parameters to be assessed: dimensions of the SLN (Fig. 2a), intranodal location of tumor (subcapsular, parenchymal and/or sinusoidal, Fig. 2b), presence of extracapsular spread (ECS, Fig. 2c), tumor penetrative depth (TPD) of deposit(s) (maximum distance of melanoma cells from the inner margin of the SLN capsule, Fig. 2d), dimensions of the three largest deposits (or all deposits if fewer than three were present, eg Fig. 2e) and an estimate of percentage cross-sectional area of SLN occupied by metastatic melanoma, based on low power examination of the sections. The dimensions of the SLN and the deposits that were measured were a) the maximal dimension of the SLN/deposit along its long axis (dimension 1), and b) the maximal dimension of the SLN/deposit perpendicular to its long axis (dimension 2). All microscopic measurements were to be made with an ocular micrometer. A consensus microscopy session was not held at the commencement of the study, as the goal of the study was to determine the existing level of inter-observer variation in the evaluation of the various parameters in current practice.

Figure 1
Guidelines for assessment.
Figure 2Figure 2Figure 2Figure 2Figure 2
Histologic characteristics of SLN deposits.

SLN tumor burden was calculated using only a microscope and an ocular micrometer. This involved utilizing the SLN dimensions and the dimensions of the three largest tumor deposits to determine the cross-sectional areas of the SLN and of the three largest deposits of melanoma, using the formula for the area of an ellipse [0.25 × π × dimension 1 × dimension 2]. The percentage cross-sectional area of the SLN occupied by metastatic melanoma was calculated from these figures. Although few SLNs and melanoma deposits in SLNs are precisely elliptical in cross-section, the method proposed provides a reasonable approximation of SLN tumor burden, while avoiding complex sectioning protocols and measurements, and the need for specialized computer/video-based morphometric equipment.

Statistical analysis was performed using SPSS 16.0 (SPSS Inc., Chicago IL, 2007). The level of inter-observer agreement was calculated using intraclass correlation coefficients (ICC). According to the guidelines of Landis and Koch,37 ICC values of <0.20 can be interpreted as poor agreement, values of 0.21-0.40 as fair, values of 0.41-0.60 as moderate, values of 0.61-0.80 as good, and values of 0.81-1.00 as excellent agreement. The degree of correlation between the estimated and calculated percentage cross-sectional area of the SLN involved by tumor (intra-observer agreement) was calculated using the Pearson correlation coefficient. A p value of <0.05 was considered statistically significant.


The results are summarized in Table 2 and detailed in Table 3. There was good to excellent agreement between observers in the measurement of maximum size of the largest deposit (ICC = 0.88), the calculation of cross-sectional area of the SLN (ICC = 0.85) and of the three largest deposits (ICC = 0.73), the calculated and estimated percentage of the cross sectional area of the SLN occupied by the 3 largest deposits (ICC=0.68 and 0.94 respectively), and the TPD of metastases (ICC = 0.84). The level of agreement for the assessment of the subcapsular versus non-subcapsular location of metastases within the SLNs was moderate (ICS = 0.50), while that for the evaluation of extra-capsular spread ECS was fair (ICC = 0.39).

Table 2
Interobserver agreement in the assessment of histologic parameters of metastatic melanoma deposits in SLNs
Table 3
Assessment of characteristics of metastatic melanoma deposits in SLNs by 7 observers: number of cases (% of number of cases scored)

The intra-observer agreement between each observer’s estimated and calculated values for the percent SLN area occupied by tumor deposits was good (median ICC = 0.78, mean ICC = 0.70, range 0.38-0.95). Among the four pathologists with greater than 5 years’ experience in dermatopathology, the agreement was similar (median ICC = 0.79, mean ICC = 0.72, range 0.51-0.81) to that among those with less than 5 years’ experience (median ICC = 0.70, mean ICC = 0.67, range 0.38-0.95).


Previous studies have shown that several characteristics of deposits of metastatic melanoma in SLNs correlate with the presence of tumor in non-SLNs in subsequent CLND specimens (Table 1). Parameters that were predictive in univariate analyses (and less so in multivariate analyses) include the location of tumor within SLNs,15, 20, 29 the TPD (centripetal thickness or maximum subcapsular depth),15, 24, 26, 30, 38-41 the presence of ECS,22 the presence of tumor in perinodal lymphatics,40 and SLN tumor burden or size.15, 20, 21, 25-28, 39, 40 Many of these parameters also predict clinical outcome and provide important prognostic information. The size of SLN tumor deposits21, 28 and involvement of all three anatomic zones (subcapsular, parenchymal and sinusoidal areas) of SLNs by tumor21 are associated with a higher risk of recurrence. SLN tumor burden (size of SLN tumor deposits)17, 25, 28, 31 and TPD (in some studies30, 39) are also significantly associated with survival. However, other studies of tumor in SLN using similar definitions have found no association between some histologic characteristics and CLND status or survival. For example, multivariate analysis in the study by Debarbieux et al26 found size of the largest SLN metastasis to be predictive of disease-specific survival, while TPD and ECS were not found to be independent predictors of poorer disease-free survival. Frankel et al42 showed no association between the intranodal location of SLN deposits and CLND status. Govindarajan et al21 demonstrated that, although specific location of SLN tumor (in subcapsular, parenchymal or sinusoidal zones) did not predict the presence or absence of tumor in a CLND specimen, the presence of tumor in all three zones was independently associated with an increased rate of recurrence. Given these conflicting results, it remains unclear which characteristics of SLN tumor deposits most accurately predict tumor in non-SLN and provide the most accurate prognostic information.

Some of these SLN tumor characteristics have been clearly defined. TPD is defined as the maximum subcapsular depth of tumor within the SLN).15, 24, 26, 30, 38-40 ECS is the presence of tumor in perinodal tissues external to the lymph node capsule associated with the presence of tumor within the SLN.22 Location of tumor within lymph nodes has been characterized as subcapsular (in the subcapsular zone), parenchymal (within the nodal parenchyma), diffuse (extensive involvement of the SLN with effacement of nodal architecture) or multifocal.15, 20, 29 Tumor burden has been assessed differently in different studies using the Starz S-classification (tumor penetrative depth),15, 30, 38-40, 43 the largest size of tumor deposits,15, 17, 20, 21, 26, 27, 39, 40 or the cross-sectional area of such deposit(s),28,40 estimated by pathologists, calculated based on micrometer measurements, or measured using computer-assisted morphometry.25 In fact, nodal tumor deposits are often irregularly-shaped, may have ill-defined borders and their evaluation is in part dependent on sectioning protocols. More extensive sectioning may reveal additional tumor deposits or demonstrate a larger deposit(s) in the deeper sections (though peripheral tumor deposits are usually smaller).44, 45 Yet despite varying levels of precision of measurement, numerous studies have shown a positive correlation between the SLN tumor burden and tumor in non-SLN positivity and clinical outcome. It is apparent that the predictive value of SLN tumor burden holds regardless of the method of its assessment. Because of the variation of methods used to assess some histologic parameters, and the subjectivity inherent in the assessment of others, we sought to determine the degree of inter-observer variation in the assessment of these parameters.

We found that reproducibility of the assessment of TPD and SLN area was excellent, that inter-observer agreement in evaluation of the cross-sectional area of SLN deposits and calculation of the percentage of the SLN area involved by tumor was good. For each individual pathologist, estimation of the percentage of the SLN area involved by tumor using low power magnification was highly reproducible and closely comparable to the percentage calculated from painstaking and time consuming ocular micrometer measurements of deposits and SLNs. The results suggest that in experienced hands, estimation of tumor burden can apparently serve as an alternative to more labour-intensive and time-consuming methods. This finding should be confirmed in studies of larger patient groups, such as the Multicenter Sentinel Lymphadenectomy Trial II (MSLT-II).

Although the level of agreement for the measurement of maximal dimension of tumor was high (ICC = 0.88), we found that in occasional cases, a “shotgun scatter” pattern of metastatic melanoma was measured by one observer as a single large metastasis, while another/other observer(s) measured the tumor as several smaller adjacent deposits. It is because of the occurrence of such discrepancies that a more reliable measure of SLN tumor burden might be the percent area of SLN involvement, and the fact that estimated values of the latter parameter correlate well with the measured and calculated values makes this parameter particularly useful and reliable in the assessment of tumor burden.

Assessment of the location of tumor deposits within SLNs was less reproducible than the measurable parameters. It is generally recognized that assessment of the precise location of melanoma deposits within SLNs can be difficult. This is especially the case if tumor deposits are irregularly shaped or multifocal, and in routine histologic sections in which the different architectural compartments are not clearly distinguishable. In this study, agreement for assessment of subcapsular location of tumor compared with all non-subcapsular locations was moderate (ICC = 0.50). For the non-subcapsular locations, interobserver agreement was moderate for parenchymal sites (ICC = 0.48) and poor for sinusoidal sites (ICC = 0.14). The location of tumor in non-subcapsular locations has in general been shown to be more often associated with non-SLN involvement than tumor confined to subcapsular locations,29 and therefore subcategorization of non-subcapsular tumor deposits seems unwarranted, particularly since such subcategorization is poorly reproducible. Providing a more precise definition of subcapsular location and having a consensual discussion of this would be likely to improve the reproducibility of its recording.

ECS is routinely reported by pathologists evaluating lymph node metastases and is a prognostically important observation.46-48 ECS is rarely seen in SLNs containing metastatic melanoma, and we found that assessment of ECS was only fairly reproducible (ICC = 0.39). There are few reports of inter-observer agreement in the assessment of ECS in the literature.49 Theunissen et al49 showed only moderate inter-observer agreement (kappa = 0.50) in initial assessment of ECS in lymph nodes containing metastatic lung cancer. In our study, re-examination of the discrepant cases showed that the presence of ECS was difficult to assess due to difficulties presented by disruption of the nodal capsule overlying the tumor. Other reasons for poor reproducibility of ECS assessment include lack of ‘full-face’ sections, artifactual displacement of intranodal tumor cells into extranodal locations, and the difficulty in assessing tumor deposits associated with extensive fibrosis and desmoplasia. While the results suggest that reproducible assessment of ECS may be problematic, the numbers of cases showing ECS in this study are small, and further study of larger numbers of cases may clarify the issue.

Compared to the quantitative variables, the inter-observer agreement in assessment of location of tumor deposits in SLNs and ECS is relatively poorer. Clearer definitions and criteria may improve the reproducibility of assessment of these tumor characteristics. For example, Theunissen et al found that inter-observer reproducibility of the assessment of ECS in lymph nodes improved (kappa = 0.78) following the introduction of clear criteria for ECS.49

The evaluation of characteristics of metastatic melanoma deposits in SLNs is clinically important for two reasons. Selected patients may be spared CLND on the basis of information derived from analysis of these parameters, for example patients with very small SLN tumor deposits.43 Already in some centers, patients with SLN tumor deposits <0.1mm in maximal dimension are not offered CLND.36 In this study, we found that only 2 cases were deemed by at least one observer to have a SLN deposit <0.1mm in size. However, 3 observers in one of these cases,and 4 observers in the other scored the deposit as being ≥0.1mm in size. Therefore clinical management of these patients according to the 0.1mm cutoff would have been different based on the pathologist evaluating the SLN. However, since only two cases fell within this size group in this study, it is difficult to draw definitive conclusions on this point. The results of MSLT-II, currently enrolling patients, will demonstrate whether or not it is appropriate to observe patients found to be SLN positive, rather than proceeding routinely to immediate CLND. In the future, staging and prediction of outcome in SLN-positive patients may be based on evaluation of characteristics of SLN tumor deposits, and future revisions of the American Joint Committee on Cancer staging system for melanoma will likely involve sub-staging of SLN based on differing outcomes associated with varying tumor loads.

Because the participating pathologists were not required to score each feature independently on each of the HE-stained and immunohistochemically stained sections, it was not possible to address the question of whether assessment of SLN parameters is more reproducible on HE-stained or immunohistochemically stained sections.

In summary, assessment of characteristics of metastatic melanoma in SLNs (such as TPD and tumor burden) is generally highly reproducible between pathologists in different institutions and with differing levels of experience. Estimated values of the percentage area of the SLN occupied by tumor correlated well with formally measured and calculated values of this parameter. This suggests that estimation of tumor burden may be sufficient, avoiding laborious and time-consuming measurement and calculation of nodal tumor burden. Validation studies are required to confirm the accuracy and reliability of the estimation method.


Sources of support: Drs Murali, Karim and Scolyer are Cancer Institute NSW Clinical Research Fellows. Dr Cochran receives funding from NIH/National Cancer Institute as part of CA29605.


CONDENSED ABSTRACT Assessment of quantitative histologic parameters of melanoma deposits in sentinel lymph nodes (SLNs) was highly reproducible between pathologists, while evaluation of the location of tumor deposits within SLNs and assessment of ECS was less reproducible. These results have important implications for the reliability and reproducibility of these parameters in staging, prediction of outcome, and clinical management of melanoma patients.

Financial disclosures: none to declare.

*Interim findings from this study were presented at the 6th Biennial International Sentinel Node Society Meeting, Sydney, 18-20 February 2008.


1. Morton DL, Wen DR, Wong JH, et al. Technical details of intraoperative lymphatic mapping for early stage melanoma. Arch Surg. 1992;127(4):392–9. [PubMed]
2. Cascinelli N, Belli F, Santinami M, et al. Sentinel lymph node biopsy in cutaneous melanoma: the WHO Melanoma Program experience. Ann Surg Oncol. 2000;7(6):469–74. [PubMed]
3. Thompson JF, Scolyer RA, Kefford RF. Cutaneous melanoma. Lancet. 2005;365(9460):687–701. [PubMed]
4. Clary BM, Brady MS, Lewis JJ, Coit DG. Sentinel lymph node biopsy in the management of patients with primary cutaneous melanoma: review of a large single-institutional experience with an emphasis on recurrence. Ann Surg. 2001;233(2):250–8. [PubMed]
5. Gershenwald JE, Thompson W, Mansfield PF, et al. Multi-institutional melanoma lymphatic mapping experience: the prognostic value of sentinel lymph node status in 612 stage I or II melanoma patients. J Clin Oncol. 1999;17(3):976–83. [PubMed]
6. Jansen L, Nieweg OE, Peterse JL, et al. Reliability of sentinel lymph node biopsy for staging melanoma. Br J Surg. 2000;87(4):484–9. [PubMed]
7. Krag DN, Meijer SJ, Weaver DL, et al. Minimal-access surgery for staging of malignant melanoma. Arch Surg. 1995;130(6):654–8. discussion 659-60. [PubMed]
8. Morton DL, Thompson JF, Essner R, et al. Validation of the accuracy of intraoperative lymphatic mapping and sentinel lymphadenectomy for early-stage melanoma: a multicenter trial. Multicenter Selective Lymphadenectomy Trial Group. Ann Surg. 1999;230(4):453–63. discussion 463-5. [PubMed]
9. Reintgen D, Cruse CW, Wells K, et al. The orderly progression of melanoma nodal metastases. Ann Surg. 1994;220(6):759–67. [PubMed]
10. Thompson JF, McCarthy WH, Bosch CM, et al. Sentinel lymph node status as an indicator of the presence of metastatic melanoma in regional lymph nodes. Melanoma Research. 1995;5(4):255–60. [PubMed]
11. Thompson JF. The Sydney Melanoma Unit experience of sentinel lymphadenectomy for melanoma. Ann Surg Oncol. 2001;8(9 Suppl):44S–47S. [PubMed]
12. Balch CM, Soong SJ, Gershenwald JE, et al. Prognostic factors analysis of 17,600 melanoma patients: validation of the American Joint Committee on Cancer melanoma staging system. J Clin Oncol. 2001;19(16):3622–34. [PubMed]
13. Urist MM, Maddox WA, Kennedy JE, Balch CM. Patient risk factors and surgical morbidity after regional lymphadenectomy in 204 melanoma patients. Cancer. 1983;51(11):2152–6. [PubMed]
14. Morton DL, Thompson JF, Cochran AJ, et al. Sentinel-node biopsy or nodal observation in melanoma. New Engl J Med. 2006;355(13):1307–17. [PubMed]
15. Scolyer RA, Li LX, McCarthy SW, et al. Micromorphometric features of positive sentinel lymph nodes predict involvement of nonsentinel nodes in patients with melanoma. Am J Clin Pathol. 2004;122(4):532–9. [PubMed]
16. Salti GI, Das Gupta TK. Predicting residual lymph node basin disease in melanoma patients with sentinel lymph node metastases. Am J Surg. 2003;186(2):98–101. [PubMed]
17. Pearlman NW, McCarter MD, Frank M, et al. Size of sentinel node metastases predicts other nodal disease and survival in malignant melanoma. Am J Surg. 2006;192(6):878–81. [PubMed]
18. Page AJ, Carlson GW, Delman KA, et al. Prediction of nonsentinel lymph node involvement in patients with a positive sentinel lymph node in malignant melanoma. Am Surg. 2007;73(7):674–8. discussion 678-9. [PubMed]
19. Ranieri JM, Wagner JD, Azuaje R, et al. Prognostic importance of lymph node tumor burden in melanoma patients staged by sentinel node biopsy. Ann Surg Oncol. 2002;9(10):975–81. [PubMed]
20. Reeves ME, Delgado R, Busam KJ, et al. Prediction of nonsentinel lymph node status in melanoma. Ann Surg Oncol. 2003;10(1):27–31. [PubMed]
21. Govindarajan A, Ghazarian DM, McCready DR, Leong WL. Histological features of melanoma sentinel lymph node metastases associated with status of the completion lymphadenectomy and rate of subsequent relapse. Ann Surg Oncol. 2007;14(2):906–12. [PubMed]
22. Sabel MS, Griffith K, Sondak VK, et al. Predictors of nonsentinel lymph node positivity in patients with a positive sentinel node for melanoma. J Am Coll Surg. 2005;201(1):37–47. [PubMed]
23. Wagner JD, Gordon MS, Chuang TY, et al. Predicting sentinel and residual lymph node basin disease after sentinel lymph node biopsy for melanoma. Cancer. 2000;89(2):453–62. [PubMed]
24. Starz H, Siedlecki K, Balda BR. Sentinel lymphonodectomy and s-classification: a successful strategy for better prediction and improvement of outcome of melanoma. Ann Surg Oncol. 2004;11(3 Suppl):162S–8S. [PubMed]
25. Vuylsteke RJ, Borgstein PJ, van Leeuwen PA, et al. Sentinel lymph node tumor load: an independent predictor of additional lymph node involvement and survival in melanoma. Annals of Surgical Oncology. 2005;12(6):440–8. [PubMed]
26. Debarbieux S, Duru G, Dalle S, et al. Sentinel lymph node biopsy in melanoma: a micromorphometric study relating to prognosis and completion lymph node dissection. Br J Dermatol. 2007;157(1):58–67. [PubMed]
27. Lee JH, Essner R, Torisu-Itakura H, et al. Factors predictive of tumor-positive nonsentinel lymph nodes after tumor-positive sentinel lymph node dissection for melanoma. J Clin Oncol. 2004;22(18):3677–84. [PubMed]
28. Cochran AJ, Wen DR, Huang RR, et al. Prediction of metastatic melanoma in nonsentinel nodes and clinical outcome based on the primary melanoma and the sentinel node. Mod Pathol. 2004;17(7):747–55. [PubMed]
29. Dewar DJ, Newell B, Green MA, et al. The microanatomic location of metastatic melanoma in sentinel lymph nodes predicts nonsentinel lymph node involvement. J Clin Oncol. 2004;22(16):3345–9. [PubMed]
30. Starz H, Balda BR, Kramer KU, et al. A micromorphometry-based concept for routine classification of sentinel lymph node metastases and its clinical relevance for patients with melanoma. Cancer. 2001;91(11):2110–21. [PubMed]
31. Roka F, Mastan P, Binder M, et al. Prediction of non-sentinel node status and outcome in sentinel node-positive melanoma patients. Eur J Surg Oncol. 2007;34(1):82–88. [PubMed]
32. Shrout PE, Fleiss JL. Intraclass correlation: uses in assessing rater reliability. Psychol Bull. 1979;2:420–428. [PubMed]
33. Fleiss JL. Statistical Methods for Rates and Proportions. John Wiley & Sons; New York: 1981. (Wiley Series in Probability and Mathematics Statistics).
34. Cohen J. A coefficient of agreement for nominal scales. Educ Psychol Measurement. 1960;20:37–46.
35. Schouten HJA. Nominal scale agreement among observers. Psychometrika. 1986;51:453–466.
36. van Akkooi AC, de Wilt JH, Verhoef C, et al. Clinical relevance of melanoma micrometastases (<0.1 mm) in sentinel nodes: are these nodes to be considered negative? Ann Oncol. 2006;17(10):1578–85. [PubMed]
37. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33(1):159–74. [PubMed]
38. Fink AM, Weihsengruber F, Spangl B, et al. S-classification of sentinel lymph node biopsy predicts the results of complete regional lymph node dissection. Melanoma Res. 2005;15(4):267–71. [PubMed]
39. Rossi CR, De Salvo GL, Bonandini E, et al. Factors predictive of nonsentinel lymph node involvement and clinical outcome in melanoma patients with metastatic sentinel lymph node. Ann Surg Oncol. 2008;15(4):1202–1210. [PubMed]
40. Satzger I, Volker B, Meier A, et al. Criteria in sentinel lymph nodes of melanoma patients that predict involvement of nonsentinel lymph nodes. Ann Surg Oncol. 2008;15(6):1723–1732. [PubMed]
41. Starz H. Pathology of the sentinel lymph node in melanoma. Semin Oncol. 2004;31(3):357–62. [PubMed]
42. Frankel TL, Griffith KA, Lowe L, et al. Do micromorphometric features of metastatic deposits within sentinel nodes predict nonsentinel lymph node involvement in melanoma? Ann Surg Oncol. 2008;15(9):2403–11. [PubMed]
43. van der Ploeg IM, Nieweg OE, Olmos RA Valdes, Kroon BB. Is completion lymph node dissection needed in case of minimal melanoma metastasis in the sentinel node? Ann Surg Oncol. 2008;15(Suppl 1):17. [PubMed]
44. Scolyer RA, Murali R, McCarthy SW, Thompson JF. Pathologic examination of sentinel lymph nodes from melanoma patients. Semin Diagn Pathol. 2008;25(2):100–11. [PubMed]
45. Scolyer RA, Murali R, Satzger I, Thompson JF. The detection and significance of melanoma micrometastases in sentinel nodes. Surg Oncol. 2008;17(3):165–74. [PubMed]
46. Horn LC, Hentschel B, Galle D, Bilek K. Extracapsular extension of pelvic lymph node metastases is of prognostic value in carcinoma of the cervix uteri. Gynecol Oncol. 2008;108(1):63–7. [PubMed]
47. Altinyollar H, Berberoglu U, Gulben K, Irkin F. The correlation of extranodal invasion with other prognostic parameters in lymph node positive breast cancer. J Surg Oncol. 2007;95(7):567–71. [PubMed]
48. Prabhudesai A, Arif S, Finlayson CJ, Kumar D. Impact of microscopic extranodal tumor deposits on the outcome of patients with rectal cancer. Dis Colon Rectum. 2003;46(11):1531–7. [PubMed]
49. Theunissen PH, Bollen EC, Koudstaal J, Thunnissen FB. Intranodal and extranodal tumour growth in early metastasised non-small cell lung cancer: problems in histological diagnosis. J Clin Pathol. 1994;47(10):920–3. [PMC free article] [PubMed]