Search tips
Search criteria 


Logo of nihpaAbout Author manuscriptsSubmit a manuscriptHHS Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
Cancer Cytopathol. Author manuscript; available in PMC 2014 February 1.
Published in final edited form as:
PMCID: PMC3479324

Inter-Rater Agreement of Anal Cytology


Most anal cancers are caused by persistent infections with carcinogenic human papillomaviruses (HPV). Similar to cervical carcinogenesis, the progression from HPV infection to anal cancer goes through precancerous lesions that can be treated to prevent invasion. In analogy to cervical cytology, anal cytology has been proposed as a screening tool for anal cancer precursors in high-risk populations. We analyzed the inter-observer reproducibility of anal cytology in a population of 363 HIV-infected men who have sex with men (MSM). Liquid-based cytology (LBC) specimens were collected in the anal dysplasia clinic before performing high-resolution anoscopy (HRA) on all subjects. Papanicolaou-stained, LBC slides were evaluated by two cytopathologists, blinded to clinical outcome and the other pathologist's results, using the revised Bethesda terminology. Overall agreement between two observers was 66% (kappa 0.54, linear weighted kappa 0.69). Using dichotomizing cytology results (ASC-US or worse vs. less than ASC-US), the agreement increased to 86% (kappa 0.69). We observed an increasing likelihood of testing positive for markers associated with HPV-related transformation, p16/Ki-67 and HPV oncogene mRNA, with increasing severity of cytology results both for individual cytologists and for consensus cytology interpretation (ptrend < 0.0001 for all). In summary, we observed moderate to good agreement between two cytopathologists evaluating anal cytology samples collected in HIV-positive MSM. A higher severity of anal cytology was associated with biomarkers of anal precancers. Anal cytology may be used for anal cancer screening in high-risk populations, and biomarkers of HPV-related transformation can serve for quality control of anal cytology.

Keywords: anal cancer screening, cytology, human papillomavirus (HPV), human immunodeficiency virus (HIV), men who have sex with men (MSM), biomarkers


Anal cancer is uncommon in the general population, with incidence rates of about 2/100,000 in the United States (1). In certain high-risk populations, such as men who have sex with men (MSM), and human immunodeficiency virus (HIV)-positive men and women, the risks for anal cancer can be much higher and may approach the cervical cancer risk in unscreened populations of women. In MSM, anal cancer rates are estimated to be 40/100,000 (2-4), and in HIV-positive MSM the risk of anal cancer may be 2-4-fold higher (2;3;5;6) or more than in HIV-negative MSM. A recent analysis of 13 cohorts found that HIV-positive MSM were at the highest risk of anal cancer, followed by HIV-positive men or women, and all were at much higher risk than HIV-uninfected populations (7).

Analogous to cervical cytology, anal cytology has been recommended as a method for screening for the prevention of anal cancer through detection of precancerous lesions, anal intraepithelial neoplasia grade 3 (AIN3) and grade 2 (AIN2), and treatment. Surprisingly, there are limited data on the inter-observer or inter-rater agreement of anal cytology, unlike for cervical cytology. A previous study of 120 cytology slides from HIV-infected men reported a weighted Kappa for agreement between 4 pathologists evaluating the slides independently was 0.54 (8).

Since anal cancer is caused by the same causal factor as cervical cancer, persistent infection by high-risk human papillomavirus (HR-HPV), HPV measurements and related biomarkers might be used as objective measures potentially for quality control of anal cytology, just as HR-HPV is used for cervical cytology, specifically for ASC-US (9-11). Examples of other potentially useful biomarkers include detection of HPV E6/E7 oncogene mRNA, p16INK4a, and HPV16 (12), the most carcinogenic HPV genotype. Comparisons of anal cytology and histology results from laboratory data might also provide benchmarks for anal cytology (8;13;14).

To examine the issue of inter-rater agreement of anal cytology and the relationship of biomarkers with anal cytologic interpretations, we conducted an analysis in a population of HIV-positive MSM enrolled at an anal cancer-screening clinic in the Kaiser Permanente Northern California HMO.


Study population

The study was based at the San Francisco Kaiser Permanente Northern California (KPNC) Anal Cancer Screening Clinic. We enrolled men who were identified as HIV-positive through the Kaiser HIV registry, who were 18 years or older, who were not diagnosed with anal cancer prior to enrollment and who provided informed consent. In total, 363 men were enrolled between August 2009 and June 2010. The study was reviewed and approved by the institutional review boards at KPNC and at NCI. All participants were asked to complete a self-administered questionnaire to collect risk factor information. Additional information on HIV status and medication, sexually transmitted diseases, and histopathology results were abstracted from the KPNC clinical database.

For 87 subjects of 271 without biopsy-proven AIN2 or AIN3 at the enrollment visit, follow-up information on outcomes from additional clinic visits up to 12/2011 was available and included in the analysis to correct for possible imperfect sensitivity of high-resolution anoscopy (13;15).

Clinical exam, evaluation, and results

During the clinical examination, two specimens were collected by inserting a wetted flocked nylon swab (16) into the anal canal up to the distal rectal vault and withdrawing with rotation and lateral pressure. Both specimens were transferred to PreservCyt medium (Hologic, Bedford, MA). A third specimen was collected for routine Chlamydia Trachomatis and Neisseria Gonorrhea testing. After specimen collection, participants received a digital anorectal exam followed by high-resolution anoscopy (HRA). All suspicious-appearing lesions in HRA were biopsied and sent for routine histopathological review by KPNC pathologists, which was graded as condyloma, and AIN 1-3. No cancers were observed in this study population.

From the first specimen, a ThinPrep slide (Hologic) was prepared for routine Pap staining and evaluation. Two pathologists (T.D. and D.T.) reviewed the slides independently. Cytology results were reported analogous to the Bethesda classification (17) for cervical cytology except otherwise noted. The following categories were used: NILM (negative), atypical squamous cells of undetermined significance (ASC-US), atypical squamous cells cannot rule out HSIL (ASC-H), low-grade squamous intraepithelial lesion (LSIL), high-grade squamous intraepithelial lesion (HSIL), favor AIN2 (HSIL-AIN2), and HSIL-AIN3. ASC-H, HSIL-AIN2, and HSIL-AIN3 were combined into a single, high-grade cytology category for this analysis.

Biomarker testing

Using the residual specimen from the first collection, mtm Laboratories (Heidelberg, Germany) performed the p16INK4a/ki-67 dual immunostaining (“p16/ki-67 staining”) using their CINtec Plus cytology kit according to their specifications. A Thinprep 2000 processor (Hologic) was used to prepare a slide, which then stained for according to the manufacturer's instructions. The CINtec Plus cytology kit was then applied to the unstained cytology slide for p16/ki-67 staining.

On the second collected specimen, Roche Molecular Systems (Pleasanton, CA) tested for HRHPV, including separate detection of HPV16, and HPV18 DNA, using their cobas 4800 HPV assay. To prepare DNA for cobas, automated sample extraction was performed as follows: 500 μL of the PreservCyt specimen was pipetted into a secondary tube (Falcon 5-ml polypropylene round-bottom tube, 12- by-75-mm style, non-pyrogenic, sterile). The tube was capped, mixed by vortexing, uncapped, placed on the x-480 specimen rack and loaded onto the x-480 sample extraction module of the cobas 4800 system. The x-480 extraction module then inputs 400 μL of this material into the specimen preparation process. The extracted DNA was then tested as previously described (16).

Norchip also tested the second specimen for HPV16, 18, 31, 33, and 45 HPV E6/E7 mRNA using their Pretect HPV-Proofer assay according to their specifications. All testing was done masked to the results of the other assays, clinical outcomes, and patient characteristics.

Statistical analysis

For the agreement between the two cytology readers, we calculated the total agreement with a binomial 95% confidence interval (95%CI). We calculated Cohen's kappa with 95%CI as a chance-corrected measure of agreement as described in (18). Since kappa does not account for the degree of disagreement between categories and treats any disagreement equally, we calculated linear weighted kappa with 95%CI for the ordered cytology categories. Thus, disagreement between adjacent categories results in lower reduction of kappa values than disagreement between non-adjacent categories. Kappa values of <0.20 were interpreted as poor, between 0.21-0.40 as fair, between 0.41-0.60 as moderate, between 0.61-0.80 as good and above 0.80 as very good. Exact versions of symmetry (4 category) and NcNemar (2 category) chi-square tests were used to test for statistically significant differences in the distribution of the cytologic interpretations between readers. A non-parametric test of trend was used to assess the trend in the proportion of positives for each biomarker for risk of AIN2 or more severe (AIN2+) with increasing severity of the cytologic interpretation (19). Finally, a Fisher's exact test was used to test for differences in the proportion of positives for each biomarker between sub-groups defined by the paired cytologic interpretations.


The 363 men enrolled in the study had median, mean, and range of age of 53 years, 53 years, and 26-79 years, respectively. Most men were users of highly active anti-retroviral therapy (93%), 89% of men had an HIV viral load <75 copies, and 97% had a CD4 count higher than 200 (82% higher than 350) at the time of enrollment. Of the 363 men who enrolled in the study, 339 (93%) men had cytologic interpretations available from both study cytopathologists and were the basis of this analysis. The 24 men who were not included in the analysis due to missing cytology interpretations had a non-significantly lower percentage of HR-HPV DNA (65% vs. 80%; p=0.09).

Table 1 shows the comparison of the cytologic interpretations by the two cytopathologists (readers). The first reader called 33% as negative, 22% as ASC-US, 20% as LSIL, and 26% as high-grade cytology. The second reader called 43% as negative, 10% as ASC-US, 24% as LSIL, and 23% as high-grade cytology. The crude agreement was 66% (95%CI = 61%-71%), kappa was 0.54 (95%CI = 0.47-0.60), and linear weighted was 0.69 (95%CI = 0.63-0.74). Reader 1 was more likely to interpret the cytology more severe (p < 0.0001). Re-categorized the cytology as negative or ASC-US or more severe, the crude agreement was 86% (95%CI = 82%-9-%) and the kappa was 0.69 (95%CI = 0.61-0.76). Reader 1 was more likely to interpret the cytology as ASC-US or more severe (p < 0.0001).

Table 1
Inter-rater agreement for cytologic interpretation by two readers. Bold type highlights exact agreement, italic type indicates the cells the contribute the greatest to disagreement.

In Table 2, we show the relationships of various biomarkers and risk of having a histologic diagnosis of AIN2+ with the individual and paired cytologic interpretations. There was a significant trend (ptrend < 0.0001) of an increasing likelihood of testing positive for any of the biomarkers and or having an AIN2+ diagnosis with increasing severity of the cytologic interpretation for each rater individually. Similarly, there was a significant trend (ptrend < 0.0001) of increasingly likelihood of testing positive for any of the biomarkers and or having an AIN2+ diagnosis with increasing severity of consensus cytologic interpretation. Although the numbers for specific pairs of discordant cytologic interpretations were small, making generalization difficult, there was a tendency for these paired results to reflect a mixture of both over-called and under-called cytologic interpretations, as indicated by the intermediate positivity of the biomarker results compared to the consensus paired results i.e., ASC-US/ASC-US<ASCUS/LSIL or LSIL/ASC-US< LSIL/LSIL.

Table 2
The relationship of biomarker results and paired cytology results from the two raters. For each paired cytology results, the number and percent positive for HPV16 DNA, high-risk HPV (HR-HPV) DNA, p16INK4a/ki-67 immunocytochemistry (p16), HPV16, 18, 31, ...

However, we observed a large number of the discordant pair results of ASC-US/Negative (rater 1/rater 2). Comparing the profiles of biomarker positivity and risk of AIN2+ (Figure 1), we noted that the profile of the ASC-US/Negative subgroup was more akin to Negative/Negative than to ASC-US/ASC-US. Specifically, the % positive for HR-HPV DNA and p16/ki-67 staining for ASC-US/Negative was significantly lower than for ASC-US/ASC-US (p = 0.02 and 0.03, respectively) but not significantly higher than for Negative/Negative (p = 0.6 and p = 1, respectively).

Figure 1
The relationship of biomarker results for paired cytology results of Negative/Negative, ASC-US/Negative , and ASC-US/ASC-US (Rater 1/Rater 2). For each paired cytology results, the percent positive for HPV16 DNA, high-risk HPV (HR-HPV) DNA, p16INK4a immunocytochemistry, ...


In our analysis, we found moderate to good agreement between two cytopathologists evaluating anal cytology using samples from HIV-infected MSM. When compared to Lytwyn et al. (8), we found a better linear-weighted kappa, 0.69 vs. 0.54 (overall for 4 pathologists), but a worse un-weighted kappa, 0.54 vs. 0.69 (median). Thus, in Lytwyn et al. (8) there was better exact agreement but when there was disagreement in the severity of the cytology the discrepancies were more pronounced compared to this analysis. Any differences in inter-rater agreement between studies may be due to differences in the screening and treatment between populations, resulting in differences in the size of the lesions and the number of diagnostically informative cells on a slide. We also used a different collection device, flocked nylon swab (16), than the typical Dacron swab, which may have altered the number of diagnostic cells on a slide. Finally, reader 2, an experienced cytopathologist who had only read cervical cytology prior to the study, received training for anal cytology from reader 1 before the study started, which might have also influenced the agreement between the cytopathologists. Of note, in our study, the histologic confirmation even of consensus HSIL cytology results was limited, due to the limited performance of high resolution anoscopy that is widely recognized (13).

With increasing annual rates of anal cancer in the U.S. (Figure 2), it will be important to establish screening programs targeting high-risk populations such as HIV-positive MSM and HIV-infected men and women (7). Although there is no established method of anal cancer screening, cytology has been recommended (2) and its use may be cost effective in high-risk populations (20). We also showed that the detection of several biomarkers and the diagnosis of AIN2+ increased with increasing severity of anal cytology, as been shown for cervical cytology. These biomarkers therefore might be useful as objective standards to help monitor and maintain performance of anal cytology. For example, retrospectively reviewing anal cytology interpreted as HSIL in conjunction with biomarker results may improve individual pathologists diagnostic accuracy and identify false negative and false positive diagnoses.

Figure 2
Annual age-adjusted anal cancer incidence rates in the U.S. for both sexes (A), males (B), and females (C). Data are from Incidence source: SEER 9 areas (San Francisco, Connecticut, Detroit, Hawaii, Iowa, New Mexico, Seattle, ...


The Intramural Research Program of NIH/National Cancer Institute supported this study.


Disclosures: Dr. Castle serves as a member of a Data and Safety Monitoring Board for Merck.

Dr. Castle has received HPV tests and testing for research at a reduced or no cost from Qiagen and Merck.

Dr. Darragh has received research supplies for anal ThinPreps from Hologic; however, not for this project.

Reference List

1. Siegel R, Naishadham D, Jemal A. Cancer statistics, 2012. CA Cancer J.Clin. 2012;62:10–29. [PubMed]
2. Chin-Hong PV, Palefsky JM. Natural history and clinical management of anal human papillomavirus disease in men and women infected with human immunodeficiency virus. Clin.Infect.Dis. 2002;35:1127–34. [PubMed]
3. Clark MA, Hartley A, Geh JI. Cancer of the anal canal. Lancet Oncol. 2004;5:149–57. [PubMed]
4. Daling JR, Weiss NS, Hislop TG, et al. Sexual practices, sexually transmitted diseases, and the incidence of anal cancer. N.Engl.J.Med. 1987;317:973–7. [PubMed]
5. Diamond C, Taylor TH, Aboumrad T, Bringman D, Anton-Culver H. Increased incidence of squamous cell anal cancer among men with AIDS in the era of highly active antiretroviral therapy. Sex Transm.Dis. 2005;32:314–20. [PubMed]
6. Rabkin CS. Association of non-acquired immunodeficiency syndrome-defining cancers with human immunodeficiency virus infection. J.Natl.Cancer Inst.Monogr. 1998:23–5. [PubMed]
7. Silverberg MJ, Lau B, Justice AC, et al. Risk of Anal Cancer in HIV-Infected and HIV-Uninfected Individuals in North America. Clin.Infect.Dis. 2012;54:1026–34. [PMC free article] [PubMed]
8. Lytwyn A, Salit IE, Raboud J, et al. Interobserver agreement in the interpretation of anal intraepithelial neoplasia. Cancer. 2005;103:1447–56. [PubMed]
9. Cibas ES, Zou KH, Crum CP, Kuo F. Using the rate of positive high-risk HPV test results for ASC-US together with the ASC-US/SIL ratio in evaluating the performance of cytopathologists. Am.J.Clin.Pathol. 2008;129:97–101. [PubMed]
10. Moriarty AT, Schwartz MR, Eversole G, et al. Human papillomavirus testing and reporting rates: practices of participants in the College of American Pathologists Interlaboratory Comparison Program in Gynecologic Cytology in 2006. Arch.Pathol.Lab Med. 2008;132:1290–4. [PubMed]
11. Scott DR, Hagmar B, Maddox P, et al. Use of human papillomavirus DNA testing to compare equivocal cervical cytologic interpretations in the United States, Scandinavia, and the United Kingdom. Cancer. 2002;96:14–20. [PubMed]
12. Sahasrabuddhe VV, Luhn P, Wentzensen N. Human papillomavirus and cervical cancer: biomarkers for improved prevention efforts. Future.Microbiol. 2011;6:1083–98. [PMC free article] [PubMed]
13. Mathews WC, Sitapati A, Caperna JC, Barber RE, Tugend A, Go U. Measurement characteristics of anal cytology, histopathology, and high-resolution anoscopic visual impression in an anal dysplasia screening program. J.Acquir.Immune.Defic.Syndr. 2004;37:1610–5. [PubMed]
14. Williams VM, Metcalf C, French MA, McCloskey JC. Audit of paired anal cytology and histopathology outcomes in patients referred to a public sexual health clinic. Sex Health. 2010;7:346–51. [PubMed]
15. Chiao EY, Giordano TP, Palefsky JM, Tyring S, El SH. Screening HIV-infected individuals for anal cancer precursor lesions: a systematic review. Clin.Infect.Dis. 2006;43:223–33. [PubMed]
16. Gage JC, Ghosh A, Borgonovo S, et al. A comparison of dacron versus Flocked nylon swabs for anal cytology specimen collection. Acta Cytol. 2011;55:364–7. [PMC free article] [PubMed]
17. Solomon D, Davey D, Kurman R, et al. The 2001 Bethesda System: terminology for reporting results of cervical cytology. JAMA. 2002;287:2114–9. [PubMed]
18. Shoukri MM. Measures of Interobserver Agreement. Chapman and Hall/CRC; 2004.
19. Cuzick J. A Wilcoxon-type test for trend. Stat.Med. 1985;4:87–90. [PubMed]
20. Goldie SJ, Kuntz KM, Weinstein MC, Freedberg KA, Palefsky JM. Cost-effectiveness of screening for anal squamous intraepithelial lesions and anal cancer in human immunodeficiency virus-negative homosexual and bisexual men. Am.J.Med. 2000;108:634–41. [PubMed]